If it linked to the source of the image, wouldn't it just be a minimalist search engine? Surely google doesn't individually license every image that shows up in image search.
I don't know what you think the site is doing but that is exactly what it is. It didn't curate the images, people didn't upload the images. It searched for an image that matched your word. No google didn't licence the image, but it only displays small thumbnails with a link to where it found the image. This is displaying the full size image with no link to where it found the image.
Just because the image is copyrighted doesn't mean you are violating copyright law by displaying it. It would be quite nice of you to display a link to the site that Google found the image on - but if this is a non-commercial site, you have a very good fair use claim. If you really wanted to be in the clear, you could change your image search default to show "images labeled for reuse." Either way, don't be cowed by copyright maximalists.
