Google Email Titled: "Important information about your Google Product Search images" 4/11/2011

Google has recently started sending out emails to customers about potentially not being able to access images. See the bottom of this post for the text of the message.

Apparently this is based on some test they are doing of the robots.txt files on the servers with the images. The test they are doing appears to be looking for specific text in the files to allow Google to access everything, and not actually testing whether their server is allowed access to the images. We do have a robots.txt file on one of our servers that is specifically crafted, and tested, to allow them access to images on our server.

We tested this when we updated it, as well as tested again once Google started sending out these messages. We even used Google's own tools to test if they can reach the image urls, and their Webmaster tools confirm that they can access the images. They also are showing all customer images in Google Product Search.

Also, for those on our second image server, they are still sending this message even though there was no robots.txt file at all (which means nothing is blocked). We have added the lines on that server just so they would stop sending the emails even though it has no impact on anything.

So the net result is that Google can access the images, and that this warning can be safely ignored. We will continue to monitor and if there is anything we need to adjust, we will. If you see any issues, let us know, but as of this time we see no reason for concern.

Just as an aside, others are also getting the emails that also have completely valid robots.txt files and images are working for them as well.


Thank you for participating in Google Product Search. It has come to our attention that a robots.txt file is preventing us from crawling some or all of the images on your site. In order for us to access and display the images you provide in your product listings, we'd like you to modify your robots.txt file to allow user-agent 'googlebot' to crawl your site.
Failure for Google to access your images may affect the visibility of your items on Google Product Search and Product Ad results.

To ensure the 'googlebot' is not being blocked, please add the following two lines of text to the end of your robots.txt file:

User-agent: googlebot

For more information on robots.txt files, please visit If you have any questions, please contact your webmaster directly.