Sunday, May 3, 2009

Robots.txt blocking blogger blogs?

Google Wemaster Tools reports for example: http://www.herikstad.net/search/label/clean%20cron is blocked by robots.txt

No worries however, according to the FAQ and http://www.google.com/support/forum/p/Webmasters/thread?tid=33c2e6597951a702&hl=en, this is just to remove results from the blog search tool included on the page (the URL contains "search" directory). Including the results would give duplicate sites in Google search and give incorrect visiting stats etc.

If you don't have "search" in your URL error, check out the link above.

No comments:

Post a Comment