Monday, 14 April 2008

google crawling forms

Google is starting to crawl forms to index the hidden web, according to their webmasterblog. They will honour a robots.txt or meta-tag forbidding the crawler to fill in the form, so if you don't want your databases polluted with google generated nonsense, use it!

No comments: