Robots.txt Crawl Rate
Recently we've had a problem with high trafic load on our servers for FeminaPortal - portal and forums for women, girls and ladies. It is well optimized and so far there were not any issues with it.
We determined that the problem was caused by Google search engine crawlers that performed many requests and that had impact our online classified site. It looks like we were being rapidly crawled by GoogleBot.