Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server.
User-agent: * Crawl-delay: 10
Source: Wikipedia
One thought on “What is Crawl-Delay in robots.txt?”
Comments are closed.
This is very important for website getting a lot of traffic and crawler pings as it prevents the servers from using too much resources at once