What is Crawl-Delay in robots.txt?

30 Dec, 2008

Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server.

User-agent: *
Crawl-delay: 10

Source: Wikipedia

Love what you've just read? Subscribe to our newsletter to receive tips, resources and special offers related to web development & design.

One Response to “What is Crawl-Delay in robots.txt?”

Prashant says:

November 30, 2009 at 7:29 pm

This is very important for website getting a lot of traffic and crawler pings as it prevents the servers from using too much resources at once

Hello! Welcome to Web development blog! My name is Ei Sabai and on this blog, I write about web development, mobile app development, latest web technologies and the likes. Read more about me or have a look at some of the tips & resources I've written.

Search

Tips & Resources

WordPress Web Hosting: Recommended web hosting providers for WordPress 3.0
iPhone Native App Development: Important steps into iPhone app development for beginners
iPhone Web App Development: Tips for iPhone web app development
Coupons for Web Developers: Get discounts on web hosting, domain names, templates, etc
10 Useful jQuery Snippets: Easy-to-use jQuery snippets for any website
HTML Email Newsletter: Step-by-step tutorial on how to code an HTML email newsletter

More Tips & Resources

Web Development Blog

by Ei Sabai Nyo