Robots.txt is a special text file on your website that instruct robots whether or not to crawl and index certain pages and directories within your site.
The generated code must be placed inside a text file called robots.txt that is located at the root level of your website. For example, the robots.txt file for google.com is located at http://www.google.com/robots.txt
Looking for a hosting provider for your website? Check out hostgator, our hosting provider of choice. Unlimited Domains, Disk Space and Bandwidth for just $7.95 per month.
Special offer: Get started from just 1 cent! Enter wordpress as the coupon code.
- Unlimited Disk Space and Bandwidth
- Flexible, Easy to Use Control Panel
- Unlimited Sub Domains, FTP Accounts, and Email Accounts
- 99.9% Uptime Guarantee
- No Contract with a 45 Day Money Back Guarantee