Rotbot Text Generator

The robots.txt file tells visiting robots (search engine crawlers) to index the content of your website pages. The robots use these text files as information to what they can and can't index within their search engine. The robot.txt file it self sits in your public html directory so that it can be viewed by the broswer.


Tip: Select user agent or use default for all agents



Enter URLs you wish to exclude