What is a robots.txt file?Creating a robots.txt file for your web site is one of the easiest things you can do, and also one of the most important. This file is like a map for the search engine crawlers and bots that come to your site to index it for search results. This file tells the search engine crawler where it can – and can’t go in your site, and what the crawler should and shouldn’t index. You can specify what each individual SE crawler can do, you can specify certain directories, and even file names.

Where is my robots.txt file?You may not have one for your site yet, if you haven’t created it. In the root directory of your site (where your home page index is), you need to create a new file and name it “robots.txt”. You can create this file in any text editor – like Notepad.

How do I write a robots.txt file?The easiest way to do it is to read some examples, and created one based on that.? Searchtools.com has one of the best basic examples you can find

Where can I find a current List of robot agents?? This list of search engine robot agents is a great resource, and frankly one of the only ones I could find that was current and up to date (as of this year).
How do I know if my robots.txt file is correct? Just use a robots.txt syntax checker for validation to make sure you set yours up right.

Where can I learn more about search engine robots and spiders?? Read this article all about search robots and spiders and how they work.