The robots.txt file

Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many web sites don't offer this file.

If your web site doesn't have a robots.txt file Till, read on to learn how to create one. If you already have a robots.txt file, read our tips to make sure that it doesn't contain errors.

What is robots.txt?

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which web pages of your site should be indexed and which web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.abc.com/robot.txt

How do I create a robots.txt file?

As mentioned above, the robots.txt file is a simple text file. Open a simple text editor to create it. The content of a robots.txt file consists of so-called "records".

A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines. Here's an example:

Things you should avoid

>> Don't use comments in the robots.txt file
>> Don't use white space at the beginning of a line. For example, don't write
>> Don't change the order of the commands. If your robots.txt file should work, don't mix it up.
>> Don't use more than one directory in a Disallow line. Do not use the following "Disallow: /support /cgi-bin/ /images"
>> Be sure to use the right case.The file names on your server are case sensitve