BODY:
Generally in case of small websites webmasters are under false assumption that they do not require to create a robot.txt file. But it is required. First of all let us define the robot.txt file. Even prior to it we need to define what a web robot is. A web robot is also called spider or crawler which should not be confused with the normal web browser as web browser is not a robot.
The main use of robots.txt file for webmasters in SEO services is to give instructions to the robot what they crawl and what should not be crawled. This can give you some control over the robots. This give you little more control over the robots and this indicates that you can issue indexing instructions to various search engines.
Robots.txt invites the search engines. Some of the good bots can also step away from your website in case you have not created robot.txt in the top level of your website. Some time there is requirement to exclude some pages from search engines. These are those pages that are still under construction and those directories that you do not want to get them indexed. You may also want to exclude those search engines whose main aim is to collect email addresses.
Robot.txt file is a simple text file created in notepad. This is required to be saved to the root directory of your website. It means that directory where your home page or index page is stored. In order to create a simple robot.txt file with the specification that allows all robots to spider your website, write the following info:
User-agent: *
Disallow:
This will allow all robots to index your pages.
In case you don’t want a specific robot to have access any of your website pages. Then do the following:
User-agent:specificbot
Disallow: /
In case you do not want a specific robot to access any of your web pages, then do the following. Suppose you do not want Googlebot to index a page names as “abc
