Robot txt Optimization

Robots.txt was popularized with the advent of AltaVista, the first popular search engine. The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code.

The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.