Understanding Robots.txt: A Comprehensive Guide for SEO

Search engine crawlers index websites regardless of the presence of robots.txt and sitemap.xml. However, using a robots.txt file allows webmasters to specify what should be excluded from indexing and to configure other important parameters.

It is important to note that search engine crawlers may ignore certain rules. For instance:

• Google Bot does not recognize the host directive and Crawl-Delay. Check the complete list of Google robots for more details.
• Specific robots may bypass general directives unless they are explicitly written for them.

Directives

Directives are rules for robots. There is a W3C specification from January 30, 1994, and an extended standard from 1996.

Commonly Used Directives

User-agent: This directive begins the robots.txt file.

Example:

User -agent: *  # instructions for all search engine robots.
User -agent: GoogleBot  # instructions for Google’s robot.
Disallow: /  # disallow directive, prohibits indexing of what follows.
Allow: /  # allow directive, specifies URLs to be indexed.
Disallow:  # does not work without the special character /.
Allow: /  # ignored if no URL follows /.

Special Characters Used in robots.txt

Be cautious with the character /; a common mistake is:

User -agent: *
Disallow: /  # this will block the entire site from indexing.

The special character * represents any character sequence, including an empty one. For example:

Disallow: /cart/*  # blocks all pages following the URL: site.com/cart/

The special character $ restricts the action of the character *, providing strict limitations:

User -agent: *
Disallow: /catalog$  # with this character, catalog will not be indexed, but catalog.html will be.

The sitemap directive specifies the path to the sitemap and looks like this:

User -agent: * 
Allow: / 
Sitemap: http://www.site.com/sitemap.xml  # must be specified with http:// or https://, use https:// if an SSL certificate is installed.

The Host directive indicates the main mirror of the website with or without www.

User -agent: *
Allow: / 
Sitemap: http://www.site.com/sitemap.xml 
Host: www.site.com  # specify the domain path without http and slashes.

The Crawl-Delay directive limits server load by setting a timeout for search engines:

User -agent: * Crawl-delay: 2  # sets a timeout of 2 seconds.
User -agent: * 
Disallow: /search 
Crawl-delay: 4.5  # sets a timeout of 4.5 seconds.

The Clean-Param directive is necessary if page addresses contain dynamic parameters that do not affect the content, such as session IDs, user IDs, referrers, etc.

Using the Clean-Param directive, search engines will avoid repeatedly crawling duplicate information, improving the efficiency of your website's crawling and reducing server load.

For example, pages with the following addresses:

www.site.com/some_dir/get_book.pl?ref=site_1&book_id=123
www.site.com/some_dir/get_book.pl?ref=site_2&book_id=123 
www.site.com/some_dir/get_book.pl?ref=site_3&book_id=123

The ref parameter is used solely to track which resource made the request and does not change the content. All three addresses will display the same page with book_id=123. So, if the directive is specified as follows:

User -agent: *
Disallow:
Clean-param: ref /some_dir/get_book.pl

The search engine will consolidate all page addresses into one:

www.site.com/some_dir/get_book.pl?ref=site_1&book_id=123,

It’s worth noting that there are several options for configuring this directive.

Cyrillic Characters in robots.txt

Using characters from the Russian alphabet is prohibited in robots.txt; Punycode must be used (a standardized method for converting Unicode character sequences into ACE sequences).

#Incorrect:
User -agent: *
Disallow: /basket
Host: internet-store.com

#Correct:
User -agent: *
Disallow: /basket
Host: https://internet-store.com

Recommendations on What to Block in robots.txt

• Administrative panel – keep in mind that the path to your admin panel will be known; ensure password security.
• Shopping cart, order forms, and delivery and order data.
• Pages with filter parameters, sorting, and comparison.

Common Errors in robots.txt

• Empty lines are not allowed in the user-agent directive, which is considered a "separator" according to robots.txt rules. This means the applicability of directives following an empty line cannot be predicted.
• In case of a conflict between two directives with the same prefix length, the Allow directive takes precedence.
• Only one Host directive is processed for each robots.txt file. If multiple directives are listed, the robot uses the first one.
• The Clean-Param directive is inter-sectional, so it can be specified anywhere in the robots.txt file. If multiple directives are specified, all will be considered by the robot.
• Some robots do not follow robots.txt rules. To prevent them from indexing your site, create separate user-agent parameters for each of them.
• The User-agent directive must always be written above any disallow directives.
• One line per directory. You cannot write multiple directories on one line.
• The file name must be exactly: robots.txt. No Robots.txt, ROBOTS.txt, etc. Only lowercase letters in the name.
• In the host directive, specify the domain path without http and slashes. Incorrect: Host: http://www.site.com/, Correct: Host: www.site.com (or site.com).
• If the site uses the secure protocol https, the host directive (for the search engine) must include the protocol, e.g., Host: https://www.site.com.

Checking for Errors in robots.txt with Tools

Utilize various online tools to check for errors in your robots.txt file.

Keep in mind that a file larger than 32 KB is read as fully allowing, regardless of its content.

Excessive filling of robots.txt. New webmasters may be impressed by articles stating that everything unnecessary must be blocked in robots.txt and begin to block almost everything except text on specific pages. This is, to put it mildly, incorrect. First, there is a recommendation not to block scripts, CSS, and other elements that may prevent the bot from seeing the site as a user does. Secondly, many errors arise from blocking one thing, which inadvertently blocks another. It is advisable to check the accessibility of a page and its elements. A common mistake is the confusion between Allow and Disallow directives. It is best to block only clearly unnecessary items for the bot, such as registration forms and redirect pages, while using canonical tags to eliminate duplicates. Note that correcting robots.txt does not mean that search engine bots will read it immediately. To expedite this process, check your robots.txt in the webmaster section.

Examples of properly configured robots.txt for different CMS:

WordPress

User -Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /tag
Disallow: /category
Disallow: /archive
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /?feed=
Disallow: /?s=

Host: site.com
Sitemap: http://site.com/sitemap.xml

ModX

User -agent: *
Disallow: /manager/
Disallow: /assets/components/
Disallow: /core/
Disallow: /connectors/
Disallow: /index.php
Disallow: *?
Host: example.com
Sitemap: http://example.com/sitemap.xml

OpenCart

User -agent: *
Disallow: /*route=account/
Disallow: /*route=affiliate/
Disallow: /*route=checkout/
Disallow: /*route=product/search
Disallow: /index.php?route=product/product*&manufacturer_id=
Disallow: /admin
Disallow: /catalog
Disallow: /download
Disallow: /export
Disallow: /system
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?order=
Disallow: /*&order=
Disallow: /*?limit=
Disallow: /*&limit=
Disallow: /*?filter_name=
Disallow: /*&filter_name=
Disallow: /*?filter_sub_category=
Disallow: /*&filter_sub_category=
Disallow: /*?filter_description=
Disallow: /*&filter_description=
Disallow: /*?tracking=
Disallow: /*&tracking=
Disallow: /*?page=
Disallow: /*&page=
Disallow: /wishlist
Disallow: /login
Disallow: /index.php?route=product/manufacturer
Disallow: /index.php?route=product/compare
Disallow: /index.php?route=product/category

Joomla

User -agent: *
Allow: /index.php?option=com_xmap&sitemap=1&view=xml
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /go.php
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Disallow: /*com_mailto*
Disallow: /*pop=*
Disallow: /*lang=en*
Disallow: /*format=*
Disallow: /*print=*
Disallow: /*task=vote*
Disallow: /*=watermark*
Disallow: /*=download*
Disallow: /*user/*
Disallow: /.html
Disallow: /index.php?
Disallow: /index.html
Disallow: /*?
Disallow: /*%
Disallow: /*&
Disallow: /index2.php
Disallow: /index.php
Disallow: /*tag
Disallow: /*print=1
Disallow: /trackback
Host: yoursite.com

In these examples, the User-Agent parameter is set to *, allowing access to all search engine robots. To configure robots.txt for specific search engines, replace the wildcard with the robot's name, such as GoogleBot, BingBot, etc.