robots.txt generator

What is robots.txt and what is it actually for

robots.txt is a plain text file that sits in the root of a domain, at https://example.com/robots.txt. Crawlers read it first, and only then start going through the site. The rules are simple: User-agent marks who a rule applies to, Disallow blocks access to a path, and Allow exempts it from a broader block. The asterisk * means all bots.

It is important to understand what the file is and what it is not. It is a recommendation, not protection. A bot that respects the rules will follow them, but a malicious scraper will not. That is why you should never put passwords, API keys, backup files or admin panels on a site hoping robots.txt will hide them. If something must not be public, protect it with authentication or .htaccess rules.

Another important role is saving crawl budget. On a site with tens of thousands of URLs, filters, sorting and calendars can waste crawler time on pages with no value. Blocking such paths helps Google focus on the content that brings traffic.

How to write robots.txt for your site

The order of steps matters, because rules are read from top to bottom and the most specific rule wins.

1. Define a group for all bots: User-agent: *
2. Add blocks for technical and duplicate paths, for example Disallow: /wp-admin/, Disallow: /search, Disallow: /*?sort=
3. Allow what sits inside a blocked path but must be indexed: Allow: /wp-admin/admin-ajax.php
4. List the sitemap: Sitemap: https://example.com/sitemap.xml
5. Save the file in the root and check it in the Google Search Console robots.txt testing tool.

A practical tip: write paths without the domain and with a leading slash. Disallow: /cart is correct, Disallow: example.com/cart is not. If you want to block everything except one path, it is possible, but easy to get wrong, so double check the result.

Should you block AI crawlers

AI companies use their own bots: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and others. Blocking prevents models from training on your text, but at the same time reduces the chance of AI search engines citing you as a source. For most businesses that live on visibility, a better strategy is to let them in and offer them clean content and an llms.txt file with a short description of the site and key pages.

There are exceptions. Sites with original research, paid content or authored text can justifiably block training while allowing citation. Some bots support this through separate names, so check the documentation before writing a rule.

Most common mistakes when creating a robots.txt file

The most common problem is blocking an entire section that also contains pages that should be indexed. Example: Disallow: /blog/ also removes the articles that bring visits. The next mistake is relying on Disallow to remove a page from the Google index. If a page is already indexed, blocking the crawler prevents Google from seeing the noindex meta tag, so the page can stay in the results with an outdated description. Use noindex for removal, and Disallow only as a temporary measure.

The third mistake is a wrong path to the sitemap file or a forgotten Sitemap line. The fourth is copying someone else's file without adapting it to your site structure. The fifth is blocking CSS and JS files, which stops Google from rendering the page properly and assessing it.

If you have multiple subdomains, remember that each domain has its own robots.txt. When a site has thousands of URLs, multiple language versions and complex rules per bot, manual maintenance becomes risky. In such cases EVI Solutions can help with technical SEO setup and automation.