Robots.txt

Robots.txt is a plain-text file at the domain root (https://example.com/robots.txt) that tells crawlers which paths to crawl and which to skip. It is public and does not protect data — a page hidden only by robots.txt is still reachable by anyone who knows the URL.

Basic syntax

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml

• User-agent selects the crawler; * means all of them.
• Disallow blocks a path; Allow makes an exception.
• Sitemap points to the sitemap location.
• Rule order can matter; crawlers generally use the most specific rule.

What people usually block

What to watch for

• Robots.txt is not security. Use authentication, permissions, or a firewall to restrict access.
• Do not block CSS/JS. Crawlers need those assets to understand the page.
• Be careful with Disallow: /. One wrong line can remove the whole site from the index.
• Check after deploy. Open /robots.txt in a browser and remove stale staging rules.
• Use noindex for specific pages. Robots.txt prevents crawling; noindex prevents indexing. They solve different problems.

After changing the file, test it with Google Search Console → Robots.txt Tester or a similar tool, then confirm important pages are still crawlable.

If the goal is to keep a page out of search results, use noindex; if the goal is to restrict access, use authentication. Robots.txt only controls crawling, not access, and non-compliant crawlers can ignore it.

FAQ