---
title: Robots.txt
description: A plain-text file at the root that tells crawlers which paths to crawl — not a security tool.
canonical: https://penasihathosting.com/en/hosting-wiki/robots-txt
type: wiki
locale: en
updated: 2026-10-29
author: Willya Randika
---

# Robots.txt

## Overview

- **Summary:** A plain-text file at the root that tells crawlers which paths to crawl — not a security tool.
- **Author:** Willya Randika ([profile](/penulis/willya-randika))

## Article

**Robots.txt** is a plain-text file at the domain root (`https://example.com/robots.txt`) that tells crawlers which paths to crawl and which to skip. It is public and does not protect data — a page hidden only by robots.txt is still reachable by anyone who knows the URL.

## Basic syntax

```text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
```

- `User-agent` selects the crawler; `*` means all of them.
- `Disallow` blocks a path; `Allow` makes an exception.
- `Sitemap` points to the sitemap location.
- Rule order can matter; crawlers generally use the most specific rule.

## What people usually block

| Path | Reason |
| --- | --- |
| `/wp-admin/` | Admin area, except `admin-ajax.php` |
| `/cart/`, `/checkout/` | Transaction pages that need no index |
| `/search?` | Internal search results |
| `/api/` | Data endpoints, not pages |
| Backup files or `.env` | Not public, but block them at the server |

## What to watch for

- **Robots.txt is not security.** Use authentication, permissions, or a firewall to restrict access.
- **Do not block CSS/JS.** Crawlers need those assets to understand the page.
- **Be careful with `Disallow: /`.** One wrong line can remove the whole site from the index.
- **Check after deploy.** Open `/robots.txt` in a browser and remove stale staging rules.
- **Use `noindex` for specific pages.** Robots.txt prevents crawling; `noindex` prevents indexing. They solve different problems.

After changing the file, test it with **Google Search Console → Robots.txt Tester** or a similar tool, then confirm important pages are still crawlable.

If the goal is to keep a page out of search results, use `noindex`; if the goal is to restrict access, use authentication. Robots.txt only controls crawling, not access, and non-compliant crawlers can ignore it.

## FAQ

<Accordion>
  <AccordionItem title="Can robots.txt hide a page from the public?">
    No. It is only an instruction for compliant crawlers. The page remains directly accessible.
  </AccordionItem>
  <AccordionItem title="Do all crawlers obey robots.txt?">
    Major crawlers usually do, but spam bots and scrapers do not. Never treat it as an access control.
  </AccordionItem>
  <AccordionItem title="How do I keep one page out of Google?">
    Use a `noindex` meta tag or the `X-Robots-Tag: noindex` header on that page, not only `Disallow`.
  </AccordionItem>
</Accordion>