Robots.txt Generator

How to use the robots.txt generator

1. Choose a starting point that matches your site.
2. Add any paths that should not be crawled.
3. Download the file and place it at the root of your domain.

About this tool

robots.txt tells crawlers which parts of a site they may fetch. It is a request, not a control: well-behaved crawlers obey it, and nothing prevents anyone else from ignoring it.

Two misunderstandings cause most of the damage. First, blocking a URL here does not remove it from search results — Google can still index a URL it has never crawled, showing it without a description. To keep a page out of results, let it be crawled and use a noindex tag. Second, the file is public, so listing a private path advertises exactly where it is.

The AI opt-out adds rules for the major training crawlers. Whether they honour it is their choice, and the list changes as new ones appear.

Common uses

• Keeping admin and search pages out of crawl budget.
• Blocking a staging site from being indexed.
• Opting out of AI training crawlers.

Frequently asked questions

Will robots.txt remove a page from Google?

No. Blocking crawling can leave the URL indexed without a description. Use a noindex meta tag on a crawlable page instead.

Where does the file go?

At the root of the domain — example.com/robots.txt. Crawlers do not look anywhere else.

Does blocking AI crawlers actually work?

Only for crawlers that choose to honour robots.txt. It is a stated preference, not enforcement.