robots.txt Generator – Allow, Disallow & Sitemap

Build a clear robots.txt file for search crawlers in under a minute.

How to use: Pick a preset or edit rules, add your sitemap URL, then copy the file.

How to use the robots.txt Generator

Choose a preset: allow all, block all, or custom. For custom (and as extras on other presets), list paths to disallow, one per line, such as /admin/ or /cart/.

Add your XML sitemap URL so crawlers can find it quickly. The generator writes standard User-agent and Disallow / Allow lines.

Copy the text into a file named robots.txt at your site root (https://yoursite.com/robots.txt). Hosts like Cloudflare Pages, Netlify and most static hosts serve it from the publish folder.

Optional AI-bot stubs are comments you can turn into real rules after you read each bot’s documentation — do not block Googlebot unless you intend to.

robots.txt is a request to well-behaved crawlers, not a security lock. Keep private pages behind authentication as well.

Take a minute to sanity-check units and labels before you rely on a number for school, work or money decisions. If a result looks surprising, change one input at a time and watch how the answer moves — that is the fastest way to learn the method. ViralScopeHub tools stay free, need no account and keep your numbers in the browser so homework and business figures are not uploaded to our servers.

Frequently asked questions

Where do I put robots.txt?

At the root of your domain: https://example.com/robots.txt

Does Disallow hide a page from Google?

It asks crawlers not to fetch the URL. Use noindex or login walls for stronger control.

Should I block everything?

Only for staging sites. Blocking all on a public site stops normal search indexing.

Is this file uploaded anywhere?

No. You copy or download it from your browser.

More games

All games