Skip to content
ZAVIA

Robots.txt Generator

Create a robots.txt file in seconds — allow or block search engines, specific bots, and folders, add your sitemap, and download the file.

Free · No sign-up · Runs in your browser — nothing is uploaded

Crawl rules

Live preview — robots.txt

# robots.txt generated with Zavia — https://zaviamagazine.dev/tools/robots-txt-generator

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Upload this file to your domain root, e.g. example.com/robots.txt — search engines only look there.

How to use

  1. Pick a preset or build custom rules.
  2. Add per-bot allow and disallow rules.
  3. Add your sitemap URL.
  4. Download robots.txt and upload it to your site root.

Formula

User-agent plus Allow and Disallow rules, one per line; Sitemap with the full URL on its own line

Example

User-agent: * / Disallow: /admin/ / Sitemap: https://example.com/sitemap.xml

FAQ

Where do I upload robots.txt?

At your domain root, e.g. example.com/robots.txt — search engines only look there.

Does robots.txt block all bots?

No — it is a polite request. Well-behaved search engines obey it, but bad bots may ignore it.

What is the difference between Disallow and noindex?

Disallow stops crawling; noindex stops indexing. A disallowed page can still appear in results if other pages link to it.

Is my data stored?

No. The file is generated entirely in your browser and downloaded to your device.

Related tools