Robots.txt Generator
Create a robots.txt file in seconds — allow or block search engines, specific bots, and folders, add your sitemap, and download the file.
Free · No sign-up · Runs in your browser — nothing is uploaded
Crawl rules
Live preview — robots.txt
# robots.txt generated with Zavia — https://zaviamagazine.dev/tools/robots-txt-generator User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Upload this file to your domain root, e.g. example.com/robots.txt — search engines only look there.
How to use
- Pick a preset or build custom rules.
- Add per-bot allow and disallow rules.
- Add your sitemap URL.
- Download robots.txt and upload it to your site root.
Formula
User-agent plus Allow and Disallow rules, one per line; Sitemap with the full URL on its own line
Example
User-agent: * / Disallow: /admin/ / Sitemap: https://example.com/sitemap.xml
FAQ
Where do I upload robots.txt?
At your domain root, e.g. example.com/robots.txt — search engines only look there.
Does robots.txt block all bots?
No — it is a polite request. Well-behaved search engines obey it, but bad bots may ignore it.
What is the difference between Disallow and noindex?
Disallow stops crawling; noindex stops indexing. A disallowed page can still appear in results if other pages link to it.
Is my data stored?
No. The file is generated entirely in your browser and downloaded to your device.
Related tools
Meta Tag Generator
Generate SEO meta tags, Open Graph and Twitter Card tags for any page — with live character and pixel-width counters and a Google search preview.
XML Sitemap Generator
Build an XML sitemap from a list of URLs — set priority, change frequency and last modified date, remove duplicates, and download sitemap.xml.
Keyword Density Checker
Check keyword density in any text — paste your content, enter a target keyword, and see density, word counts, and your top 10 keywords.
Schema Markup Generator (JSON-LD)
Generate JSON-LD schema markup — FAQ and Article structured data with live preview. Copy and paste into your page to win rich results.