Set user-agent groups, allow/disallow paths and sitemap URLs. The preview updates as you edit.
Upload the file to your website root (for example https://example.com/robots.txt) for compliant crawlers to read it. Downloading here does not publish the file. Rules do not secure private content or guarantee rankings.
It is a plain-text policy file for crawlers that choose to follow it - not a firewall or authentication layer.
This tool outputs User-agent, Allow, Disallow and Sitemap lines supported by major crawlers - not vendor-specific extensions.
We flag site-wide blocks and paths that often cover CSS or JavaScript so you can avoid accidental crawl or rendering issues.
Your rules never leave the browser during generation. You control upload to your own server.
robots.txt is a plain-text file at the root of a host (for example https://example.com/robots.txt) that tells well-behaved crawlers which paths they should or should not request. This generator runs entirely in your browser: you build groups, preview the file, then copy or download it. Compathub does not upload it to your server.
Robots.txt is not authentication. It does not stop people from visiting URLs directly, and not every bot obeys it.
* for all agents, or a specific product name)./ or common asset paths.Lines after a User-agent line belong to that group until the next group begins. You can add multiple user-agent lines in one group when they share the same rules.
| Rule | Effect for compliant crawlers |
|---|---|
Disallow: /admin/ | Discourages crawling URLs under /admin/ - still not a substitute for login protection. |
Allow: /public/ with narrower disallows elsewhere | Helps express exceptions when you use broad disallow patterns (behaviour depends on crawler). |
Sitemap: https://example.com/sitemap.xml | Advertises where your sitemap lives; does not create the sitemap file itself. |
Blocking a URL in robots.txt asks crawlers not to fetch it. That is different from removing a URL from a search index. If a URL is blocked, some crawlers may never see a noindex directive on the page itself - Google documents this interaction in Search Central guidance on robots meta tags.
To keep a page off results while staying crawlable, use on-page noindex (and remove internal links), not robots.txt alone.
Disallowing CSS, JavaScript, or image paths can interfere with how crawlers render pages. Block only paths you deliberately want unfetched, and test after publishing.
www vs apex each need their own file unless one redirects to the other)./robots.txt in a browser or use Search Console’s robots report after upload.Does Compathub host my robots.txt? No - you download and deploy it.
Is this llms.txt? No - llms.txt is an optional documentation-style file for humans and tools, not crawl permission rules.
Need a sitemap URL to list? Use the XML Sitemap Generator, then paste the absolute sitemap address into the Sitemap field here.