Enter a domain or URL to scan your sitemap, or fill in the form manually. The preview updates as you edit.
Uses publicly accessible robots.txt and sitemap.xml on your server, then fetches a limited number of HTML pages for titles and descriptions. Some sites block automated requests.
Upload the file to your site root (for example https://example.com/llms.txt). Downloading here does not publish it. This is not a substitute for robots.txt and does not guarantee AI crawling, indexing or citations.
A proposed convention for human-readable site context - not a standard enforced by browsers or search engines.
Title, summary quote, optional sections and linked pages - formatted so people and tools can skim your intent.
Unlike robots.txt, llms.txt does not block or permit crawling. Use robots.txt and proper security for access policies.
Publishing llms.txt does not promise SEO gains, rankings or that AI products will read or follow it.
llms.txt is a plain-text (often Markdown) file at your site root that summarizes what your site is about and points readers - human or automated - to important pages. It is meant to be concise and curated, not a dump of every URL on your domain.
This generator helps you draft that file in the browser: site name, URL, short description, optional overview, instructions, and a list of linked pages with descriptions. Copy or download when you are satisfied; Compathub does not publish the file for you.
| File | Primary purpose |
|---|---|
| robots.txt | Crawl permission hints for bots (generator). Not a page catalogue for humans. |
| sitemap.xml | Machine-readable URL list for search discovery (generator). Not a narrative site guide. |
| llms.txt | Short site summary plus selected links and context for documentation-oriented use. |
Scan mode sends the website URL you enter to Compathub’s server. The scan reads robots/sitemap sources when available, scores candidate URLs (up to 500), and fetches up to 48 HTML pages for titles and descriptions. It processes a limited number of sitemap files per run. Pages behind login, omitted from sitemaps, or blocked by robots may never appear; redirects to other hostnames are stopped for safety.
Scanning does not call third-party AI models to write your file; it collects public metadata you can edit. Manual mode keeps everything local to your browser until you download.
Upload the finished file so it is reachable at https://yourdomain/llms.txt. There is no universal guarantee that search engines, AI assistants, or other products will read, prefer, or cite it. It is not a Google ranking requirement and does not replace sitemaps, structured data, or good on-page content.
Will llms.txt boost SEO or AI citations? No promises - treat it as optional documentation.
Can I start from my sitemap? Scan may reuse sitemap URLs, but you should still curate the final list.
For crawl rules and XML discovery lists, use the Robots.txt Generator and XML Sitemap Generator alongside this file.