Scan your site for URLs or enter pages manually. Up to 1,000 URLs per file when generating.
Discovers URLs from robots.txt, sitemaps, and same-site HTML links (limited depth). Respects robots.txt where readable. Does not run JavaScript or submit forms.
Stage 1 of 6 - Validating the website
| Include | URL | Title | Status | Notes | Actions |
|---|
You must upload the file to your own server and reference it in robots.txt if desired. Compathub does not submit sitemaps to Google or other search engines.
It formats URLs you provide - or URLs discovered by the optional scan - into sitemap protocol XML. Scans are bounded and do not execute JavaScript on the target site.
Uses the standard sitemaps.org namespace and escapes special characters so parsers can read the file safely.
URLs on a different host from your base URL are reported clearly - they are not silently rewritten to your domain.
Generation happens locally in your browser. Lists of URLs are not sent to Compathub servers.
An XML sitemap lists URLs you want search engines to know about, using the standard sitemap namespace. Compathub helps you compose that file in two ways:
A sitemap helps discovery; it does not guarantee indexing, ranking, or crawl frequency. See Google’s sitemap overview for how major search engines treat sitemaps.
Include canonical, indexable pages you care about - typically 200 OK HTML URLs you want crawled. Generally exclude:
?session= noise, print views) when a clean canonical already exists.noindex or blocked intentionally from public indexing.Use absolute https:// URLs on your live hostname. The generator validates hostname consistency when you mix URLs.
You can maintain up to 1,000 URLs per generation in the editor (a responsiveness cap in this tool). The sitemap protocol allows larger files in production (50,000 URLs / 50 MB per file); split very large sites into multiple sitemap files and a sitemap index if you outgrow a single file.
Optional lastmod, changefreq, and priority fields are hints only - set last modified when you know the date; do not fabricate values.
Scan mode normalizes your input to a site origin, reads robots/sitemap sources when reachable, fetches HTML within configured limits, and respects same-site rules. You choose a discovery cap (100, 500, 1,000, or 5,000 URLs). The job may stop early on timeouts, rate limits, robots blocks, or redirects to other hosts - warnings appear in the UI.
Scan output is a starting list. Review it before publishing: remove admin paths, faceted URLs, or duplicates the crawler should not advertise.
sitemap.xml and upload it to your site (commonly /sitemap.xml).Sitemap: line in robots.txt - use the Robots.txt Generator.Does Compathub host the sitemap? No.
Is this the same as llms.txt? No - llms.txt is a separate, human-oriented summary format, not an XML crawl list.
After publishing, spot-check important URLs with the Redirect Checker and Website Response Time Checker so sitemap entries match the responses and latency you expect.