Build the crawl input before you run the crawl.
A URL list is a scope decision. Normalize obvious formatting differences, keep ambiguous variants visible, and know what still requires a live request.
Do not let cleanup redefine the site.
Removing a fragment is usually appropriate for an HTTP crawl target. Removing every query parameter is not. Parameters may control pagination, filters, language, or content. The safe workflow separates obvious tracking parameters from variants that need review.
- 01 / Extract
Identify the URL source.
A URL sitemap and a sitemap index both use
<loc>, but they do not contain the same targets. A URL sitemap lists pages. A sitemap index lists child sitemap files. Keep those files out of the page queue unless the next process explicitly expects sitemap URLs. For a table, map the URL column instead of treating every cell as a crawl target. - 02 / Normalize
Make each rule explicit.
Lowercasing the hostname is safe; lowercasing the path can change the resource on case-sensitive servers. Removing fragments does not change the HTTP request target. Adding HTTPS to a bare domain is a convenience assumption and still needs a live redirect or status check.
Use a narrow default: remove common campaign parameters, strip fragments, preserve content parameters, and report invalid entries.
- 03 / Variants
Review what looks similar.
/productsand/Products, a trailing slash variant, and two parameter orders may resolve to the same page—or not. Group by host and flag suspicious forms. A preparatory tool should surface this ambiguity instead of silently choosing a canonical URL. - 04 / Scope
Export only the intended crawl set.
Set the allowed protocol and host, then apply narrow include and exclude patterns. Review the resource classification before export. A PDF, image, script, feed, or child sitemap may belong in a separate crawl. Download exclusions with their reasons instead of deleting them without a record.
- 05 / Verify
Separate normalization from observation.
The exported list does not prove that a URL exists, returns 200, is allowed by robots.txt, declares itself canonical, or contains indexable content. Check one URL with AnalyseSpider or run the larger list through an actual crawler or site audit.
One page survives the declared crawl scope.
The synthetic input mixes HTTP and HTTPS, an alternate host, a tracking duplicate, a private path, a PDF, and invalid text. The recipe keeps HTTPS URLs on the exact host under /docs/* and excludes PDFs.
- Accepted
https://example.com/docs/start- Excluded
- Four normalized URLs, each with a recorded reason.
- Rejected
- One value that cannot be parsed as an HTTP URL.
URL and sitemap contracts.
- Sitemaps XML protocol
The protocol defines the required
<loc>URL element and sitemap structure. - Google Search URL structure guidance
Reference for crawlable URL structures and parameter handling considerations.
- DevAwesome workbench core contract
Executable assertions for sitemap extraction, tracking-parameter removal, deduplication, invalid input, and host grouping.
No request is made here.
The browser-local workbench parses pasted text. It does not fetch the sitemap or any listed URL.
Treat the list as an input artifact.
Record the source, chosen rules, date, and excluded rows when crawl scope matters. That makes a later result explainable instead of merely repeatable.