DevAwesomeOpen a tool
← All workflows

Build a clean crawl list

Normalize copied URLs, expose invalid entries, and export one stable list before a crawler touches it.

  1. 1Extract URLs
  2. 2Choose rules
  3. 3Review exceptions
  4. 4Export the list
No crawl.The browser parses your text without requesting any URL.
Crawl list workbenchExtract, normalize, group, review, export
1

Paste the source

The workbench extracts URLs from lists, tables, Markdown links, or pasted sitemap XML.

Reading the pasted source…

2

Choose normalization rules

Parameters may change page content. Remove only what the next crawl does not need.

3

Review hosts and exceptions

Invalid entries and suspicious variants stay visible instead of disappearing silently.

URLs by host
HostURLs
Suspicious variants
    Invalid entries
      4

      Export the crawl input

      Lines are best for a simple queue. CSV and JSON retain the review fields.

      Output

      This output has not been crawled. Status, canonical, robots, and content still need a live request.

      Ready

      Use this workflow when…

      • A crawl export contains the same page in several URL forms.
      • You copied URLs from a spreadsheet, sitemap, Markdown file, or issue.
      • You need to separate invalid text before a crawl or import.

      What changes, exactly?

      Every normalization choice is visible. The workflow cannot decide whether two parameter variants contain different content.

      Before
      https://EXAMPLE.com/
https://example.com
example.com/page/#top
example.com/page?utm_source=news
example.com/about/
      Unique URLs
      3
      Duplicates
      2
      Invalid
      0
      After
      https://example.com/
https://example.com/page
https://example.com/about

      A normalized URL is not proof that the page exists, returns 200, redirects correctly, or should be canonical. Those checks require a live request.

      Check these edge cases

      Do not remove information simply because it looks untidy.

      Edge caseInputDecision
      Content parameter?page=2Keep unless proven redundant
      Tracking parameter?utm_source=emailRemove in tracking mode
      Fragment/guide#setupRemove for an HTTP crawl target
      Bare domainexample.comAdd HTTPS only when intended

      Where the result can go

      Download the result first. It stays useful without another product.

      1Keep the file

      Copy or download the cleaned output and use it in any compatible workflow.

      2Check one URL live

      AnalyseSpider requests one public URL and reports the response, directives, links, and page signals it receives.

      Open AnalyseSpiderDevAwesome and AnalyseSpider are operated by Matthias Ramahi. This is a related workflow handoff, not an independent recommendation.

      Where the result can go

      Download the result first. It stays useful without another product.

      1Keep the file

      Copy or download the cleaned output and use it in any compatible workflow.

      2Run the complete list as a site audit

      Contextter Site Audit is the next step when the reviewed list belongs to a larger crawl, prioritization, and monitoring workflow.

      Open Contextter Site AuditDevAwesome and Contextter are operated by Matthias Ramahi. This is a related workflow handoff, not an independent recommendation.

      Use the cleaned list as the input for the next distinct check.

      Crawl-list guideUnderstand safe normalization, ambiguous variants, and the live-check boundary.Robots.txt testerTest a crawler and path against pasted rules.Debug indexabilityCombine the signals for one page.