DevAwesomeOpen a tool
← All workflows

Build a clean crawl list

Extract copied URLs, define exactly what belongs in scope, apply pasted robots.txt rules, and export a crawl plan with the winning rule for every blocked URL.

  1. 1Extract URLs
  2. 2Choose rules
  3. 3Define scope
  4. 4Apply robots
  5. 5Review groups
  6. 6Export files
No crawl.The browser parses your text without requesting any URL.
Crawl list workbenchExtract, normalize, group, review, export
1

Add the source

Paste content or load a local file. The workbench extracts URLs from lists, tables, Markdown links, or sitemap XML.

Reading the pasted source…

2

Choose normalization rules

Parameters may change page content. Remove only what the next crawl does not need.

3

Apply robots.txt to the complete list

Paste the rules for the selected host and choose the crawler. Every page URL is evaluated locally; the winning rule stays in the export.

One robots.txt belongs to one host. URLs outside the chosen scope remain separately classified.

Review blocked and uncertain URLs
    4

    Review hosts and exceptions

    Invalid entries and suspicious variants stay visible instead of disappearing silently.

    URLs by host
    HostURLs
    Suspicious variants
      Excluded by scope
        Invalid entries
          5

          Export the crawl input and plan

          The crawl list contains targets. The crawl-plan report retains allowed, blocked, excluded, invalid, resource, and review groups.

          Output

          robots.txt is evaluated against pasted rules. No URL is fetched, so status, canonical, rendered directives, and current server rules still need a live request.

          Ready

          Crawl-list recipeSave or reload the rules. Input and output are never included.
          Recipe preflight

          Review the saved mappings

            Crawl-list run manifestDownload settings, counts, and SHA-256 receipts for this run. Raw input and output stay out of the file.
            Crawl-plan fixture

            One allowed page, one block, one resource, one invalid value.

            The synthetic fixture shows how the same input is separated into operational groups and preserves the winning robots.txt rule.

            Download URL inputDownload robots.txtDownload expected plan

            No URL was requested. The report is valid only for the pasted rule set and selected user-agent.

            Use this workflow when…

            • A crawl export contains the same page in several URL forms.
            • You copied URLs from a spreadsheet, sitemap, sitemap index, Markdown file, or issue.
            • You need to restrict a crawl by host, subdomain, protocol, path pattern, or resource type.
            • You need an export that explains why each rejected URL was excluded.

            What changes, exactly?

            Every normalization choice is visible. The workflow cannot decide whether two parameter variants contain different content.

            Before
            https://EXAMPLE.com/
https://example.com
example.com/page/#top
example.com/page?utm_source=news
example.com/about/
            Unique URLs
            3
            Duplicates
            2
            Invalid
            0
            After
            https://example.com/
https://example.com/page
https://example.com/about

            A normalized URL is not proof that the page exists, returns 200, redirects correctly, or should be canonical. Those checks require a live request.

            Check these edge cases

            Do not remove information simply because it looks untidy.

            Edge caseInputDecision
            Content parameter?page=2Keep unless proven redundant
            Tracking parameter?utm_source=emailRemove in tracking mode
            Fragment/guide#setupRemove for an HTTP crawl target
            Bare domainexample.comAdd HTTPS only when intended
            Sitemap indexsitemapindex → child sitemapTreat as discovery input, not a page target
            Subdomain boundaryshop.example.comInclude only when scope allows it

            Where the result can go

            Download the result first. It stays useful without another product.

            1Keep the file

            Copy or download the cleaned output and use it in any compatible workflow.

            2Check one URL live

            AnalyseSpider requests one public URL and reports the response, directives, links, and page signals it receives.

            Open AnalyseSpiderDevAwesome and AnalyseSpider are operated by Matthias Ramahi. This is a related workflow handoff, not an independent recommendation.

            Where the result can go

            Download the result first. It stays useful without another product.

            1Keep the file

            Copy or download the cleaned output and use it in any compatible workflow.

            2Run the complete list as a site audit

            Crawl Foundry Site Audit is the next step when the reviewed list belongs to a larger crawl, prioritization, and monitoring workflow.

            Open Crawl Foundry Site AuditDevAwesome and Crawl Foundry are operated by Matthias Ramahi. This is a related workflow handoff, not an independent recommendation.

            Use the cleaned list as the input for the next distinct check.

            Crawl-list guideUnderstand safe normalization, ambiguous variants, and the live-check boundary.Robots.txt testerTest a crawler and path against pasted rules.Debug indexabilityCombine the signals for one page.