Indexability is a chain of conditions, not one tag.
A page can return 200 and still be ineligible for indexing. Check the response, crawl access, index directives, and canonical target separately before interpreting the combined result.
Response → crawl → directives → canonical.
Start with what the crawler receives. A non-success response can end the diagnosis early. A robots.txt block can hide page-level directives from a crawler. A crawlable 200 page can still carry noindex or point its canonical elsewhere.
- 01 / Response
Record the final status.
Follow redirects and record the final URL and status. A 404 is not rescued by a canonical tag in an old HTML body. A soft 404 may return 200 but still needs content-level review, which this signal-only debugger cannot perform.
- 02 / Crawl access
Check robots.txt for the crawler.
Test the exact user-agent and URL path. Robots.txt controls crawling, not canonical selection. If a crawler cannot fetch a page, it may not see a page-level
noindexrule at all.Conflict to avoid: blocking a URL in robots.txt while relying on a meta or header
noindexon that same URL. - 03 / Directives
Read HTML and headers.
Check the robots meta tag and every
X-Robots-Tagresponse header. A restrictive rule can come from either location. Header directives are especially important for PDFs and other non-HTML resources. - 04 / Canonical
Compare the declared target.
A canonical pointing elsewhere is a preference signal that this URL should not be the representative version. Confirm that the target is reachable, indexable, relevant, and consistent with redirects, sitemap entries, and internal links. Search engines can choose a different canonical.
- 05 / Confirmation
Separate eligibility from indexing.
Passing these technical conditions means the supplied signals do not block eligibility. It does not promise selection, indexing, ranking, or a particular snippet. Confirm the fetched and selected state with a live crawl and the search engine's inspection tooling.
How the signals are defined.
- Google robots meta and X-Robots-Tag specification
Defines page- and header-level indexing controls and explains why the page must be crawlable for those rules to be read.
- Google canonical URL methods
Explains redirects, canonical annotations, sitemap inclusion, and the difference between a signal and a rule.
- DevAwesome workbench core contract
Exercises the production diagnosis for a crawlable 200 response carrying
noindex.
A crawlable response is not the same as an indexable page.
The fixture combines a 200 response, a self-canonical, a meta noindex, an indexing HTTP header, and a robots.txt block for the tested path. The parser extracts each signal; the diagnosis reports the restrictive directive and the crawl conflict separately.
- Expected verdict
- Blocked by a noindex directive.
- Conflict
- Googlebot is blocked from fetching the path carrying that directive.
- Boundary
- No live request and no claim about Google's selected state.
The debugger uses supplied observations.
It does not request the page, render JavaScript, inspect content quality, or query a search index.
Save a diagnosis, not a promise.
The workflow can export the extracted signals, verdict, blockers, and next checks as JSON. Attach that report to the live evidence that produced the inputs.