Nishi Family › Compare › Web Scraping and Extraction

Nishi Compare · capabilities, evidence and delivery

Web Scraping and Extraction

Compare capabilities and delivery evidence. Source discovery, qualification, deployment and accepted outcomes are separate; an open contract names work still to verify.

Nishi vs Firecrawl and Apify and Bright Data and Zyte

Overview

The purpose, declared position and evidence coverage of this domain. Source presence and completed acceptance are different measures.

Where we are. Measured 2026-08-19. Sovereign TLS fetch, range fetch, a persisted frontier crawler, extraction, media harvest, near-duplicate detection, Common Crawl, WARC and CDX and a full-text index are present; the exceeds are sovereignty and zero-cost determinism. Absent: headless rendering, LLM extraction, managed scale, egress rotation and bot-wall handling -- the last two bounded by ethics (owned egress only, no CAPTCHA solving).

Where we need to go. Render JavaScript pages, extract structure with every field grounded, schedule on the swarm, and handle egress and bot walls within ethics -- never trading the sovereign, zero-SaaS stack for reach.

The unit. 1 u = one measured session-leg. Calibration from landed rungs: the mangagen panel compositor went from existing substrate to shipped and live-verified in ONE leg (2026-08-13); the citations rung went from 3 to 55 domains in one leg across seven seats (2026-08-18); a greenfield engine with a bite-proven gate has measured 2 to 4 legs. Estimates recalibrate as rungs land and PR7 actuals write back.

Cost to render: 2.5 u. Through M0.

Cost to extract and scale: 5.5 u. Through M1.

Cost to reach within ethics: 8 u. Everything below.

Cost to crawl smart (crawl4ai-class policies, sovereign): 14 u. Through M5: the six 2026-08-24 rungs lifted from the crawl4ai concept source.

Why a Chrome TLS fingerprint needed a Chrome header set: measured 2026-09-03. fab.com answered 403 to the UA-alone profile and 200 (885,467 B, cf-mitigated absent) to the full profile OVER THE SAME TLS STACK. The estate had already paid for a Chrome-JA3 ClientHello (nx_tls13_chrome_session) and then sent a header set no Chrome has ever sent -- a fingerprint MISMATCH is a stronger signal than either half alone. The TLS half shipped 2026; the header half did not, until R10.

Latest recorded release

No valid dated release entry is recorded for this domain.

Release entries describe recorded changes; they do not establish that every capability passed evaluation.

19 of 23 capability source symbols found|2 of them authored stronger-claims|4 open|coverage 826/1000|adoption 14 full / 5 partial

Evidence profile — what the gaps on this board actually are

Measured by nx_swcompare_evidence, read back by nx_evprofile_lib. Every figure is a count with its denominator — there is deliberately no score, no grade and no percentage anywhere in this band, because a stored scalar is a field a seat can edit and a counted partition is not.

proven able to fail counts the gates that have a RECORDED RED — nx_gate_bite mutated the gate subject, rebuilt it, watched the gate go red, and that record is inside the shared TTL. never bitten is its complement over the same denominator: those gates ran and were green, and nothing has ever shown them able to detect anything, so their green is a statement about this run and not about the gate. green at 0/0 is a separate and much weaker observation — the gate printed GREEN on a zero denominator, so its own tooth counter says it examined nothing. A gate can be green, non-zero, and still never bitten; that is the common case and it is now visible instead of implied.

partition: grounded + unsupported = 19 vs present 19 · named + unnamed + withheld = 4 vs open 4 · both reconcile

liar-kill conj=GPqN · first failing conjunct: quorum

graded document: BUILDROOT tree, 5372 bytes · gates map: PRIMARY · stamped 0d 13h ago · source ../knowledge/status/evstamp_webscraping.verdict

Gap class	What it is, and the work it names
GATE-FAILING	A declared gate ran and was not GREEN. The remedy is to FIX the gate it names. This is deliberately NOT merged with CLAIM-ONLY: one needs a gate written and the other needs a gate repaired, and a single merged number sends a seat at the wrong work.
FLIP-READY	A watch contract symbol is now present in the organ it named: the cell can flip on a pure data edit with no new code. These are the cheapest closures on this board and they belong before any new work.

This band reports the referee counts. The per-axis worklist rows — which axis is unsupported, which watch is ready to flip, which gap is unnamed — are printed by nx_swcompare_evidence webscraping itself and are not carried on the stamp, so this page names the classes and the producer names the rows. That split is stated rather than hidden: a count without a worklist is not actionable, and this band is honest about which half of that it is.

Production map

Follow the dependencies, declared acceptance criteria and recorded priorities. Inspect source binding before treating a rank as executable work.

Ranking source binding: PLAN_MATRIX_BOUND_ONLY. Recorded priorities require current acceptance evidence and resource checks before execution.

Ranking matches the captured plan and matrix only. Latest execution outcome, research freshness, accepted delivery and investment return are unverified.

Recorded priority estimates

Order from nx_compare_rank (nx_dr_ocm: (deficit + cost-of-delay + option + enables) x sponsor x self-sufficiency x momentum / cost). FINISH rows are rungs whose symbol is present but whose organ is short of full adoption: listed before new work by this heuristic. Priority is not measured delivery cost or execution readiness. Stamp: # asof=1790081611 domain=webscraping target_version=0.1 rungs=11 done=6 open=5 finish=2 ranker=nx_dr_ocm

#	Stage	Rung	Priority	Derivation
F	FINISH	Domain map with soft-404 fingerprint (R8) dm_scan dossier	REGISTERED-UNAUTHORISED	registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked)
F	FINISH	Pacing decay (R9) pace_decay dossier	PROMOTED-UNREGISTERED	a real binary nobody can call over MCP: /api/tools/register it; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked)
#1	0.1	Headless rendering (R0) hs_render dossier	360	v=9 m=1 c=25
#2	later	Egress rotation on owned hosts (R3) pp_rotate dossier	900	v=9 m=1 c=10
#3	later	Scheduling at scale on the swarm (R2) ssch_plan dossier	800	v=12 m=1 c=15
#4	later	LLM structured extraction (R1) lx_extract dossier	400	v=6 m=1 c=15
#5	later	Request-identity coherence (client hints + fetch metadata) (R10) hc_put_hints dossier	0	v=0 m=1 c=5

Declared roadmap — contract, acceptance, executor, effort

Rung	Closes with	Definition of done (pre-declared)	Executor	Est.
Headless rendering (R0)	hs_render	The sovereign browser's JS execution renders SPA pages into the crawl; shared contract with the search domain's JS-rendered row; the browser census FETCH-EMPTY records flip	Organ	2.5 u
LLM structured extraction (R1)	lx_extract	Schema-guided JSON extraction on the sovereign seat with every field grounded to a source span; a field without a span is refused, not invented; precision on a banked page set PRE-DECLARED	Local model	1.5 u
Scheduling at scale on the swarm (R2)	ssch_plan	Crawl and extraction jobs planned and leased through the swarm job journal with robots and pace rules as data; a stalled job is re-leased, never lost	Organ	1.5 u
Egress rotation on owned hosts (R3)	pp_rotate	Rotation across sovereign-owned egress endpoints only (never residential botnets); the pace gate stays in the path and the policy is rows	Organ	1 u
Bot-wall classification (R4)	abt_classify	The research-fetch bot-wall witness generalised: Cloudflare challenge, cookie walls, Anubis and go-away recognised from captured specimens and reported as a verdict, never solved by a CAPTCHA farm; robots and terms are respected by construction	Organ	1.5 u
Crawl sufficiency (adaptive stop) (R5)	cs_confidence	Coverage, consistency and saturation in integer permil over the pages fetched so far, weights and stop bars as conf rows, simhash-clustered duplicates discounted; the crawler and the research fetch stop when the bar is met and print which of the three moved; gate bite-proven on a fixture that must NOT stop early and one that must [crawl4ai-adaptive]	Organ	1.5 u
Boilerplate block scoring (R6)	bd_fit_text	Per-block text and link density scored in permil with weights and threshold as conf rows; the crawler indexes the fit text; accept rule: on the banked fixture every article sentence survives and the nav and footer blocks are dropped, and a page with no boilerplate is byte-identical [crawl4ai-fit]	Organ	1 u
Frontier priority and aging (R7)	wc_priority	The pull window is ranked shallow-first with a query-term bonus and rotated by a persisted cursor so every pending row gets a turn; accept rule: on a fixture frontier no row waits more than one full rotation and the ranking is a pure gate-tested function [crawl4ai-readme]	Organ	1 u
Domain map with soft-404 fingerprint (R8)	dm_scan	Sitemap index recursion with gzip, feeds, robots path mining, crt.sh subdomains and Wayback CDX composed into one appended-as-decided census whose partition sums; a soft-404 fingerprint taken from a known-bad path classifies probes; robots and pace stay in the path [crawl4ai-readme]	Organ	2 u
Pacing decay (R9)	pace_decay	A throttled host's backoff halves on each success instead of resetting to base, so one 200 after a 429 cannot re-trigger the limit; the pace gate's teeth state the declared policy	Organ	0.5 u

Milestones

Milestone	Rungs	Cumulative
M0 · Render	R0	2.5 u
M1 · Extract and scale	R1,R2	5.5 u
M2 · Reach within ethics	R3,R4	8 u
M3 · Know when to stop	R5	9.5 u
M4 · Extract and prioritise	R6,R7	11.5 u
M5 · Map and pace	R8,R9	14 u

Rung	Closes with	Definition of done (pre-declared)	Executor	Est.
Request-identity coherence (client hints + fetch metadata) (R10)	hc_put_hints	The sovereign request presents ONE coherent identity: the Chrome UA is accompanied by the sec-ch-ua client hints and the Sec-Fetch-* metadata a real Chrome sends, and its sec-ch-ua version EQUALS its UA version. Identity-gated: HC_UA_BOT emits none of it, so the honest crawler identity stays honest, and a caller that declared its own UA is never contradicted. Done-rule: nx_http_client_gate GREEN with three named neg-controls, bite-proven RED when the call site is removed. This is the COMPLEMENT of R4, not a substitute: R4 RECOGNISES a wall, R10 stops walls that key only on header shape from firing at all -- neither solves a challenge, and Anubis-class proof-of-work stays refused by policy	Organ	0.5 u

Milestones

Milestone	Rungs	Cumulative
M6 · Honest identity coherence	R10	14.5 u

Ladder verdict. NOT DECLARED. This board names no dated best-in-class or frontier target and no rung roles (sotatarget and rungrole rows on its plan); the ranker labels it NO-LADDER until it does, and until then its rungs climb toward a target nobody has written down. Bars read NO-BAR against the current month 2026-10.

Bar freshness, counted
Class	Count	Note
bars	0	sotabar rows on the plan
fresh	0	the bar month is the current month
attested	0	an older bar with a this-month barscan reporting no newer entry
stale	0	a this-month scan saw a newer month the bar ignores
unattested	0	nobody looked this month
malformed	0	a sotabar row missing a field
duplicate	0	bar ids an earlier row already carries
bars, summed	0	RECONCILED: the parts sum to the total 0
current month	2026-10	the month every bar is judged against

Treatment verdict. NOT DECLARED. No rung on this board declares its three-step treatment (a treat row naming the MCP row, the API route, the primitive it composes, the workflow that composes it and the agent beat that runs it); the ranker labels every rung TREATMENT-UNDECLARED until it does.

Inspect a rung and its prerequisites

Declared nodes 11. Rank input binding: PLAN_MATRIX_BOUND_ONLY. Dependency order is authored. Implementation, acceptance evidence, authority and resource readiness are unverified. No action is recommended or dispatched here.

Plan SHA-256 a9bccada4dfea577d351f8525bdc2d734456b41c234f8894aec993bdffe18ec8. Target rows 0; role rows 0. Existing risks and release worklog retain their own scope; no node completion is inferred.

Runtime observation coverage: 0 of 11 rungs have current, identity-bound execution observations. Observation may report RED. Accepted completion remains unknown; source presence, qualification and deployment do not establish it.

Use Enter or Space on a rung to inspect its contract. Prerequisite links locate another rung in this list; open its summary to inspect it. Estimates are authored effort, not forecasts.

R0 — Headless rendering
Prerequisites: None declared; this does not establish execution eligibility.
Contract: hs_render
Acceptance: The sovereign browser's JS execution renders SPA pages into the crawl; shared contract with the search domain's JS-rendered row; the browser census FETCH-EMPTY records flip
Authored effort: 2.5. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R1 — LLM structured extraction
Prerequisites: None declared; this does not establish execution eligibility.
Contract: lx_extract
Acceptance: Schema-guided JSON extraction on the sovereign seat with every field grounded to a source span; a field without a span is refused, not invented; precision on a banked page set PRE-DECLARED
Authored effort: 1.5. Executor kind: Local model. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R2 — Scheduling at scale on the swarm
Prerequisites: None declared; this does not establish execution eligibility.
Contract: ssch_plan
Acceptance: Crawl and extraction jobs planned and leased through the swarm job journal with robots and pace rules as data; a stalled job is re-leased, never lost
Authored effort: 1.5. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R3 — Egress rotation on owned hosts
Prerequisites: None declared; this does not establish execution eligibility.
Contract: pp_rotate
Acceptance: Rotation across sovereign-owned egress endpoints only (never residential botnets); the pace gate stays in the path and the policy is rows
Authored effort: 1. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R4 — Bot-wall classification
Prerequisites: None declared; this does not establish execution eligibility.
Contract: abt_classify
Acceptance: The research-fetch bot-wall witness generalised: Cloudflare challenge, cookie walls, Anubis and go-away recognised from captured specimens and reported as a verdict, never solved by a CAPTCHA farm; robots and terms are respected by construction
Authored effort: 1.5. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R5 — Crawl sufficiency (adaptive stop)
Prerequisites: None declared; this does not establish execution eligibility.
Contract: cs_confidence
Acceptance: Coverage, consistency and saturation in integer permil over the pages fetched so far, weights and stop bars as conf rows, simhash-clustered duplicates discounted; the crawler and the research fetch stop when the bar is met and print which of the three moved; gate bite-proven on a fixture that must NOT stop early and one that must [@crawl4ai-adaptive]
Authored effort: 1.5. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R6 — Boilerplate block scoring
Prerequisites: None declared; this does not establish execution eligibility.
Contract: bd_fit_text
Acceptance: Per-block text and link density scored in permil with weights and threshold as conf rows; the crawler indexes the fit text; accept rule: on the banked fixture every article sentence survives and the nav and footer blocks are dropped, and a page with no boilerplate is byte-identical [@crawl4ai-fit]
Authored effort: 1. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R7 — Frontier priority and aging
Prerequisites: None declared; this does not establish execution eligibility.
Contract: wc_priority
Acceptance: The pull window is ranked shallow-first with a query-term bonus and rotated by a persisted cursor so every pending row gets a turn; accept rule: on a fixture frontier no row waits more than one full rotation and the ranking is a pure gate-tested function [@crawl4ai-readme]
Authored effort: 1. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R8 — Domain map with soft-404 fingerprint
Prerequisites: None declared; this does not establish execution eligibility.
Contract: dm_scan
Acceptance: Sitemap index recursion with gzip, feeds, robots path mining, crt.sh subdomains and Wayback CDX composed into one appended-as-decided census whose partition sums; a soft-404 fingerprint taken from a known-bad path classifies probes; robots and pace stay in the path [@crawl4ai-readme]
Authored effort: 2. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R9 — Pacing decay
Prerequisites: None declared; this does not establish execution eligibility.
Contract: pace_decay
Acceptance: A throttled host's backoff halves on each success instead of resetting to base, so one 200 after a 429 cannot re-trigger the limit; the pace gate's teeth state the declared policy
Authored effort: 0.5. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.
R10 — Request-identity coherence (client hints + fetch metadata)
Prerequisites: None declared; this does not establish execution eligibility.
Contract: hc_put_hints
Acceptance: The sovereign request presents ONE coherent identity: the Chrome UA is accompanied by the sec-ch-ua client hints and the Sec-Fetch-* metadata a real Chrome sends, and its sec-ch-ua version EQUALS its UA version. Identity-gated: HC_UA_BOT emits none of it, so the honest crawler identity stays honest, and a caller that declared its own UA is never contradicted. Done-rule: nx_http_client_gate GREEN with three named neg-controls, bite-proven RED when the call site is removed. This is the COMPLEMENT of R4, not a substitute: R4 RECOGNISES a wall, R10 stops walls that key only on header shape from firing at all -- neither solves a challenge, and Anubis-class proof-of-work stays refused by policy
Authored effort: 0.5. Executor kind: Organ. Responsible, accountable and verifier not established. Inspect retained worklog.
Runtime observation: UNKNOWN; outcome UNKNOWN; observed epoch -1. Declared owner: . Receipt , SHA-256 . Identity-bound observation is not accepted completion. Qualification and deployment remain unverified by this projection.

Learning and practice paths

No structured learning path is declared for this plan. Existing research, roadmap and worklog remain available above.

Rung dossiers — every record that bears on a rung, in one place

Joined by symbol, never typed. Each dossier is assembled on every publish from the rows this board already carries: the plan's rung row, the matrix row that carries its contract symbol (its status re-measured by the one symbol ruler), the gate the organ declares and the verdict the evidence referee recorded on its last run, the reference marks in the rung text, the design rows that name the rung and the work log filed against it. The capability matrix links each cell to its dossier and each dossier links back to its cell and forward to the rungs it stands on; a dependency naming no rung on this plan is written as UNRESOLVED rather than linked. Absences are named: NO-MATRIX-ROW, NO-GATE-DECLARED, UNRECORDED.

Gate verdicts: UNSTAMPED — no per-gate verdict artifact is published for this domain yet (nx_swcompare_evidence webscraping writes it beside its stamp); every gate below reads UNRECORDED until it runs.

Gate declarations: knowledge/organ_gate.conf read from this working directory (1 rows) and from the estate root (196 rows); rows in both counted once: 0; declarations read: 197. Every declaration in either copy counts, so a stub beside the publisher can never hide the root.

R0 Headless rendering · cell WATCHING · gate NO-GATE-DECLARED

Contract	`hs_render` · executor Organ · 2.5 u · stands on nothing
Definition of done	The sovereign browser's JS execution renders SPA pages into the crawl; shared contract with the search domain's JS-rendered row; the browser census FETCH-EMPTY records flip
Matrix cell	JS-rendered / headless scraping · `runtime/nx_headless_scrape.nx` · WATCHING decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_headless_scrape`; declare `nx_headless_scrape TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R1 LLM structured extraction · cell WATCHING · gate NO-GATE-DECLARED

Contract	`lx_extract` · executor Local model · 1.5 u · stands on nothing
Definition of done	Schema-guided JSON extraction on the sovereign seat with every field grounded to a source span; a field without a span is refused, not invented; precision on a banked page set PRE-DECLARED
Matrix cell	LLM structured extraction · `runtime/nx_llm_extract.nx` · WATCHING decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_llm_extract`; declare `nx_llm_extract TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R2 Scheduling at scale on the swarm · cell WATCHING · gate NO-GATE-DECLARED

Contract	`ssch_plan` · executor Organ · 1.5 u · stands on nothing
Definition of done	Crawl and extraction jobs planned and leased through the swarm job journal with robots and pace rules as data; a stalled job is re-leased, never lost
Matrix cell	Managed scheduling / scale · `runtime/nx_scrape_scheduler.nx` · WATCHING decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_scrape_scheduler`; declare `nx_scrape_scheduler TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R3 Egress rotation on owned hosts · cell WATCHING · gate NO-GATE-DECLARED

Contract	`pp_rotate` · executor Organ · 1 u · stands on nothing
Definition of done	Rotation across sovereign-owned egress endpoints only (never residential botnets); the pace gate stays in the path and the policy is rows
Matrix cell	Proxy rotation / residential IPs · `runtime/nx_proxy_pool.nx` · WATCHING decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_proxy_pool`; declare `nx_proxy_pool TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R4 Bot-wall classification · cell LANDED · gate NO-GATE-DECLARED

Contract	`abt_classify` · executor Organ · 1.5 u · stands on nothing
Definition of done	The research-fetch bot-wall witness generalised: Cloudflare challenge, cookie walls, Anubis and go-away recognised from captured specimens and reported as a verdict, never solved by a CAPTCHA farm; robots and terms are respected by construction
Matrix cell	Anti-bot / CAPTCHA handling · `runtime/nx_antibot.nx` · LANDED decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_antibot`; declare `nx_antibot TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R5 Crawl sufficiency (adaptive stop) · cell LANDED · gate NO-GATE-DECLARED

Contract	`cs_confidence` · executor Organ · 1.5 u · stands on nothing
Definition of done	Coverage, consistency and saturation in integer permil over the pages fetched so far, weights and stop bars as conf rows, simhash-clustered duplicates discounted; the crawler and the research fetch stop when the bar is met and print which of the three moved; gate bite-proven on a fixture that must NOT stop early and one that must [crawl4ai-adaptive]
Matrix cell	Adaptive stop -- crawl sufficiency (coverage, consistency, saturation) · `runtime/nx_crawl_sufficiency.nx` · LANDED decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_crawl_sufficiency`; declare `nx_crawl_sufficiency TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R6 Boilerplate block scoring · cell LANDED · gate NO-GATE-DECLARED

Contract	`bd_fit_text` · executor Organ · 1 u · stands on nothing
Definition of done	Per-block text and link density scored in permil with weights and threshold as conf rows; the crawler indexes the fit text; accept rule: on the banked fixture every article sentence survives and the nav and footer blocks are dropped, and a page with no boilerplate is byte-identical [crawl4ai-fit]
Matrix cell	Boilerplate block scoring (text and link density) · `runtime/nx_block_density.nx` · LANDED decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_block_density`; declare `nx_block_density TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R7 Frontier priority and aging · cell LANDED · gate NO-GATE-DECLARED

Contract	`wc_priority` · executor Organ · 1 u · stands on nothing
Definition of done	The pull window is ranked shallow-first with a query-term bonus and rotated by a persisted cursor so every pending row gets a turn; accept rule: on a fixture frontier no row waits more than one full rotation and the ranking is a pure gate-tested function [crawl4ai-readme]
Matrix cell	Frontier priority and aging (best-first pull, rotating window) · `runtime/_hdl_build/nx_web_crawl_step.nx` · LANDED decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_web_crawl_step`; declare `nx_web_crawl_step TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R8 Domain map with soft-404 fingerprint · cell LANDED · gate NO-GATE-DECLARED

Contract	`dm_scan` · executor Organ · 2 u · stands on nothing
Definition of done	Sitemap index recursion with gzip, feeds, robots path mining, crt.sh subdomains and Wayback CDX composed into one appended-as-decided census whose partition sums; a soft-404 fingerprint taken from a known-bad path classifies probes; robots and pace stay in the path [crawl4ai-readme]
Matrix cell	Domain map (sitemap index recursion, feeds, robots path mining, CT-log subdomains, Wayback CDX) · `runtime/_hdl_build/nx_domain_map.nx` · LANDED decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_domain_map`; declare `nx_domain_map TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R9 Pacing decay · cell LANDED · gate NO-GATE-DECLARED

Contract	`pace_decay` · executor Organ · 0.5 u · stands on nothing
Definition of done	A throttled host's backoff halves on each success instead of resetting to base, so one 200 after a 429 cannot re-trigger the limit; the pace gate's teeth state the declared policy
Matrix cell	Adaptive per-host pacing (Retry-After, robots delay, persisted backoff that halves on success) · `runtime/_hdl_build/nx_crawl_pace.nx` · LANDED decl
Gate	NO-GATE-DECLARED — `knowledge/organ_gate.conf` carries no row for `nx_crawl_pace`; declare `nx_crawl_pace TAB <gate>` and the ship loop proves it and the verdict lands here.
Work log	no log row names this rung yet

R10 Request-identity coherence (client hints + fetch metadata) · cell NO-MATRIX-ROW · gate NO-MATRIX-ROW

Contract	`hc_put_hints` · executor Organ · 0.5 u · stands on nothing
Definition of done	The sovereign request presents ONE coherent identity: the Chrome UA is accompanied by the sec-ch-ua client hints and the Sec-Fetch-* metadata a real Chrome sends, and its sec-ch-ua version EQUALS its UA version. Identity-gated: HC_UA_BOT emits none of it, so the honest crawler identity stays honest, and a caller that declared its own UA is never contradicted. Done-rule: nx_http_client_gate GREEN with three named neg-controls, bite-proven RED when the call site is removed. This is the COMPLEMENT of R4, not a substitute: R4 RECOGNISES a wall, R10 stops walls that key only on header shape from firing at all -- neither solves a challenge, and Anubis-class proof-of-work stays refused by policy
Matrix cell	NO-MATRIX-ROW — no matrix row carries `hc_put_hints`; the ranker reads this rung as unmapped and no cell can flip for it. Add the row and the join completes on the next publish.
Gate	the gate join needs a matrix row: no organ is named for this rung
Work log	no log row names this rung yet

dossiers=11 cells_joined=10 no_matrix_row=1 (sum 11) gates_recorded=0 gates_unrecorded=0 no_gate_declared=10 (sum 10 of cells_joined) evgates=UNSTAMPED

Capability comparisons

Compare the field, search individual capabilities and open their source and adoption evidence. Documented presence does not establish comparative quality.

Position map — centrality and distinctiveness

The four-quadrant map the field uses for brand strategy (Dawar and Bagga, HBR June 2015), re-derived from this matrix on every publish. Centrality is the share of the category's feature mass a player covers, each feature weighted by how many hold it; distinctiveness is the average lead over each rival on the rows the player holds; breadth is the depth-weighted share of the whole matrix (the bubble); depth is how deeply the rows held are held; momentum is the day-over-day move off the spine (green rising, red falling, grey until day two); the dashed path runs first day → previous day → today. Dividers are the category means. Axes are fitted to the field of play, so read the tick numerals, not the frame. Rival marks are documented presence, so a rival's position reads the record, never its quality. The picture grades its own readability below; the table beside it is the same data for a screen reader or a second method.

Rows. 23 feature rows mapped of 23 matrix rows; 0 substrate rows (kernels, bounds and instruments the plan's rungrole marks substrate) are the estate's scaffolding, not category features, and stay off the map

Views. 2D cut: centrality, distinctiveness · 3D cube: centrality, distinctiveness, breadth · axes registered: centrality, distinctiveness, breadth, depth, momentum · a board picks its own in knowledge/compare/webscraping.cdmap (cut|x|y, cube|x|y|z); absent = the HBR defaults

bubble area = breadth · ring = momentum (green rising, red falling, grey until day two) · dashed = category means · axes fitted to the field of play: centrality 780–940, distinctiveness 0–500 of 0–1000 permil (the full range put every player in one corner)
readability of the 2D cut, self-graded by the layout ruler: label overlaps 0 · labels over marks 0 · off-canvas 0 · unresolved labels 0 · mark overlaps 0 (a fact of the data: two players that close are that close) · data spread 609 permil of the plot · quadrant words unseated 0
text contrast, measured with wcag2-ratio (floors from contrast.conf), light theme: labels 16.24 (floor 4.50) · notes 16.24 (floor 4.50) · quadrant words 5.89 (floor 4.50) · tick numerals 5.89 (floor 4.50) · axis titles 5.59 (floor 4.50) · dark theme: labels 13.78 (floor 4.50) · quadrant words 5.55 (floor 4.50) · axis titles 5.97 (floor 4.50) · dark classes under their floor 0 (one figure serves both themes: a dark shortfall is a token to fix, never a class to hide) · classes refused under their floor 0 (a refused class is not drawn; the scale classes are measured, never hidden) · export: SVG PNG (receipt, rendered by the estate's own rasteriser from this page)

axes fitted: centrality 780–940 · distinctiveness 0–500 · breadth 350–600 permil · farther bubbles are painted first, nearer ones over them · the floor shadow is each bubble's (x, y) at height 0
readability of the 3D cube, self-graded by the layout ruler: label overlaps 0 · labels over marks 0 · off-canvas 0 · unresolved labels 0 · mark overlaps 1 (a fact of the data: two players that close are that close) · data spread 457 permil of the plot

Nishi Firecrawl Apify Bright Data Zyte
10 panels over 5 registered axes, every pair once (the lower axis on x, the higher on y) · each panel fitted to its own field of play, first and last tick numerals shown · no labels in a small cut, the legend names the colours; the grade below reports the mark terms only (MM summed over the panels, spread averaged)
readability of the small multiples, self-graded by the layout ruler: label overlaps 0 · labels over marks 0 · off-canvas 0 · unresolved labels 0 · mark overlaps 3 (a fact of the data: two players that close are that close) · data spread 518 permil of the plot

Player	Quadrant	centrality	distinctiveness	breadth	depth	momentum	Rows held	Days on spine	First seen
Nishi	Unconventional	802	372	579	701	0 since 2026-09-23	19	10	2026-09-15
Firecrawl	Peripheral	851	155	449	688	0 since 2026-09-23	15	10	2026-09-15
Apify	Mainstream	901	98	434	625	0 since 2026-09-23	16	10	2026-09-15
Bright Data	Mainstream	864	111	434	666	0 since 2026-09-23	15	10	2026-09-15
Zyte	Mainstream	888	72	405	583	0 since 2026-09-23	16	10	2026-09-15

Day	Matrix rows reviewed	Players recorded
2026-09-15	23	5
2026-09-16	23	5
2026-09-17	23	5
2026-09-18	23	5
2026-09-19	23	5
2026-09-20	23	5
2026-09-21	23	5
2026-09-22	23	5
2026-09-23	23	5
2026-10-08	23	5

How to read this evidence. A Nishi source mark records whether the named symbol is declared in the selected source file. It does not establish working behavior or accepted delivery. A watching tag names an expected source entry; discovering it updates source coverage only. A dark tag means the file exists but the named symbol was not found: inspect the source and contract before assuming the capability is missing or renaming code. Peer codes are authored assessments; unassessed means no comparison is established, not absence. Qualification, deployment identity, runtime observations and acceptance must carry their own evidence. Adoption records the catalog's observed path through source, build, promotion, registration and invocation; libraries use importer evidence and gates use declared execution surfaces. A full catalog path does not establish quality, freshness or a successful user outcome. Partial and unknown states retain their stated next action. Census stamps: importers asof 1790637268, gate census asof 1790104131 (unix seconds; -1 = census absent).

Capability matrix — measured against source

◉ authored stronger claim; source present● present◐ partial○ absent? unassessed; excluded from relative comparison · click any capability for its evidence

Peer cell coverage: authored assessed 92, unassessed 0. Authored assessments are not verified outcomes.

Capability	Nishi	Firecrawl	Apify	Bright Data	Zyte
Sovereign HTTP fetch (TLS) Measured: `nx_https_fetch_follow` exists in `runtime/nx_https_fetch_follow.nx`, verified at emit. All fetch over HTTP; Nishi owns its TLS rival marks uncited — the Yes, Best or Part codes on this row are an observation read with no reference mark behind them Adoption: `LIB-WIRED importers=241 nonval=217` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	●	●	●	●
Byte-range fetch Measured: `nx_https_fetch_range` exists in `runtime/nx_https_fetch_follow.nx`, verified at emit. Partial / resumable fetch rival marks uncited — the Yes, Best or Part codes on this row are an observation read with no reference mark behind them Adoption: `LIB-WIRED importers=241 nonval=217` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	●	●	●	●
Frontier-persisted crawler Measured: `nx_web_crawl_step` exists in `runtime/_hdl_build/nx_web_crawl_step.nx`, verified at emit. Resumable BFS crawl frontier [rfc9309] Adoption: `RUN-BY:fork:nx_hostctl` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	●	●	●	●
HTML content extraction Measured: `nx_html_to_text` exists in `runtime/nx_html_to_text.nx`, verified at emit. Firecrawl leads (clean markdown extract) [kohlschuetter2010] Adoption: `LIB-WIRED importers=57 nonval=42` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; fully adopted on the estate ladderCons behind Firecrawl (leads)	●	◉	●	●	●
Media / asset harvest Measured: `nx_media_harvest` exists in `runtime/nx_media_harvest.nx`, verified at emit. Image / video / stream URL extract rival marks uncited — the Yes, Best or Part codes on this row are an observation read with no reference mark behind them Adoption: `LIB-WIRED importers=6 nonval=5` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	●	●	●	●
Near-duplicate detection Measured: `nx_simhash_is_near_dup` exists in `runtime/nx_simhash.nx`, verified at emit. Content dedup; Bright Data mature [manku2007]. ROW REPOINTED 2026-09-01 and this ONE row was the whole domain RED: it named nx_web_ingest, which is a CALLER, and the bare token simhash, which the OLD substring ruler matched against a COMMENT -- the line carrying it was a prose sentence about near-dup killing, not a definition. The one-symbol ruler correctly refused it. The DEFINING organ is nx_simhash.nx and the function that answers this axis is nx_simhash_is_near_dup; its siblings nx_simhash_fingerprint, nx_simhash_min_hamming and nx_simhash_cluster are the rest of the family. A ROW GROUNDED BY A COMMENT IS A CLAIM THAT READS AS EVIDENCE UNTIL A RULER THAT MEASURES DEFINITIONS ARRIVES Adoption: `LIB-WIRED importers=24 nonval=20` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Zyte; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	◐	◐	●	◐
Common Crawl ingest (index-only) Measured: `nx_cc_ingest` exists in `runtime/_hdl_build/nx_cc_ingest.nx`, verified at emit. CDX + range-fetch, Memex-style [commoncrawl] Adoption: `REGISTERED-UNAUTHORISED` — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). not adopted: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked) Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, ZyteCons not adopted yet: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked)	●	○	○	○	◐
WARC archive parse Measured: `nx_warc_index` exists in `runtime/nx_warc_index.nx`, verified at emit. Web-archive format; live scrapers do not [warc11-iso28500] Adoption: `LIB-GATE-ONLY importers=1` — PARTIAL: imported only by validation organs (gates, tests, benches): wire it into a shipping program. not adopted: imported only by validation organs (gates, tests, benches): wire it into a shipping program Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, ZyteCons not adopted yet: imported only by validation organs (gates, tests, benches): wire it into a shipping program	●	○	○	○	○
CDX capture-index query Measured: `nx_cdx_parse` exists in `runtime/nx_cdx_parse.nx`, verified at emit. Wayback / Common Crawl capture index Adoption: `LIB-WIRED importers=8 nonval=6` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, Zyte; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	○	○	○	○
Full-text index of scraped data Measured: `nx_search_inverted` exists in `runtime/nx_search_inverted.nx`, verified at emit. Nishi indexes; platforms mostly deliver raw rival marks uncited — the Yes, Best or Part codes on this row are an observation read with no reference mark behind them Adoption: `LIB-WIRED importers=38 nonval=32` — fully adopted (top of its ladder). Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, Zyte; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	○	◐	◐	◐
JS-rendered / headless scraping Open — watching `runtime/nx_headless_scrape.nx` : `hs_render`, re-measured on every compare beat. Ship that symbol and this mark flips itself; the `comparewatch-` plane row flips with it. Firecrawl / Apify lead; Nishi static-HTML only rival marks uncited — the Yes, Best or Part codes on this row are an observation read with no reference mark behind them dossierwatching hs_render Pros none measured yetCons behind Firecrawl (leads), Apify (leads), Bright Data, Zyte; open contract; expected source declaration not established	○	◉	◉	●	●
Proxy rotation / residential IPs Open — watching `runtime/nx_proxy_pool.nx` : `pp_rotate`, re-measured on every compare beat. Ship that symbol and this mark flips itself; the `comparewatch-` plane row flips with it. Bright Data leads (residential pool); Nishi none [brightdata-residential] dossierwatching pp_rotate Pros none measured yetCons behind Firecrawl, Apify (leads), Bright Data (leads), Zyte; open contract; expected source declaration not established	○	●	◉	◉	●
Anti-bot / CAPTCHA handling Measured: `abt_classify` exists in `runtime/nx_antibot.nx`, verified at emit. Rivals bypass or solve; Nishi CLASSIFIES from specimen rows (tier, vendor, marker) and names the vendor, never solves [crawl4ai-readme] Adoption: `LIB-WIRED importers=3 nonval=1` — fully adopted (top of its ladder). dossier Pros Implementing source symbol found; runtime acceptance separate; fully adopted on the estate ladderCons behind Firecrawl (leads), Bright Data (leads)	●	◉	●	◉	●
LLM structured extraction Open — watching `runtime/nx_llm_extract.nx` : `lx_extract`, re-measured on every compare beat. Ship that symbol and this mark flips itself; the `comparewatch-` plane row flips with it. Firecrawl leads (schema / JSON extract); Nishi none [firecrawl-docs] dossierwatching lx_extract Pros none measured yetCons behind Firecrawl (leads), Apify, Bright Data, Zyte; open contract; expected source declaration not established	○	◉	●	●	●
Managed scheduling / scale Open — watching `runtime/nx_scrape_scheduler.nx` : `ssch_plan`, re-measured on every compare beat. Ship that symbol and this mark flips itself; the `comparewatch-` plane row flips with it. Apify actors + cloud scale; Nishi operator-stepped [apify-actors] dossierwatching ssch_plan Pros none measured yetCons behind Firecrawl, Apify (leads), Bright Data (leads), Zyte (leads); open contract; expected source declaration not established	○	●	◉	◉	◉
Adaptive stop -- crawl sufficiency (coverage, consistency, saturation) Measured: `cs_confidence` exists in `runtime/nx_crawl_sufficiency.nx`, verified at emit. Stop when enough is known: integer coverage + on-topic consistency + new-term saturation, with simhash-discounted redundancy so copies cannot inflate coverage; no rival column's mirrored docs mention it (grep 2026-08-24) [crawl4ai-adaptive] Adoption: `LIB-WIRED importers=3 nonval=2` — fully adopted (top of its ladder). dossier Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, Zyte; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	○	○	○	○
Boilerplate block scoring (text and link density) Measured: `bd_fit_text` exists in `runtime/nx_block_density.nx`, verified at emit. Kohlschuetter block features as permil scores with weights and threshold as conf rows, feeding the index a fit text instead of nav and footer [kohlschuetter2010] [crawl4ai-fit] Adoption: `LIB-WIRED importers=6 nonval=5` — fully adopted (top of its ladder). dossier Pros Implementing source symbol found; runtime acceptance separate; fully adopted on the estate ladderCons behind Firecrawl (leads)	●	◉	●	●	●
Frontier priority and aging (best-first pull, rotating window) Measured: `wc_priority` exists in `runtime/_hdl_build/nx_web_crawl_step.nx`, verified at emit. Shallow-first and query-term ranking of the pull window plus a persisted rotation cursor so no pending row starves behind dead-host rows (measured 93 pct of the window); rival codes = the depth and queue options in their crawl APIs, not re-read this pass [crawl4ai-readme] Adoption: `RUN-BY:fork:nx_hostctl` — fully adopted (top of its ladder). dossier Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, Zyte; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	●	◐	◐	◐	◐
Domain map (sitemap index recursion, feeds, robots path mining, CT-log subdomains, Wayback CDX) Measured: `dm_scan` exists in `runtime/_hdl_build/nx_domain_map.nx`, verified at emit. Every URL under a domain before crawling it, composed from the sitemap, feed, robots and CDX incumbents; Firecrawl and Apify document a map or sitemap step, coded Part pending a mirror read [crawl4ai-readme] Adoption: `REGISTERED-UNAUTHORISED` — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). dossiernot adopted: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked) Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, ZyteCons not adopted yet: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked)	●	◐	◐	○	○
Soft-404 fingerprint Measured: `dm_soft404` exists in `runtime/_hdl_build/nx_domain_map.nx`, verified at emit. A known-bad path's status, length and title fingerprint so a 200 that is really not-found never poisons discovery or host health [crawl4ai-readme] Adoption: `REGISTERED-UNAUTHORISED` — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). not adopted: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked) Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, ZyteCons not adopted yet: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked)	●	○	○	○	○
Adaptive per-host pacing (Retry-After, robots delay, persisted backoff that halves on success) Measured: `pace_decay` exists in `runtime/_hdl_build/nx_crawl_pace.nx`, verified at emit. Rate limiting persists across processes and honours the host's own numbers; rivals pace inside their managed cloud, opaque to the caller [rfc9309] Adoption: `PROMOTED-UNREGISTERED` — PARTIAL: a real binary nobody can call over MCP: /api/tools/register it; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). dossiernot adopted: a real binary nobody can call over MCP: /api/tools/register it; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked) Pros Implementing source symbol found; runtime acceptance separate; ahead of Firecrawl, Apify, Bright Data, ZyteCons not adopted yet: a real binary nobody can call over MCP: /api/tools/register it; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked)	●	◐	◐	◐	◐
Sovereign -- own language, compiler, TLS, index (zero SaaS dep) Source present; authored stronger claim: `ss_fold_cp` in `runtime/nx_seg_store.nx`, verified at emit. All four are paid SaaS on big-cloud; Nishi owns the stack [zyte-api] Adoption: `LIB-WIRED importers=498 nonval=372` — fully adopted (top of its ladder). Pros Source symbol found; stronger capability claim is authored; ahead of Firecrawl, Apify, Bright Data, Zyte; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	◉	○	○	○	○
Runs on your hardware, zero cost, integer-deterministic Source present; authored stronger claim: `idf_q10` in `runtime/nx_intlog.nx`, verified at emit. No per-request billing, no vendor lock, reproducible Adoption: `LIB-WIRED importers=9 nonval=7` — fully adopted (top of its ladder). Pros Source symbol found; stronger capability claim is authored; ahead of Firecrawl, Apify, Bright Data, Zyte; fully adopted on the estate ladderCons none on the measured axes (rival marks are documented presence, not depth)	◉	○	○	○	○

Rival-claim provenance. Every Yes, Best or Part mark in a rival column is a claim about someone else's product. A cited row carries a reference mark that resolves to a pinned mirror (the refs gate measures that); an uncited row is an observation read and is badged in its evidence until a reference lands. The badge is the receipt, never the proof: a mark proves a mirror exists, not that the mirror supports the code. The rows are counted below.

Rival claims, counted
Class	Count	Note
rows with rival marks	17	rows carrying a Yes, Best or Part in a rival column
cited	12	a reference mark resolves to a pinned mirror
uncited	5	an observation read, badged until a reference lands
rows, summed	17	RECONCILED: the parts sum to the total 17

Delivery and evidence

Inspect risks, technical debt, rendered observations, experiments and references. Read scope and limitations alongside every result.

On these two registers. Rows are declared in the domain's plan file and carry the debt id, which is the join key back to the sovereign debt plane — that plane, not this page, is the authority on state. Reconciling them automatically (the regen reading the plane and refreshing these rows) is a named, owed rung; until it lands, treat an id here as a pointer to look up, not a status to trust.

Honest verdict. The coverage above records source-symbol presence. Stronger comparisons are authored claims, not measured runtime superiority or accepted outcomes. A watch contract names an expected source symbol; finding it changes source discovery only. Qualification, deployment and runtime acceptance require their own evidence.

Person · product · place — not yet measured for this domain

Every compare carries this layer. Declare knowledge/compare/webscraping.ppp (rows surface|nishi or c1..c4|label|url|connect naming OUR live surface and each rival's front door), run nx_ppp_probe domain webscraping, and this section fills itself on the next beat: the same ruler on both sides — privacy and CX (third-party hosts, tracker classes, cookies, security headers), design and longevity (design hygiene, computed WCAG contrast, render-blocking resources, unsized media, script weight, theme and motion queries), findability (landmarks, skip link, on-site search, breadcrumb, headings, internal links).

The field — discovered, not chosen

Rows written by nx_field_discover from webscraping.seeds: the industry's own lists (Wikipedia wikitext, GitHub topics, awesome lists) read mechanically, every candidate counted across seeds. The matrix columns above are a SEAT'S pick; this band is the population they were picked from, and the stats line measures one against the other. A rival here is a lead, never a verdict — it earns a column when its capabilities are read and pinned.

Discovery receipt. One row per counter the run wrote, in the receipt's own order; the rival rows follow in their own table.

Field discovery, counted
Class	Count	Note
seeds	4
scanned	4
fetched	4
reused	0
failed	0
named	4
candidates	4
mentions	4
capped	0

Rank	Rival	Seeds	Mentions	First seed	Kind	Link
1	Firecrawl	1	1	col0-firecrawl	named	https://docs.firecrawl.dev/introduction
2	Apify	1	1	col1-apify	named	https://docs.apify.com/platform/actors
3	Bright Data	1	1	col2-bright	named	https://docs.brightdata.com/proxy-networks/residential/introduction
4	Zyte	1	1	col3-zyte	named	https://docs.zyte.com/zyte-api/get-started.html

Exclusive and shared capabilities

One capability is graded on one board; every other board references it. The fleet duplicate census (nx_rivalcensus dupes, on the fleetdupes clock row) reads every board's matrix and names each capability graded as a row on two or more boards, with a home derived from evidence: rung when exactly one board's plan contracts the symbol (that board is starred), gates when no plan contracts it and exactly one board's executed gate map runs the organ's gate (that board is plussed), ambiguous when several boards carry the deciding evidence (a decision for their seats), and none when no evidence decides it, with the reason beside it: unmapped (no board of the duplicate declares a gates map; declare the maps) or unproven (a map exists and runs none of the organ's gates; gate the organ on its proving board, or declare it a shared library by policy). A home elsewhere means this board should carry an @xrow reference to the home board's row rather than a graded row of its own; a home here means the other boards should. The table is this board's slice of one estate-wide census: the shape is shared, the rows are the board's.

This board grades 23 capabilities: 14 exclusive to it and 9 shared with other boards. Shared rows on this page: 9 = home here 0 + home elsewhere 1 + ambiguous 2 + undecided 6 + refused 0 (partition 9); undecided = unmapped 0 + unproven 6 + unevidenced 0; homonyms 0; since the previous census: new 0, resolved 0. Census FRESH age_s=6159.

Capability	Organ	Boards grading it	Home	Rule	Evidence	Standing on this board
idf_q10	`nx_intlog.nx`	search+, webscraping, deepresearch*+, researchreach+	deepresearch	rung	plan	home elsewhere adopt: `@xrow deepresearch\|idf_q10\|home by rung (fleet duplicate census)`
nx_search_inverted	`nx_search_inverted.nx`	search, webscraping	—	none	unproven	undecided
nx_https_fetch_follow	`nx_https_fetch_follow.nx`	search, webscraping	—	none	unproven	undecided
nx_html_to_text	`nx_html_to_text.nx`	search, webscraping	—	none	unproven	undecided
nx_cc_ingest	`nx_cc_ingest.nx`	search, webscraping	—	none	unproven	undecided
hs_render	`nx_headless_scrape.nx`	search, webscraping	—	ambiguous	plan	ambiguous
ss_fold_cp	`nx_seg_store.nx`	search, webscraping, deepresearch, automation	—	none	unproven	undecided
nx_media_harvest	`nx_media_harvest.nx`	webscraping, mediaingest	—	none	unproven	undecided
pp_rotate	`nx_proxy_pool.nx`	webscraping, mediaingest	—	ambiguous	plan	ambiguous

Gaps from the record — what the estate does that no board carries

The record census (nx_goalmap record) reads the invoked-tool population and every plan queue row and files each organ or directive that NO matrix, plan or gates row names. A row here is a callout the boards missed: adjudicate it onto a board or declare it infrastructure. Census state STALE (age 1395264 s), sources read 4 of 7 declared — a BLIND census is a FLOOR: unread sources can only add rows.

kind	name	board	source	evidence

rows shown 0|this board's directives 0|estate-wide un-boarded organs 452 (listed in full on /compare/ecosystem)|census rows 639|malformed 0 (counted, never rendered)

References

Beyond a link list. Every reference below resolves twice — the publisher's copy and, where banked, the estate's own non-rottable library mirror with a content pin — and carries its evidence class plus the exact claim on this page it grounds. Keyed marks like [key] in the matrix notes jump here. A dash means honestly absent, never assumed.

[firecrawl-docs] Firecrawl. Introduction -- search the web, scrape any page, and interact with it through one API (scrape to clean markdown, crawl, extract with a schema). Vendor documentation. publisher · read in our library knowledge/fetched/cmp_webscraping_firecrawl.html · pin h5080b1c4b7fb8810091ce9bc9168bc551eec2c13e5aff79422f59d0e417f574f · accessed 2026-08-18 · vendor-docGrounds: The Firecrawl column: Best on "HTML content extraction" (clean markdown extract), "JS-rendered / headless scraping" and "LLM structured extraction" (schema / JSON extract) -- these are the product's documented core verbs, which is why the notes name Firecrawl as leader on those three rows.
[apify-actors] Apify. Actors -- develop, run and share serverless cloud programs; build web scraping and automation tools and publish them on the Apify platform. Vendor documentation. publisher · read in our library knowledge/fetched/cmp_webscraping_apify.html · pin hd14dbeba5d90eb7996ee9e02a18c82e5b13765005e6b0450e7f4f59c1483ef7a · accessed 2026-08-18 · vendor-docGrounds: The Apify column: Best on "Managed scheduling / scale" (the note says Apify actors + cloud scale) and "JS-rendered / headless scraping" -- Actors are the documented unit of managed, scheduled, cloud-run scrapers the row grades against.
[brightdata-residential] Bright Data. Introduction to Residential proxies -- the residential proxy network product documentation. publisher · read in our library knowledge/fetched/cmp_webscraping_brightdata.html · pin h4b7879eb665b048a082d933b78fa064374984e7c433bd7d62d418a346a5fb596 · accessed 2026-08-18 · vendor-docGrounds: The Bright Data column: Best on "Proxy rotation / residential IPs" (the note: Bright Data leads with a residential pool) and "Anti-bot / CAPTCHA handling" -- the residential-network documentation is the capability the row codes, and its absence in Nishi (_ABSENT_ nx_proxy_pool) is a declared gap, not an oversight.
[zyte-api] Zyte. Zyte API: Get Started -- the managed fetch and extraction API (browser rendering and ban handling behind one endpoint). Vendor documentation. publisher · read in our library knowledge/fetched/cmp_webscraping_zyte.html · pin hbbb44a4c57d5c40e27a0f3a77f4ea34c1776f8ee4e86d32cd054b0c4db4b77df · accessed 2026-08-18 · vendor-docGrounds: The Zyte column codes (Yes on fetch, crawl, extraction, headless and anti-bot rows; Best on "Managed scheduling / scale"): Zyte API is the documented managed endpoint those codes describe, and like the other three it is paid SaaS -- the foil for the "Sovereign -- own language, compiler, TLS, index (zero SaaS dep)" exceed row.
[warc11-iso28500] International Internet Preservation Consortium. The WARC Format 1.1 (ISO 28500:2017), the web-archive record format specification. publisher · read in our library knowledge/fetched/cmp_webscraping_warc11.html · pin h0c0bd8cc4b351eddd2b04aade67b2ddf9737961e087371998e01ab8deacde39d · accessed 2026-08-18 · published-standardGrounds: The "WARC archive parse" row (nx_warc_index): the record types, headers and framing nx_warc_index parses are this specification; the row's all-zero rival codes are honest because live scrapers do not read archives, and the row note says so.
[commoncrawl] Common Crawl Foundation. Get Started -- the open web-crawl archive (WARC/WAT/WET) and its CDX capture index. Vendor page. publisher · read in our library knowledge/fetched/cmp_webscraping_commoncrawl.html · pin h7ff73e446d2367d1e2d8478a3b0e0e9205042a3012f319f20c1f3be0c8ef4570 · accessed 2026-08-18 · datasetGrounds: The "Common Crawl ingest (index-only)" and "CDX capture-index query" rows: nx_cc_ingest queries the CDX index and range-fetches WARC records from this archive (the Memex-style index-only ingest the note names) instead of crawling the majors' scale itself.
[manku2007] Manku, G.S., Jain, A., Das Sarma, A. Detecting Near-Duplicates for Web Crawling. WWW 2007. publisher · read in our library knowledge/fetched/cmp_webscraping_manku2007.html · pin hbc8355bd8703aafa5f9bed15c80bb36dea262e1937727ed6e8a96caff12e9179 · accessed 2026-08-18 · published-paperGrounds: The "Near-duplicate detection" row (symbol simhash in nx_web_ingest): 64-bit simhash fingerprints compared by Hamming distance is this paper's method for crawl-time near-duplicate detection; the row's Part codes for the SaaS scrapers reflect that only Bright Data documents mature dedup.
[rfc9309] Koster, M., Illyes, G., Zeller, H., Sassman, L. RFC 9309: Robots Exclusion Protocol. IETF, September 2022. publisher · read in our library knowledge/fetched/cmp_webscraping_rfc9309.html · pin hbbaa280636d3e38f0a1a130a5ce40e9753db1ed18c860945b0bf795ff8dc1a87 · accessed 2026-08-18 · published-standardGrounds: The "Frontier-persisted crawler" and "Sovereign HTTP fetch (TLS)" rows: the robots.txt rules a crawl frontier must honour are this standard, the published bar every column's Yes code (and nx_web_crawl_step) is held to.
[kohlschuetter2010] Kohlschuetter, C., Fankhauser, P., Nejdl, W. Boilerplate Detection using Shallow Text Features. WSDM 2010, ACM. doi:10.1145/1718487.1718542. publisher · read in our library knowledge/fetched/cmp_webscraping_kohlschuetter2010.html · pin h2e81e88b012c07dbe8add5ac6c1b558d2bf8c3b87adc2ff7248bafa74060b42b · accessed 2026-08-18 · published-paperGrounds: The "Boilerplate block scoring" row (nx_block_density): block-level text density and link density as boilerplate features is this paper's result and the published method the row is graded against; the older "HTML content extraction" row is a tag-stripping extractor and cites it only as the bar it does not yet meet.
[crawl4ai-readme] unclecode. Crawl4AI: Open-source LLM Friendly Web Crawler and Scraper -- README (79,320 stars, pushed 2026-08-24). Source repository documentation. publisher · read in our library knowledge/fetched/cmp_webscraping_crawl4ai_readme.md · pin h157214ef86639f896087ca220056ec190f36692a8e17affec9417f638d57e718 · accessed 2026-08-24 · source-readGrounds: The concept source for the "Frontier priority and aging", "Domain map" and "Soft-404 fingerprint" rows: the README documents best-first deep crawling with scorers and filters, DomainMapper's eight discovery sources with soft-404 detection, and a memory-adaptive dispatcher with a fairness timeout -- the policies these rows lift into sovereign organs; its own bytes are Python over Playwright, which is why it is a concept source and not a column.
[crawl4ai-adaptive] unclecode. Crawl4AI documentation: Adaptive Web Crawling -- coverage, consistency and saturation as a three-layer sufficiency score with a confidence stop. publisher · read in our library knowledge/fetched/cmp_webscraping_crawl4ai_adaptive.md · pin h1013777b589438f0f4a73d8053ca895b62373a8b9ac7610d9674b92e6e3cdb90 · accessed 2026-08-24 · source-readGrounds: The "Adaptive stop -- crawl sufficiency" row: the three metrics and the confidence-threshold stop are this document's method; the sovereign row keeps the shape in integer permil and discounts near-duplicate pages, which the source's pairwise-Jaccard consistency rewards.
[crawl4ai-fit] unclecode. Crawl4AI documentation: Fit Markdown with Pruning and BM25 -- per-node text density, link density and tag weights against a fixed or dynamic threshold. publisher · read in our library knowledge/fetched/cmp_webscraping_crawl4ai_fit.md · pin h96a5945694d8a86ac025d2661b748b2c5fc24a7d74d151e66aab8331eef1af09 · accessed 2026-08-24 · source-readGrounds: The "Boilerplate block scoring" row: the pruning filter's text-density and link-density features are the Kohlschuetter shallow features applied per node; the sovereign row keeps the two published features, moves the 0.48 threshold and the weights into conf rows, and grades against a banked fixture instead of a fixed number.

generated by nx_swcompare_matrix (sovereign NishiLang organ) from knowledge/compare/webscraping.matrix · source checks show implementation presence; runtime and user-outcome evidence are reported separately · JavaScript supports page controls