nx_block_density.nx

nx_block_density.nx -- BOILERPLATE BLOCK SCORING for HTML: the /compare/webscraping R6 contract (symbol bd_fit_text). Kohlschuetter, Fankhauser, Nejdl (WSDM 2010) shallow text features -- per-block TEXT DENSITY and LINK DENSITY -- computed in integer permil over the flat tag stream, so an indexer stores the article and not the nav, footer and sidebar around it. WHY THIS SHAPE (2026-08-24): nx_html_to_text is a tag-stripping renderer with 51 importers and a frozen byte contract; the /compare row that credited it with a "block-aware boilerplate strip" overclaimed -- no density scoring existed anywhere in the tree (link_density / text_density: 0 matches, corpus complete). This lib is the missing ruler. It COMPOSES nx_html_to_text for the final text so the estate still has exactly ONE HTML-to-text renderer: fit = score blocks -> rebuild HTML from the kept spans -> nx_html_to_text. No second entity table, no second whitespace policy, no second tag list. A BLOCK is the span between two block-level tag boundaries (open or close of p, div, li, h1-h6, section, article, aside, nav, header, footer, main, table, tr, td, th, blockquote, pre, ul, ol, dl, dt, dd, figure, figcaption, form, hr, body). script / style / noscript / template / svg bodies are never text and are removed from every block before counting; sup / sub text (citation brackets) is not counted as words. Per block: text bytes (whitespace runs collapsed), markup bytes, words, link words (words that begin inside an <a>). link_density = link_words * 1000 / words, in permil. THE DECISION IS THE PUBLISHED ONE, NOT AN INVENTED SCORE. Calibration on four real pages (2026-08-24, two Wikipedia articles, a Python docs page, an LWN article) refuted a weighted text-density score: Wikipedia prose measures ~7 pct text density and ~60 pct character link density because every link carries ~100 bytes of markup and citation brackets are anchor text -- the score dropped 86 real paragraphs on one page. Kohlschuetter's densitometric classifier uses WORD COUNTS of the previous, current and next block plus word-based link density; the thresholds below are the learned values his reference implementation (boilerpipe NumWordsRulesClassifier) ships, cited on the /compare row: if curr.ld <= ld_curr_max: if prev.ld <= ld_prev_max: if curr.words <= words_curr_short: if next.words <= words_next_short: (prev.words <= words_prev_short ? BOILERPLATE : CONTENT) else CONTENT else CONTENT else: if curr.words <= words_curr_long: (next.words <= words_next_long ? BOILERPLATE : CONTENT) else CONTENT else BOILERPLATE prev/next are the nearest blocks WITH words (empty wrapper and close-tag blocks are structural and kept so a page with no boilerplate rebuilds byte-identical). Blocks inside a semantic element the conf drops (nav / footer / aside by default) are dropped by construction. If nothing survives, or the page has one block, the FULL text is returned and bd_last_fallback_g says so -- a filter that can return an empty page is worse than no filter.