code wiki / (root) / nx_html_tokenizer.nx

nx_html_tokenizer.nx

buildroot/runtime/nx_html_tokenizer.nx

28417 B726 linesdepth 0pulls 0 transitivereach 267 importersview sourcekind librarytopic html
docsdependenciesstructsconstsfunctions

about

nx_html_tokenizer.nx -- HTML tokenizer for the Nishi browser. Phase 1 of docs/NISHI_BROWSER_ROADMAP.md. Recognises: - <!DOCTYPE html> (and any doctype line; we don't parse its body) - <tag> start tag - </tag> end tag - <tag /> self-closing - <tag attr="v"> start tag with attributes - <!-- ... --> comment - text anything outside tags Returns a stream of tokens via a caller-iterated cursor. No allocations beyond the caller's buffer. Sealed token kinds for the next layer (HTML tree builder, Phase 1b). What it does NOT handle yet (named L1b improvements): - HTML5 raw-text / RCDATA tags (script / style / textarea) - foreign content (SVG, MathML namespace switches) - character entity references beyond the basic 5 (amp/lt/gt/quot/apos) - CDATA sections - explicit attribute-value escaping rules around > inside attrs Subset is sufficient for andelinwest.com + ide.html + most modern hand-written HTML. Per cardinal feedback-honest-perf-verdict: we list exactly what's missing. genealogy_id: whatwg_html_tokenizer_2024 + html5_spec_chapter_12 + webwhatwg_html_tokenizer_state_machine lineage_id: nishi_browser_html_tokenizer_q10 nx_safety_envelope: intended_use: "HTML5 tokenizer -- substrate browser front- end + a11y-grader input + general HTML parse" sil_target: SIL2 (HTML parsing; XSS vectors gate on tokenizer correctness) asil_target: QM dal_target: DAL C evidence: [WHATWG_HTML5_canonical_basis, sealed_token_kind_enum,

dependencies 0 imports · 37 importers

nx_html_tokenizer.nx nx_align_gate.nx nx_browse.nx nx_browser.nx nx_browser_demo_html.nx nx_browser_demo_html_nishi.nx nx_browser_demo_styled.nx nx_browser_forms.nx nx_browser_render.nx nx_cdata_probe.nx nx_css_boxmap_test.nx

diagram shows first 10 each side; +0 more imports, +27 more importers in the complete lists below.

imports: none

imported by: nx_align_gate.nxnx_browse.nxnx_browser.nxnx_browser_demo_html.nxnx_browser_demo_html_nishi.nxnx_browser_demo_styled.nxnx_browser_forms.nxnx_browser_render.nxnx_cdata_probe.nxnx_css_boxmap_test.nxnx_css_cascade_test.nxnx_css_extract_test.nxnx_dom_query.nxnx_domtree.nxnx_flexgrow_gate.nxnx_flexgrow_probe.nxnx_float_gate.nxnx_float_test.nxnx_grid_gate.nxnx_grid_test.nxnx_grid_track_gate.nxnx_html5_raw_text_test.nxnx_html_extract_imgs.nxnx_html_extract_links.nxnx_js_eval.nxnx_justify_gate.nxnx_layout_debug.nxnx_layout_from_dom.nxnx_layout_from_dom_test.nxnx_negmargin_gate.nxnx_real_page_gate.nxnx_render_html.nxnx_render_wiki_styled_window.nxnx_table_cell_gate.nxnx_tok_attr_probe.nxnx_tok_debug.nxnx_wiki_boxdump.nx

structs

70struct HtmlToken
81struct HtmlCursor

consts

58const NX_HTML_TOK_UNKNOWN: i64 = 0
59const NX_HTML_TOK_DOCTYPE: i64 = 1
60const NX_HTML_TOK_START_TAG: i64 = 2
61const NX_HTML_TOK_END_TAG: i64 = 3
62const NX_HTML_TOK_SELF_CLOSING: i64 = 4
63const NX_HTML_TOK_TEXT: i64 = 5
64const NX_HTML_TOK_COMMENT: i64 = 6
65const NX_HTML_TOK_EOF: i64 = 7
66const NX_HTML_TOK_N: i64 = 8

functions

87func nx_html_cursor_init(c: *HtmlCursor, src: *u8, src_len: i64) -> i64
called by 23: mainbr_extractbf_parsebr_extractbm_extractcc_extract+17
95func _lc(b: i64) -> i64
called by 7: _is_httpnish_fetch_render_is_void_elementnx_html_name_eqnx_html_is_raw_text_tagnx_html_consume_raw_text+1
105func _is_ws(b: i64) -> i64
called by 2: _skip_wsnx_html_consume_raw_text
115func _is_name(b: i64) -> i64
called by 1: _scan_name
126func _skip_ws(src: *u8, src_len: i64, p: i64) -> i64
calls 1: _is_ws
136func _scan_name(src: *u8, src_len: i64, p: i64) -> i64
called by 1: nx_html_next_token calls 1: _is_name
151func _is_void_element(src: *u8, name_off: i64, name_len: i64) -> i64
called by 1: nx_html_next_token calls 1: _lc
208func nx_html_next_token(c: *HtmlCursor, out: *HtmlToken) -> i64
called by 23: mainbr_extractbf_parsebr_extractbm_extractcc_extract+17 calls 2: _scan_name_is_void_element
396func nx_html_name_eq(src: *u8, name_off: i64, name_len: i64,
calls 1: _lc
408func nx_html_tok_kind_is_valid(k: i64) -> i64
432func nx_html_is_raw_text_tag(src: *u8, name_off: i64, name_len: i64) -> i64
called by 6: br_extractmainnx_html_extract_imgsnx_html_extract_linksnx_layout_from_domrh_extract_styles calls 1: _lc
484func nx_html_consume_raw_text(c: *HtmlCursor, tag_name: *u8, tag_name_len: i64,
called by 6: br_extractmainnx_html_extract_imgsnx_html_extract_linksnx_layout_from_domrh_extract_styles calls 2: _lc_is_ws
553func _entity_emit_u32(cp_in: i64, dst: *u8, dst_pos: i64, dst_cap: i64) -> i64
called by 1: nx_html_entity_decode
586func _entity_name_eq(src: *u8, sp: i64, src_len: i64, lit: *u8, nlen: i64) -> i64
called by 1: _lookup_named_entity calls 1: _lc
597func _hex_val(b: i64) -> i64
called by 1: _parse_numeric_entity
605func _dec_val(b: i64) -> i64
called by 1: _parse_numeric_entity
613func _parse_numeric_entity(src: *u8, body_off: i64, body_len: i64) -> i64
called by 1: nx_html_entity_decode calls 2: _hex_val_dec_val
639func _lookup_named_entity(src: *u8, body_off: i64, body_len: i64) -> i64
called by 1: nx_html_entity_decode calls 1: _entity_name_eq
669func nx_html_entity_decode(src: *u8, src_len: i64, dst: *u8, dst_cap: i64) -> i64
called by 1: main calls 3: _parse_numeric_entity_lookup_named_entity_entity_emit_u32