code wiki / (root) / nx_mvault_fetch.nx

nx_mvault_fetch.nx

buildroot/runtime/nx_mvault_fetch.nx

22746 B501 linesdepth 21pulls 150 transitivereach 2 importersview sourcekind librarytopic mvault
docsdependenciesstructsconstsfunctions

about

nx_mvault_fetch.nx -- album/gallery ingest for the vault (a BunkrDownloader-class capability, sovereign and data-driven). OPERATOR DECISION 2026-07-31: adapters are a DATA TABLE, not code. A new site is an INSERT, not a rebuild. That is rule 11 (policy in data, never buried in code) and it is also what keeps this from becoming the patch cascade that per-site scraper code always becomes -- every host that needs a `if host == "..."` branch is a redeploy, and the branch list is never complete. WHAT AN ADAPTER SAYS: album pages differ only in WHERE the links live. So a row describes that shape -- link_tag/link_attr/link_pfx where the per-item links are on an ALBUM page item_tag/item_attr where the DIRECT media url is on an ITEM page Two-hop by design: album -> item page -> CDN url is the near-universal shape, and a one-hop site is expressed by leaving the item rule empty. ★★★ ORDERING: this writes into the COLLECTION AXIS (nx_mvault_coll) so every fetched file records WHICH ALBUM and WHICH SITE it came from AT INGEST TIME. Ingesting first and grouping later is what produced the 07-30 backfill of 35,615 mis-typed records; provenance captured at ingest is free, recovered later it is a full-corpus pass. This module is the OFFLINE half -- adapter storage + extraction. It is pure and fully gateable with no network, which is deliberate: the parsing rules are where the bugs live, and they should be provable without a live site. license_tier: ORIGINAL

dependencies 7 imports · 2 importers

nx_syscalls.nx nx_canon_cid.nx nx_registry.nx nx_mvault_record.nx nx_mvault_tag.nx nx_mvault_coll.nx nx_https_fetch_lib.nx nx_mvault_fetch.nx nx_mvault.nx nx_mvault_fetch_gate.nx

imports: nx_syscalls.nxnx_canon_cid.nxnx_registry.nxnx_mvault_record.nxnx_mvault_tag.nxnx_mvault_coll.nxnx_https_fetch_lib.nx

imported by: nx_mvault.nxnx_mvault_fetch_gate.nx

structs

none

consts

36const MVF_PFX: *u8 = "knowledge/store/mvfetch-\x00"
37const MVF_KP: *u8 = "ad:\x00"
38const MVF_IDS: *u8 = "ad:ids\x00"
40const MVF_REC_CAP: i64 = 4096
41const MVF_URL_CAP: i64 = 2048
318const MVF_PAGE_CAP: i64 = 8388608
319const MVF_LINKS_CAP: i64 = 1048576
320const MVF_ITEMS_CAP: i64 = 1048576
321const MVF_MAX_ITEMS: i64 = 4096

functions

43func mvf_strlen(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} return n }
called by 9: mvf_host_ofmvf_origin_ofmvf_resolvemvf_extractmvf_adapter_find_pfxmvf_basename+3
44func mvf_cat(dst: *u8, off: i64, s: *u8) -> i64 { var i: i64=0; while s[i]!=(0 as u8){dst[off+i]=s[i]; i=i+1} return off+i }
called by 2: mvf_fetch_album_pfxmain
46func mvf_lower(c: i64) -> i64 { if c >= 65 { if c <= 90 { return c + 32 } } return c }
called by 1: mvf_at_ci
49func mvf_at_ci(buf: *u8, n: i64, at: i64, s: *u8) -> i64
called by 1: mvf_extract calls 1: mvf_lower
60func mvf_has_pfx(buf: *u8, at: i64, len: i64, pfx: *u8) -> i64
called by 1: mvf_extract
72func mvf_host_of(url: *u8, out: *u8) -> i64
called by 1: mvf_fetch_album_pfx calls 1: mvf_strlen
93func mvf_origin_of(url: *u8, out: *u8) -> i64
called by 1: mvf_resolve calls 1: mvf_strlen
114func mvf_resolve(base: *u8, href: *u8, out: *u8, cap: i64) -> i64
called by 2: mvf_fetch_album_pfxmain calls 4: mvf_strlensys_mmapmvf_origin_ofsys_munmap
151func mvf_extract(html: *u8, n: i64, tag: *u8, attr: *u8, pfx: *u8, out: *u8, cap: i64) -> i64
called by 2: mvf_fetch_album_pfxmain calls 3: mvf_at_cimvf_strlenmvf_has_pfx
216func mvf_adapter_rec(out: *u8, host: *u8, link_tag: *u8, link_attr: *u8, link_pfx: *u8,
called by 1: mvf_adapter_put_pfx calls 3: sys_mmapcanon_encodesys_munmap
240func mvf_adapter_put_pfx(prefix: *u8, host: *u8, link_tag: *u8, link_attr: *u8, link_pfx: *u8,
called by 2: mvf_adapter_putmain calls 4: sys_mmapmvf_adapter_recreg_putsys_munmap
253func mvf_adapter_find_pfx(prefix: *u8, host: *u8, ptrout: *i64, lenout: *i64) -> i64
called by 2: mvf_adapter_findmain calls 2: mvf_strlenreg_get
268func mvf_adapter_list_pfx(prefix: *u8, out: *u8, cap: i64) -> i64
called by 1: mvf_adapter_list calls 1: reg_index
278func mvf_basename(url: *u8, out: *u8, cap: i64) -> i64
called by 2: mvf_fetch_album_pfxmain calls 1: mvf_strlen
305func mvf_field(rec: *u8, rl: i64, key: *u8, out: *u8, cap: i64) -> i64
called by 2: mvf_fetch_album_pfxmain calls 3: sys_mmapmv_rec_fieldmvf_strlen
332func mvf_fetch_album_pfx(cpfx: *u8, store: i64, album_url: *u8, dest_dir: *u8,
called by 1: mvf_fetch_album calls 20: sys_mmapmvf_host_ofmvf_adapter_findmvf_fieldhf_fetchhf_body_off+14
496func mvf_adapter_put(host: *u8, link_tag: *u8, link_attr: *u8, link_pfx: *u8, item_tag: *u8, item_attr: *u8, ua: *u8) -> i64
called by 1: do_fetch calls 1: mvf_adapter_put_pfx
499func mvf_adapter_find(host: *u8, ptrout: *i64, lenout: *i64) -> i64 { return mvf_adapter_find_pfx(MVF_PFX, host, ptrout, lenout) }
called by 1: mvf_fetch_album_pfx calls 1: mvf_adapter_find_pfx
500func mvf_adapter_list(out: *u8, cap: i64) -> i64 { return mvf_adapter_list_pfx(MVF_PFX, out, cap) }
called by 1: do_fetch calls 1: mvf_adapter_list_pfx
501func mvf_fetch_album(store: i64, album_url: *u8, dest_dir: *u8, report: *u8, rcap: i64) -> i64 { return mvf_fetch_album_pfx(MVC_PFX, store, album_url, dest_dir, report, rcap) }
called by 1: do_fetch calls 1: mvf_fetch_album_pfx