nx_docstore.nx
buildroot/runtime/nx_docstore.nx
about
nx_docstore.nx -- persistent crawl content store (CAS/DV2.0 foundation).
module: nishi-core.search.docstore
depends: nx_syscalls.nx, nx_str.nx
capability: CORE_IO
wired_status: FULLY_WIRED
WHY: a real crawl must SURVIVE restarts and RESUME -- you cannot re-crawl the
web from scratch each run (and re-fetching hammers hosts -> blacklisting).
Each fetched doc is appended as a length-prefixed record; on restart the
store reloads and the inverted index is rebuilt, so the corpus persists and
the frontier can continue. Record: [u32 url_len][url][u32 text_len][text],
little-endian. The seed of the content-addressed store / DV2.0 ingest.
dependencies 2 imports · 3 importers
imports: nx_syscalls.nxnx_str.nx
imported by: nx_crawl_https.nxnx_crawl_main.nxnx_docstore_test.nx
structs
| none |
consts
| none |
functions
| 18 | func _ds_put_u32(buf: *u8, off: i64, v: i64) -> i64 called by 1: nx_docstore_append |
| 25 | func _ds_get_u32(buf: *u8, off: i64) -> i64 called by 1: nx_docstore_load |
| 30 | func nx_docstore_reset(path: *u8) -> i64 |
| 38 | func nx_docstore_append(path: *u8, url: *u8, ulen: i64, text: *u8, tlen: i64) -> i64 |
| 59 | func nx_docstore_load(path: *u8, urls: **u8, ulens: *i64, texts: **u8, |