code wiki / (root) / nx_4chan.nx

nx_4chan.nx

buildroot/runtime/nx_4chan.nx

40895 B729 linesdepth 21pulls 148 transitivereach 0 importersview sourcekind tool
docsdependenciesstructsconstsfunctions

about

nx_4chan.nx -- SOVEREIGN 4chan BROWSE / PREVIEW / DOWNLOAD adapter: the first of the media-manager per-source ingesters ported off Python (mm/src/sites/fourchan.py) into the estate. Emits JSON on stdout so the gallery/vault surface consumes it directly; downloads land in a plain directory that `nx_mvault_walk <dir> real 4chan commit` then records into the vault (CID dedup + DSM layout). COMPOSITION, NOT REIMPLEMENTATION: the vault already owns classification/dedup/layout (B1-B10), so this organ deliberately stops at "validated bytes in a directory" and hands off. nx_4chan boards -- board list (+nsfw flag) nx_4chan catalog <board> [max] -- thread previews: subject, comment, counts, THUMB+FULL urls nx_4chan thread <board> <threadno> [max] -- every media item in a thread nx_4chan fetch <board> <threadno> <dir> [max] -- download the thread's media into <dir> nx_4chan selftest -- offline gate: parser teeth incl. neg-controls ★ WHY A BRACE-DEPTH PARSER AND NOT KEY-SCANNING. catalog.json embeds a `last_replies` array whose post objects each carry their OWN "no": field, so scanning for `"no":` reports replies AS THREADS -- and it fails silently, inflating a 15-thread page into 60 "threads". Depth tracking is immune to field order and to nesting. It is also STRING-AWARE: a 4chan comment routinely contains { } and ", and a structural scanner that reads braces inside string data mis-nests the whole document. Conveniently ALL THREE endpoints put their objects of interest at brace-depth 2: catalog.json [ {page {1}, threads:[ {thread {2}, last_replies:[ {reply {3}} ]} ]} ] thread.json { {1} "posts":[ {post {2}} ] } boards.json { {1} "boards":[ {board {2}} ] } so one scanner serves all three and `want_depth` stays a parameter rather than three constants. LEGAL/RATE POSTURE: read-only public JSON API, one polite delay between media fetches. General capability; the operator points it where they like. license_tier: ORIGINAL

dependencies 6 imports · 0 importers

nx_syscalls.nx nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_https_fetch_follow.nx nx_json_lib.nx nx_crawl_pace.nx nx_4chan.nx

imports: nx_syscalls.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_json_lib.nxnx_crawl_pace.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main fc_puts sys_write sys_exit fc_streq fc_selftest sys_mmap nxa_die sys_write ↻ sys_exit ↻ nxa_lock_take nxa_lock_addr sys_write ↻ nxa_lock_give nxa_lock_addr ↻ nxa_report_overrun sys_write ↻ nxa_dump_printable sys_write ↻ nxa_dump_sizes sys_write ↻ fc_apps fc_scan_objects fc_puts ↻ fc_putn sys_mmap ↻ fc_num sys_mmap ↻ sys_write ↻ jx_get_int jx_len sys_mmap ↻ jx_find jx_len ↻ jx_skip_ws fc_slen fc_emit_text fc_slen ↻ fc_media_url fc_apps ↻

structs

none

consts

44const FC_API_HOST: *u8 = "a.4cdn.org"
45const FC_API_HOST_LEN: i64 = 10
46const FC_IMG_HOST: *u8 = "i.4cdn.org"
47const FC_IMG_HOST_LEN: i64 = 10
48const FC_MAGIC_1000000: i64 = 1000000
49const FC_MAGIC_1700000000001: i64 = 1700000000001
51const FC_MAGIC_4194304: i64 = 4194304
52const FC_MAGIC_2048: i64 = 2048
53const FC_MAGIC_65536: i64 = 65536
54const FC_MAGIC_1024: i64 = 1024
55const FC_MAGIC_512: i64 = 512
57const FC_JSON_CAP: i64 = 8388608 // catalog.json for a busy board runs ~1-3MB
58const FC_IMG_CAP: i64 = 33554432 // 32MB: /gif/ webm legitimately exceeds the manga 16MB cap
59const FC_MAX_OBJ: i64 = 4096 // objects enumerated per document (threads/posts/boards)
60const FC_OUT_CAP: i64 = 4194304 // emitted JSON
61const FC_TEXT_CAP: i64 = 8192 // one sub/com field
62const FC_MIN_BYTES: i64 = 512 // below this a "media file" is an error page or a spacer
63const FC_REDIRECTS: i64 = 6
66const FC_FETCH_DELAY_MS: i64 = 350
71const FC_FETCH_RETRIES: i64 = 4

functions

73func fc_puts(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
74func fc_slen(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
75func fc_apps(dst: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { dst[o + i] = s[i]; i = i + 1 } return o + i }
76func fc_num(dst: *u8, o: i64, v: i64) -> i64
88func fc_putn(v: i64) -> i64 { let b: *u8 = sys_mmap(32); let e: i64 = fc_num(b, 0, v); sys_write(1, b, e); return 0 }
called by 1: fc_selftest calls 3: sys_mmapfc_numsys_write
89func fc_streq(a: *u8, b: *u8) -> i64 { var i: i64 = 0; while a[i] != (0 as u8) { if a[i] != b[i] { return 0 } i = i + 1 } if b[i] == (0 as u8) { return 1 } return 0 }
90func fc_sleep_ms(ms: i64) -> i64
calls 1: sys_mmap
104func fc_scan_objects(buf: *u8, n: i64, want_depth: i64, offs: *i64, lens: *i64, cap: i64) -> i64
145func fc_emit_text(dst: *u8, o0: i64, src: *u8, maxout: i64) -> i64
called by 3: fc_boardsfc_listfc_selftest calls 1: fc_slen
185func fc_media_url(dst: *u8, o0: i64, board: *u8, tim: i64, ext: *u8, thumb: i64) -> i64
199func fc_media_complete(b: *u8, n: i64) -> i64
219func fc_mkdir_p(path: *u8) -> i64
240func fc_fetch_one(url: *u8, store: *TrustStore, dir: *u8, name: *u8) -> i64
271func fc_get_json(url: *u8, store: *TrustStore, buf: *u8, out_status: *i64) -> i64
275func fc_api_url(dst: *u8, board: *u8, tail: *u8, thread: i64) -> i64
285func fc_boards(store: *TrustStore) -> i64
329func fc_list(store: *TrustStore, board: *u8, thread: i64, maxn: i64, mode: i64) -> i64
422func fc_fetch(store: *TrustStore, board: *u8, thread: i64, dir: *u8, maxn: i64) -> i64
505func fc_selftest() -> i64
667func fc_atoi(s: *u8) -> i64
677func fc_saveone(store: *TrustStore, board: *u8, file: *u8, dir: *u8) -> i64
692func main(argc: i64, argv: *i64) -> i64