nx_freshness.nx
buildroot/runtime/nx_freshness.nx
about
nx_freshness.nx -- LIB: adaptive RE-CRAWL scheduling = index freshness without wasting crawl budget. Per URL the
scheduler adapts the re-crawl interval to observed change: a page that CHANGED since last crawl is crawled SOONER
(halve the interval); a page that DID NOT change BACKS OFF (double it), bounded [MIN,MAX]. Important pages (high
PageRank priority) get a shorter EFFECTIVE interval so the head of the web stays fresh. This keeps a billion-page
index current on a finite crawler -- the freshness half of "a bigger BETTER index". Built + gated NOW; runs over the
live corpus on the NAS (the scheduler drives the crawler; last-content-hash from the ingested doc). No float.
license_tier: ORIGINAL
dependencies 1 imports · 1 importers
imports: nx_syscalls.nx
imported by: nx_freshness_gate.nx
structs
| none |
consts
| 9 | const FR_MAGIC_2000: i64 = 2000 |
| 11 | const FR_MIN: i64 = 3600 // 1h floor (seconds) |
| 12 | const FR_MAX: i64 = 2592000 // 30d ceiling |
functions
| 15 | func fr_update_interval(interval: i64, changed: i64) -> i64 called by 1: main |
| 23 | func fr_due(last_crawl: i64, interval: i64, now: i64) -> i64 { if now >= last_crawl + interval { return 1 } return 0 } called by 1: main |
| 25 | func fr_effective_interval(interval: i64, priority_permille: i64) -> i64 called by 1: main |