nx_joblost_gate.nx

nx_joblost_gate.nx -- WHICH ASYNC JOBS DIED, as distinct from which are still running. WHY THIS EXISTS, measured 2026-08-07. Under load (loadavg 18.18 on the 8-thread box, nx_tools_api_se in D-state, /api/health 503 "upstream backend gave no response inside the edge window") nx_tools_api claims a job id with O_CREAT|O_EXCL and can then die before the worker ever produces output. The caller gets the bare JSON object `{}`. Two of my writes were destroyed that way inside five minutes and I only noticed because I happened to read the row back. The estate's standing rule for a dropped response -- VERIFY BY ARTEFACT -- fails silently here, because the artefact was never written and the job record is EMPTY rather than ABSENT. --------------------------------------------------------------------------------------------- v1 OF THIS GATE WAS WRONG AND SHIPPED GREEN. It counted every zero-byte .claim as a LOST job. Reading nx_tools_api afterwards showed the claim is created EMPTY and only filled at completion (ta_job_put -> sys_renameat), so a zero-byte claim is ALSO the normal representation of a job that is still RUNNING. v1 therefore merged "in flight" with "died" under the alarming name. ★★★★★★ I WROTE "UNKNOWN IS ITS OWN BUCKET" INTO THIS FILE'S OWN HEADER AND THEN BROKE IT IN THE SAME HOUR. A LAW YOU CAN RECITE IS NOT A LAW YOU HAVE APPLIED. v1 also declared the imprecision as unavoidable -- "separating in flight from wedged needs a clock and this gate does not have one". IT HAD ONE ALL ALONG: the job id IS the epoch second, printed in every filename. ★★★★★★ THE INFORMATION I DECLARED UNAVAILABLE WAS ENCODED IN THE NAME OF THE FILE I WAS ALREADY READING. "THIS CANNOT BE MEASURED" IS A CLAIM THAT MUST BE EARNED, EXACTLY LIKE ANY OTHER. FIVE BUCKETS, each with a different remedy, none folded into another: DONE state=DONE was written -- healthy, nothing to do INFLIGHT empty claim, no .out, young -- probably running; NOT an alarm LOST empty claim, no .out, OLD -- the id was reserved and nothing ever came back ORPHAN empty claim but .out EXISTS -- the worker produced output and never renamed the claim, so a poller waiting on state=DONE waits forever REAPED a tombstone was APPENDED -- adjudicated and closed by nx_jobclaim_reap; the claim keeps its original marker, so only a reader that tests REAPED FIRST can see it (2026-08-20) The age threshold is DECLARED and argv-overridable, and it is deliberately generous: jobs run with tmo=0 (no timeout) precisely so they can outlive the request window, so a tight bound would accuse healthy long jobs. ★ A HEURISTIC THAT GATES AN ALARM MUST BE WRONG IN THE DIRECTION OF SILENCE. license_tier: ORIGINAL expect_exit: 0