Nishi Family › Compare › GPU Worker Mesh and Distributed Inference
Nishi Compare · measured, not asserted
GPU Worker Mesh and Distributed Inference
Nishi vs the field — every Nishi cell is measured against real organ source at emit time; each gap names the watch contract that will close it.
A NAS orchestrator dispatching capability-scoped jobs to GPU workers (laptop 5080 images, west 3090 video, cloud) and pushing results to the gallery -- Nishi vs NVIDIA Triton and Ray Serve and Modal and KServe
Layer 1 · Executive
Where we are. Measured 2026-08-19. Every row of the mesh is present and proven: ocap dispatch, cross-host routing, sovereign actuation, daemonless governance, a content-addressed registry, batching scheduler, load balancing, autoscale to zero, streaming, admission, multi-model residency -- with three exceeds on sovereignty and capability scoping. The matrix carries no symbol gaps; this plan admits four watch contracts on the rows where Triton, Ray and Modal lead: circuit-breaker failover, on-demand model load, fused batched execution and a live remote replica.
Where we need to go. Make the mesh robust and fast -- breakers, on-demand residency, fused batches -- then prove the cluster with a remote replica serving live, keeping zero per-call cost and the ocap grant as the exceed.
The unit. 1 u = one measured session-leg. Calibration from landed rungs: the mangagen panel compositor went from existing substrate to shipped and live-verified in ONE leg (2026-08-13); the citations rung went from 3 to 55 domains in one leg across seven seats (2026-08-18); a greenfield engine with a bite-proven gate has measured 2 to 4 legs. Estimates recalibrate as rungs land and PR7 actuals write back.
Cost to robust: 2.5 u. Through M0.
Cost to throughput: 5 u. Through M1.
Cost to cluster: 7 u. Everything below.
16 of 21 capabilities measured|3 of them measured exceeds|5 open|coverage 761/1000|adoption 10 full / 6 partial
Layer 2 · Roadmap
Do this next — computed by the ranker, never chosen by a seat
Order from nx_compare_rank (nx_dr_ocm: (deficit + cost-of-delay + option + enables) x sponsor x self-sufficiency x momentum / cost). FINISH rows are rungs whose symbol is present but whose organ is short of full adoption: the cheapest closures on this board, listed before any new work. Stamp: # asof=1787883480 domain=workermesh target_version=0.1 rungs=4 done=0 open=4 finish=0 ranker=nx_dr_ocm
| # | Stage | Rung | Priority | Derivation |
|---|---|---|---|---|
| #1 | 0.1 | Circuit breaker (R0) wd_circuit | 1700 | v=17 m=1 c=10 |
| #2 | 0.1 | On-demand load and unload (R1) mx_load_on_demand | 800 | v=12 m=1 c=15 |
| #3 | later | Remote replica live (R3) lb_remote_replica | 600 | v=12 m=1 c=20 |
| #4 | later | Fused batched execution (R2) mb_fused_exec | 240 | v=6 m=1 c=25 |
Critical path — contract, done-rule, executor, cost
| Rung | Closes with | Definition of done (pre-declared) | Executor | Est. |
|---|---|---|---|---|
| Circuit breaker (R0) | wd_circuit | Per-worker failure history opens a breaker after N failures, half-opens on a schedule and closes on success; N and the schedule are conf rows; a flapping fixture is shed, a healthy one is routed | Organ | 1 u |
| On-demand load and unload (R1) | mx_load_on_demand | A request for a non-resident model triggers load under the VRAM budget with LRU eviction and the request completes; idle models unload on the autoscale schedule; VRAM before and after printed | Organ | 1.5 u |
| Fused batched execution (R2) | mb_fused_exec | A batching-capable backend executes n greater than 1 as one forward pass; measured seconds per image at batch 4 below sequential by a PRE-DECLARED margin on the 5080 | Organ | 2.5 u |
| Remote replica live (R3) after R0 | lb_remote_replica | The 3090 box or a cloud replica joins as an endpoint row, serves live traffic through the balancer and is skipped when it drops; the routing log names the replica per request | Organ | 2 u |
Milestones
| Milestone | Rungs | Cumulative |
|---|---|---|
| M0 · Robust | R0,R1 | 2.5 u |
| M1 · Throughput | R2 | 5 u |
| M2 · Cluster | R3 | 7 u |
Layer 3 · Engineering
How this is scored. Every Nishi mark is measured: the generator reads the real organ source on disk and requires the implementing symbol to exist (no self-grading). A watching tag names the organ and symbol contracted to close a gap — the mark flips itself on the next compare beat when that workstream ships, and the
comparewatch- plane row flips with it. The flip is necessary, not sufficient: it proves the symbol exists, never that the capability is good. The bar is the rung's pre-declared done-rule, proven by its gate — a symbol shipped without the behaviour behind it is a defect, and the flip is exactly what makes that defect visible instead of quiet. Competitor marks record documented capability presence — presence, not depth or scale. Adoption is measured too: every measured row carries where its organ stands on the estate's ladder (source → built → promoted → registered → invoked; libraries by importer reach minus validation importers; gates by the execution surfaces that run them). A row is fully adopted only at the top of its ladder; anything short is tagged partial with the exact remedy, so a build nobody promoted can no longer read as shipped. Census stamps: importers asof 1787849099, gate census asof 1787855507 (unix seconds; -1 = census absent).Capability matrix — measured against source
◉ leads / measured exceed● present◐ partial○ absent · click any capability for its evidence
| Capability | Nishi | NVIDIA Triton | Ray Serve | Modal | KServe |
|---|---|---|---|---|---|
Capability-scoped job dispatchMeasured:WORKERDISPATCHGATE exists in runtime/nx_worker_dispatch.nx, verified at emit. Ray and Modal are Best at programmatic dispatch [ray-serve] [modal-scale]; Nishi routes by job type to the right worker class and enforces the grant deny-by-default Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ● | ● | ◉ | ◉ | ● |
Cross-host GPU worker routingMeasured:wd_route exists in runtime/nx_worker_dispatch.nx, verified at emit. Ray and K8s-native KServe route across a cluster as Best [moritz2018] [kserve-kpa]; Nishi routes image to the 5080 and video to the 3090 and other to cloud Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ● | ● | ◉ | ◉ | ◉ |
Sovereign end-to-end actuationMeasured:MESHGENGATE exists in runtime/nx_mesh_gen.nx, verified at emit. All frameworks serve inference; Nishi proved a single organ POST-decode-publish loop over its own TCP with zero shell Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ● | ● | ● | ● | ● |
Native GPU container governance daemonlessOpen — no implementing organ is measured for this axis yet. Triton and Ray need a server daemon and Modal is cloud and KServe needs Kubernetes; Nishi governs the worker natively via Job Objects no daemon no VM | ○ | ◐ | ◐ | ◐ | ◐ |
Content-addressed model and artifact registryMeasured:CONTAINERREGISTRYGATE exists in runtime/nx_container_registry.nx, verified at emit. Triton model repository is the reference; Nishi has a SHA-256 content-addressed registry with integrity-verified pull Adoption: SOURCE-ONLY — PARTIAL: source exists, never compiled: /api/build it. | ● | ◉ | ● | ◉ | ◉ |
Push results to consumable storageMeasured:mg_gen exists in runtime/nx_mesh_gen.nx, verified at emit. All can write outputs; Nishi decodes the worker response and publishes to the gallery in the same sovereign organ Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ● | ● | ● | ● | ● |
Model load and unloadMeasured:sd-server exists in runtime/nx_pe_container_sdserver.nx, verified at emit. Triton and Modal do dynamic model load and unload as Best; Nishi loads and unloads the worker (VRAM 0 to 10.9GB to 0) but manually not yet on demand Adoption: SOURCE-ONLY — PARTIAL: source exists, never compiled: /api/build it. | ● | ◉ | ● | ◉ | ◉ |
Worker health probe and livenessMeasured:wd_probe exists in runtime/nx_worker_dispatch.nx, verified at emit. Mature frameworks have rich health and failover as Best; Nishi has a basic reachability probe Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ● | ◉ | ◉ | ◉ | ◉ |
Dynamic request batchingMeasured:MESHBATCHGATE exists in runtime/nx_mesh_batch.nx, verified at emit. Triton fuses requests into one GPU forward pass [triton-modelconfig] [crankshaw2017]; Nishi has a sovereign batch SCHEDULER (coalesce by preferred-size OR max-delay, sim 32 into 4 batches of 8) AND live n=K batched EXECUTION (proven: genbatch 3 = 3 images in ONE round-trip on the 5080); MEASURED-HONEST -- the current sd.cpp engine runs n greater than 1 SEQUENTIALLY (20.3s per image batched vs 19.3s sequential) so today the win is fewer dispatches and connections not fewer forward passes; true fused throughput needs a batching-capable backend -- Nishi now HAS the compute primitive (nx_batch_infer, a sovereign fused batched-GEMM, bit-exact vs sequential via weight-stationary reuse); the CPU reuse win is modest (compute-bound), the large win is GPU kernel-launch amortization Adoption: REGISTERED-UNAUTHORISED — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). | ● | ◉ | ● | ● | ● |
Multi-worker load balancing across replicasMeasured:MESHLBGATE exists in runtime/nx_mesh_lb.nx, verified at emit. Balancing across replicas is Best in every framework; Nishi now health-probes each replica (bounded 1s non-blocking) and round-robins among the healthy ones skipping the down -- PROVEN live routing to the laptop 5080 and skipping the offline west 3090; west or a cloud replica joins as one endpoint row Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ● | ◉ | ◉ | ◉ | ◉ |
Autoscaling scale-to-zero on demandMeasured:MESHAUTOSCALEGATE exists in runtime/nx_mesh_autoscale.nx, verified at emit. Modal and KServe scale-to-zero are Best [modal-scale] [knative-autoscaling]; Nishi has a proven policy controller AND now a HANDS-OFF host-agent (nx_mesh_hostagent) that autonomously forks the worker on demand and reaps it on idle -- PROVEN with a real process (probe 0 to 1 to 0, native fork and kill lifecycle); Nishi idle is a TRUE 0 MB 0 procs so scale-to-zero costs almost nothing Adoption: REGISTERED-UNAUTHORISED — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). | ● | ● | ◉ | ◉ | ◉ |
Streaming responsesMeasured:MESHSTREAMGATE exists in runtime/nx_mesh_stream.nx, verified at emit. Token and chunk streaming is standard; Nishi now emits a valid Server-Sent-Events stream -- accepted then one progress event per step (step total pct) then a terminal result with the artifact URL, consumable by a browser EventSource and the same shape as our live /api/events [whatwg-sse]; the natural frame-delivery channel for video on the west 3090 Adoption: LIVE — fully adopted (top of its ladder). | ● | ● | ◉ | ◉ | ● |
Request queue and admission controlMeasured:MESHBATCHGATE exists in runtime/nx_mesh_batch.nx, verified at emit. Queueing and backpressure protect the GPU under load; Nishi now has a bounded queue with admission control -- sim proved 100 requests at cap 64 admits 64 and sheds 36 with backpressure, never over-cap, and no request starves Adoption: REGISTERED-UNAUTHORISED — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). | ● | ◉ | ◉ | ◉ | ● |
Multi-model multiplexing on one GPUMeasured:MESHMUXGATE exists in runtime/nx_mesh_mux.nx, verified at emit. Serving many models on shared GPU memory is Best in the frameworks; Nishi now has a VRAM-budgeted LRU model residency -- route by model-id, keep the hot models loaded, evict the least-recently-used to fit a new one, refuse an oversized model; composes with autoscale load-unload and the content-addressed model registry Adoption: REGISTERED-UNAUTHORISED — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). | ● | ◉ | ◉ | ◉ | ◉ |
EXCEED sovereign: own-stack no serving frameworkMeasured exceed:MESHGENGATE in runtime/nx_mesh_gen.nx, verified at emit. The whole mesh is our own TCP client plus base64 plus container governance plus dispatch -- no Triton no Ray no CUDA serving runtime no cloud dependency Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ◉ | ○ | ○ | ○ | ○ |
| EXCEED unique | |||||
capability-scoped resource accessMeasured exceed:wd_authorize in runtime/nx_worker_dispatch.nx, verified at emit. A job may only call the worker and storage its capability grants -- deny-by-default MCP-style trusted-to-do-work enforcement none of the frameworks have this model [miller2006] Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ◉ | ○ | ○ | ○ | ○ |
EXCEED sovereign: zero-cost on your hardware auditableMeasured exceed:mg_gen in runtime/nx_mesh_gen.nx, verified at emit. Runs on your own GPUs at no per-call cost with the whole loop readable bits-up; the managed frameworks are cloud-metered or heavyweight infra Adoption: RUN-BY:fork:nx_mesh_run — fully adopted (top of its ladder). | ◉ | ○ | ○ | ○ | ○ |
Failover with health history and circuit breakerOpen — watchingruntime/nx_worker_dispatch.nx : wd_circuit, re-measured on every compare beat. Ship that symbol and this mark flips itself; the comparewatch- plane row flips with it. The reachability probe exists; a breaker that opens on repeated failure and half-opens on a schedule is the mature-framework bar | ○ | ◉ | ◉ | ◉ | ◉ |
On-demand model load and unloadOpen — watchingruntime/nx_mesh_mux.nx : mx_load_on_demand, re-measured on every compare beat. Ship that symbol and this mark flips itself; the comparewatch- plane row flips with it. Load and unload exist manually; on-demand by request with the LRU residency already shipped closes the row | ○ | ◉ | ● | ◉ | ◉ |
Fused batched forward on the workerOpen — watchingruntime/nx_mesh_batch.nx : mb_fused_exec, re-measured on every compare beat. Ship that symbol and this mark flips itself; the comparewatch- plane row flips with it. The scheduler batches and the backend runs sequentially; a batching-capable backend turns fewer dispatches into fewer forward passes | ○ | ◉ | ● | ● | ● |
Remote replica live across hostsOpen — watchingruntime/nx_mesh_lb.nx : lb_remote_replica, re-measured on every compare beat. Ship that symbol and this mark flips itself; the comparewatch- plane row flips with it. Round-robin over healthy replicas is proven on the LAN; a remote replica (west 3090, cloud) serving live is the cluster proof | ○ | ● | ◉ | ◉ | ◉ |
On these two registers. Rows are declared in the domain's plan file and carry the debt id, which is the join key back to the sovereign debt plane — that plane, not this page, is the authority on state. Reconciling them automatically (the regen reading the plane and refreshing these rows) is a named, owed rung; until it lands, treat an id here as a pointer to look up, not a status to trust.
Honest verdict. The coverage above is capability presence measured against source — not depth, scale, or polish, where mature rivals may lead. Exceeds are claimed only where a mechanism backs them. Every open gap is a watch contract: it names the organ and symbol that closes it, and this page flips the cell itself when that workstream ships.
Person · product · place — not yet measured for this domain
Every compare carries this layer. Declare
knowledge/compare/workermesh.ppp (rows surface|nishi or c1..c4|label|url|connect naming OUR live surface and each rival's front door), run nx_ppp_probe domain workermesh, and this section fills itself on the next beat: the same ruler on both sides — privacy and CX (third-party hosts, tracker classes, cookies, security headers), design and longevity (design hygiene, computed WCAG contrast, render-blocking resources, unsized media, script weight, theme and motion queries), findability (landmarks, skip link, on-site search, breadcrumb, headings, internal links).References
Beyond a link list. Every reference below resolves twice — the publisher's copy and, where banked, the estate's own non-rottable library mirror with a content pin — and carries its evidence class plus the exact claim on this page it grounds. Keyed marks like [key] in the matrix notes jump here. A dash means honestly absent, never assumed.
- [triton-modelconfig] NVIDIA. Triton Inference Server -- Model Configuration (user guide): model repository layout, instance groups, dynamic_batching (preferred batch sizes, max queue delay), sequence batching and model warmup. publisher · read in our library
knowledge/fetched/cmp_workermesh_triton-modelconfig.html· pinh531bec727c4e04515c0d9657db12bd5cb71532af826e22864f6e15c29ec95b2d· accessed 2026-08-18 · vendor-docGrounds: The NVIDIA Triton column: Dynamic request batching (Triton fuses requests into one GPU forward pass -- the preferred-size OR max-delay policy nx_mesh_batch mirrors), Content-addressed model and artifact registry (the model repository is the reference), Model load and unload (Best) and Multi-model multiplexing on one GPU. - [ray-serve] Anyscale / Ray project. Ray Serve: Scalable and Programmable Serving (docs.ray.io/en/latest/serve): deployments, replicas, autoscaling, request batching, model composition and streaming responses on a Ray cluster. publisher · read in our library
knowledge/fetched/cmp_workermesh_ray-serve.html· pinh4d68895f641dfbfb10eadc8fe24ef29c9655836f7a5b5a14e0d58dacde8331d6· accessed 2026-08-18 · vendor-docGrounds: The Ray Serve column: Capability-scoped job dispatch and Cross-host GPU worker routing (Best at programmatic dispatch across a cluster), Multi-worker load balancing across replicas, Streaming responses and Autoscaling scale-to-zero (Best) -- the serving layer our sovereign mesh_gen loop is graded against. - [moritz2018] Moritz, Nishihara, Wang, Tumanov, Liaw, Liang, Elibol, Yang, Paul, Jordan, Stoica. Ray: A Distributed Framework for Emerging AI Applications. USENIX OSDI 2018. publisher · read in our library
knowledge/fetched/cmp_workermesh_moritz2018.html· pinh78f4b1e52b0a2fdb2e1853f519f36ee1b9b4dec27b0b3f6afb6423ff8941eca0· accessed 2026-08-18 · published-paperGrounds: The distributed substrate under the Ray Serve column: the dynamic task graph, distributed scheduler and object store that make Cross-host GPU worker routing and Multi-worker load balancing across replicas Best in Ray -- the published architecture our single-organ TCP dispatch is contrasted with. - [modal-scale] Modal Labs. Scaling out (modal.com/docs/guide/scale): every Function is an autoscaling pool of containers, scale to zero by default when idle, min_containers and max_containers, and concurrent inputs. publisher · read in our library
knowledge/fetched/cmp_workermesh_modal-scale.html· pinhf7099caee48204296a4bceed81e37633dba524236a09c1d1a818c3f1906b2d37· accessed 2026-08-18 · vendor-docGrounds: The Modal column: Autoscaling scale-to-zero on demand (Best -- the bar the nx_mesh_hostagent fork-on-demand reap-on-idle proof is measured against), Capability-scoped job dispatch (Best programmatic dispatch), Model load and unload and the cloud-metered contrast in EXCEED sovereign: zero-cost on your hardware auditable. - [kserve-kpa] KServe Authors. Autoscaling with Knative Pod Autoscaler (kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/kpa-autoscaler): concurrency-based autoscaling of InferenceServices, scale to zero via minReplicas 0 in Knative serverless mode. publisher · read in our library
knowledge/fetched/cmp_workermesh_kserve-kpa.html· pinh87aee1efb0ca25cdd3c53a55700281cd3cb3e9f657597b2f5585bed6413c63ac· accessed 2026-08-18 · vendor-docGrounds: The KServe column: Autoscaling scale-to-zero on demand (Best) and Cross-host GPU worker routing (K8s-native routing across a cluster as Best), plus the needs Kubernetes note in Native GPU container governance daemonless. - [crankshaw2017] Crankshaw, Wang, Zhou, Franklin, Gonzalez, Stoica. Clipper: A Low-Latency Online Prediction Serving System. USENIX NSDI 2017. publisher · read in our library
knowledge/fetched/cmp_workermesh_crankshaw2017.html· pinh13f076342574c8cc9773645ca3421a5247c599c8b777af81524d6321192e0373· accessed 2026-08-18 · published-paperGrounds: The Dynamic request batching and Request queue and admission control rows: adaptive batching that trades queueing delay for GPU throughput under a latency objective is this paper's contribution -- the published bar for the coalesce-by-preferred-size-OR-max-delay scheduler and the bounded queue with backpressure (100 at cap 64 admits 64 sheds 36). - [whatwg-sse] WHATWG. HTML Living Standard, section 9.2 Server-sent events: the EventSource interface and the text/event-stream wire format. publisher · read in our library
knowledge/fetched/cmp_probe_whatwg_sse.html· pinh56e4bbeb7c54f2f55a01cdfd9cbc68ec25bb03849caa18814690b3049573761b· accessed 2026-08-18 · published-standardGrounds: The Streaming responses row: nx_mesh_stream emits a valid Server-Sent-Events stream (accepted, per-step progress, terminal result with the artifact URL) consumable by a browser EventSource -- the wire format is normative here and is the same shape as our live /api/events. Not mirrored: the sovereign fetcher could not complete the TLS 1.2 body transfer from html.spec.whatwg.org (verified live via a second reader). - [miller2006] Miller, M. S. Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control. PhD dissertation, Johns Hopkins University, 2006 (JScholarship record; author copy at erights.org). publisher · read in our library
knowledge/fetched/cmp_workermesh_miller2006-jhu.html· pinh3bb2fcee5cba1d79f76b834d676e96b8005f68f978d4713248b3103f4abbc170· accessed 2026-08-18 · published-paperGrounds: The EXCEED unique: capability-scoped resource access row and Capability-scoped job dispatch: a job may only call the worker and storage its capability grants, deny-by-default -- the object-capability discipline this dissertation formalises, which none of the four frameworks model. - [knative-autoscaling] Knative Authors. Knative Serving -- Autoscaling (knative.dev/docs/serving/autoscaling): the Knative Pod Autoscaler, concurrency and RPS targets, scale-to-zero and scale bounds. publisher · read in our library
knowledge/fetched/cmp_workermesh_knative-autoscaling.html· pinh4af390da890a19b69db70d699695d1d8f30cbcbacfd951314c8bbc603344d312· accessed 2026-08-18 · vendor-docGrounds: The Autoscaling scale-to-zero on demand row: KServe's scale-to-zero is Knative's KPA, so the Knative autoscaler is the mechanism behind the KServe Best cell -- and the class the swarm register names as KubeEdge/Knative-class scale-to-zero.
generated by nx_swcompare_matrix (sovereign NishiLang organ) from knowledge/compare/workermesh.matrix · every Nishi cell verified against organ source at emit time · watch cells re-measured on every compare beat · zero JS, zero trackers