Documentation  /  Node packaging & compute backend

HiNet Node Packaging & Compute-Backend — Planning Addendum

Companion to HiNet-Foundational-Model-Plan.md. Status: decision pass for the POC. Verified dev box: Apple M4 Max, 128 GB unified memory, arm64, macOS 26.5.1; uv 0.7.2 / Python 3.12.10; Ollama 0.6.6 (no models pulled); MLX not yet installed. Facts flagged "verified" are corroborated by sources at the end; "inferred" are engineering judgment.

The node ships as a native app (macOS first). Model heavy-lifting (inference and the overnight personalization training) runs through one ComputeBackend interface — LocalBackend for the POC, with a user-owned, attestable cloud-sandbox backend as a forward-compat drop-in. The POC is purely local.

A. Recommended App Architecture for the POC

Decision: Tauri v2 app (Rust core + web UI) wrapping a single 127.0.0.1-bound Python sidecar (hinetd). Not pure SwiftUI for the POC, not pure-Python-in-a-window.

Why this and not the alternatives. The load-bearing constraint is that the POC must do on-device LoRA training, not just inference, and the entire training ecosystem for the candidate bases lives in Python mlx-lm (LoRA/QLoRA/DoRA, MoE per-expert targeting, eval, fuse — verified). Pure Swift/mlx-swift forces re-implementing and maintaining that training stack in Swift, which Apple itself positions as research, not production — a velocity tax paid every training iteration (verified). Pure-Python shells (Briefcase/Toga, pywebview, py2app) run fastest day-1 but produce a "Python-process-in-a-window," not the owned-product shell the owner asked for, and moving off them is a full UI rewrite. Tauri keeps 100% of the Python brain intact, ships a genuine signed/notarized native .app with menu-bar presence, and lets the web UI iterate fast on the consent/vault/training-status screens.

Tauri vs. SwiftUI (the one divergence across briefs): SwiftUI gives the best native polish and cleanest entitlements story, but only at the cost of the Python training brain. Resolution: Tauri now; SwiftUI is a later front-end swap, not a POC track. Because the brain sits behind an HTTP seam, the shell is disposable — at product stage you can replace the web view with SwiftUI while keeping the identical Python sidecar, the same FastAPI contract, and the same security core. The view is the only thing rewritten.

Layering (native-app-over-local-service):

Critical trust boundary: secrets and the consent decision never enter Python. The Rust core authorizes a unit of work; the brain computes. The sidecar binds loopback only and requires the per-launch bearer token — a localhost daemon holding a personal vault is not "just localhost."

Fallback (named, not chosen): if Tauri externalBin notarization (tauri#11992, still open — verified) blocks the team > ~2 days, fall back to Electron + the identical Python sidecar (battle-tested nested-binary signing via electron-builder/@electron/notarize), not pure Swift. Same sidecar, same seam, same security core; you trade Chromium's memory footprint (competes with the model for unified memory) for a smoother signing path.

B. On-Device Model Runtime & Training

Engine: MLX / mlx-lm (~v0.31.x — verified) as the single compute engine for BOTH inference and training. MLX is the only Mac stack that does fast inference and native on-device LoRA/QLoRA. Ollama (0.6.6, installed) stays as zero-config bootstrap/fallback + GGUF coverage, not the production engine (~2–3× slower on MoE, no first-class training).

In-process vs. sidecar: sidecar, decisively. The model runs in the Python sidecar, not in-process (neither mlx-swift in-Rust nor embedded CPython). In-process mlx-swift has a thinner training surface and — fatally — would force a second, parallel network code path when the cloud backend lands, defeating the seam. Run inference/embeddings over an OpenAI-compatible HTTP surface (mlx_lm.server, or a hardened server isolating models as subprocesses for clean memory reclaim on unload) plus a small job API for training. Keep in-process mlx-swift on the roadmap only as a later latency optimization behind the same interface.

Inference. On 128 GB unified memory everything fits with headroom (4-bit MLX: Qwen3-Next-80B-A3B ≈ 43 GB (production base); Qwen3-30B-A3B ≈ 19.5 GB (icore-fast tier); gpt-oss-20b ≈ 11–17 GB; dense 14B ≈ 8 GB; 8B ≈ 4.5 GB). Prefer MoE (80B-A3B / 30B-A3B, ~3B active) for latency — faster than dense 14B despite more total params. 4-bit by default, 6/8-bit where the quality bar demands it (you have the RAM). Quantize on-box with mlx_lm.convert -q. Re-benchmark prefill at real RAG context lengths — MLX has trailed GGUF on long-prompt prefill, and the M4 Max lacks M5's matmul Neural Accelerators (~3.5–4× TTFT advantage on M5 — verified), so this box is prefill/compute-bound. That gap is itself an argument for the cloud seam. (tok/s figures are config/quant/context-sensitive — ballpark, not SLAs.)

Training. Overnight LoRA/QLoRA via mlx-lm. Start the pipeline on a small dense Qwen3 (4B/8B) for tight train→eval loops, then graduate to Qwen3-Next-80B-A3B MoE as the production personalization base (Qwen3-30B-A3B is now the icore-fast tier, not the production base). - Two hard gotchas: (1) MLX training needs HF-format weights, not GGUF — keep separate inference (GGUF/MLX-OK) and training (HF/MLX-only) weight paths. (2) Don't use gpt-oss-20b as a training base — native MXFP4 MoE weights stored as nn.Parameter force a bf16 up-cast that kills the FP4 win (mlx-lm#361 reports NaN-loss). Fine for inference. - MoE LoRA must target the expert/SwitchLinear layers — mlx-lm/mlx-tune detect MoE and apply per-expert LoRA (verified); a dense-LoRA config silently trains the wrong tensors. The model passport must carry the target-module set. - Command shape: mlx_lm.convert --hf-path Qwen/Qwen3-8B -q --q-bits 4 --mlx-path ./models/qwen3-8b-4bit mlx_lm.lora --model ./models/qwen3-8b-4bit --train --data ./ds \ --iters 600 --num-layers 16 --batch-size 4 # 4-bit base ⇒ auto-QLoRA QLoRA peak ≈ 7 GB (8B) / 12 GB (14B); full-precision LoRA fits even on 32B at 128 GB. A single 8B adapter is ~15–45 min, so an overnight window is over-provisioned — use it for multi-epoch, adapter sweeps, the eval gate. - Keep personalization as hot-swappable LoRA adapters loaded at runtime; only mlx_lm.fuse to a standalone model when you need an exportable artifact (this is the §3.1 folded-checkpoint face). Don't re-fuse the base nightly. - Engine pool gates on free unified memory (health() reports headroom): scheduling a train job while serving a large model can OOM the whole machine — there's no swap-friendly VRAM.

Avoid on Mac: HF peft/trl on PyTorch-MPS (op-gaps, silent CPU fallback) and Unsloth (needs Triton, no native Mac yet — verified). mlx-tune/mlx-lm-lora are the community bridges for RL objectives (DPO/ORPO/GRPO) later; keep the trainer pluggable, don't anchor on a community wrapper for the POC.

C. The ComputeBackend Seam

One interface between the node brain and all model engines. POC ships LocalBackend only — build no cloud backend yet, but route every model call through the interface from commit one. Baking local in (calling MLX in-process, hardcoding localhost) turns the future cloud drop-in into a rewrite.

Interface (async; every method returns a BackendReceipt alongside its payload):

get_capabilities() -> Capabilities   # device class, max ctx, dtypes, train-capable,
                                      #   attestation_level: none|local|tee, $/throughput hints
health()           -> Health         # liveness, loaded models, FREE UNIFIED-MEM, queue depth
load_model(ModelRef) / unload(handle)# ModelRef = passport(base_id, tokenizer/template hash,
                                      #   quant) + adapter_stack[] incl. MoE target modules
embed(EmbedJob)         -> (vectors, receipt)
infer(InferJob)         -> stream[Token] + receipt     # streaming first-class from day one
train_adapter(TrainJob) -> (AdapterArtifact, receipt)  # returns weights+metrics, never raw data
evaluate(EvalJob)       -> (Scorecard, receipt)        # the promotion gate

Streaming must be in infer() now even though LocalBackend is the only impl — retrofitting it later forces the exact app rewrite the seam exists to prevent.

Load-bearing principle: the seam moves computation, never authority. The atomic unit crossing the seam is a node-signed, consent-checked WorkOrder; every backend returns a BackendReceipt the node verifies and writes into the audit hash-chain.

WorkOrder{ op, model_ref(passport + adapter_stack[]),
           inputs_ref(inline | vault-sealed content-addressed handles),
           consent_grant_id + policy_snapshot_hash,   # the node's DECISION, already made
           sensitivity_ceiling, nonce, issued_at, expiry,
           node_sig: Ed25519 over canonical bytes }
→ WorkResult{ payload, BackendReceipt{ backend_id, model_hash, adapter_hashes,
           input_hash, output_hash, attestation_quote?, started/ended, usage } }

The node verifies input_hash/output_hash/model_hash/adapter_hashes against what it sent before writing the AuditEvent — non-negotiable, or a backend can silently swap the model or prompt and the audit chain certifies a lie. A backend cannot run anything that isn't a node-signed WorkOrder. (This WorkOrder/BackendReceipt envelope is also the natural attach point for the foundational plan's TOPLOC contribution receipts.)

LocalBackend (POC): wraps MLX-LM (mlx_lm.server for infer/embed) + mlx-lm/mlx-tune LoRA (train_adapter) + a local held-out eval harness (evaluate); Ollama as fallback. attestation_level = "local" (trust root = "the owner's own machine"). A single in-process engine pool keyed by ModelRef, gated on health() free-mem.

CloudSandboxBackend (forward-compat target, NOT built): a user-owned, attestable tenant — not an "approved remote API" (calling Anthropic/OpenAI is a different, lower-trust capability; keep them distinct or the ownership guarantee blurs). It is the confidential-computing pattern (Azure/GCP/NVIDIA primitives, production today — verified) repointed so the owned node is the relying party: 1. Per-user isolated CVM + GPU TEE (Intel TDX / AMD SEV-SNP CPU TEE; H100/Blackwell GPU TEE) the user provisions and pays for (BYO-cloud). 2. Attest-then-release-key: the node fetches the composite CPU+GPU attestation quote, verifies it (Google Cloud Attestation / Intel Trust Authority / NVIDIA NRAS), and only then HPKE-seals the WorkOrder and the wrapping key that unseals the personal LoRA/weights to the key in that same verified quote (avoids the attestation TOCTOU hole). No valid quote → node refuses to send. 3. Ownership preserved: weights + personal LoRA + data live encrypted in the user's tenant under customer-managed keys (CMK/BYOK); plaintext exists only inside the attested TEE for the job's lifetime; user can wipe the tenant. HiNet-the-project is never in the trust path. 4. Drop-in: implements the identical interface; attestation_level = "tee" + a tenant_id. The only new node-side logic is quote verification + key-sealing — entirely on the owned node.

Purely local in the POC (and forever): Ed25519 identity (Keychain), SQLCipher vault, the consent-policy decision, the audit hash-chain, memory/RAG index, the personal LoRA and all weights, WorkOrder signing + receipt verification, the eval gate. Forward-compat hooks to build now: the ComputeBackend interface, streaming infer(), the signed-WorkOrder / verified-BackendReceipt envelope, Capabilities.attestation_level, and a tenant_id-ready ModelRef. That is the entire cost of keeping the cloud backend a drop-in.

D. POC Build Notes

Repo / app layout (single repo):

hinet-node/
  app/                 # Tauri v2 — Rust core + web UI (React/Svelte, match team skill)
    src-tauri/         #   Rust: identity (ed25519-dalek+Keychain), SQLCipher vault,
                       #   consent engine, audit chain, sidecar lifecycle, bearer-token mint
  hinetd/              # Python 3.12 sidecar (uv venv): FastAPI loopback + ComputeBackend
    backends/local.py  #   LocalBackend(MLX-LM + Ollama fallback)
    backends/base.py   #   ComputeBackend interface + WorkOrder/BackendReceipt models
    train/  eval/  rag/
  models/              # HF (training) + MLX-quant (inference) weight paths — kept separate
  adapters/            # hot-swappable LoRA adapters + scorecards

• Sidecar packaging: PyInstaller --onedir (not --onefile — onefile has clean-shutdown issues), named *-aarch64-apple-darwin, wired via Tauri externalBin + tauri-plugin-shell. The brain ships inside the .app.

Packaging / signing posture (research POC — minimum that survives Gatekeeper; defer the rest): - Developer-ID signed + notarized + stapled, outside the App Store (the App Store sandbox fights a Python sidecar that spawns children and writes weights/vault; revisit at product stage). - Hardened Runtime mandatory (codesign -o runtime). For the PyInstaller+MLX sidecar these entitlements are required or the notarized build crashes at launch (works in dev): com.apple.security.cs.allow-jit, com.apple.security.cs.allow-unsigned-executable-memory, com.apple.security.cs.disable-library-validation. Apply to the main executable only, not nested .dylibs; no comments in the entitlements plist (the notary rejects them). - tauri#11992 is the single biggest POC risk (open — verified): the .app notarizes fine until externalBin is added, then fails on unsigned nested Mach-O. Mitigation: sign every Mach-O in the PyInstaller tree inside-out (codesign -o runtime deep) before signing/notarizing the .app. Budget 1–2 days; don't gate CI on a tight timeout (notarization can spike to 15–20+ min; use the submission-ID resume flow). If it stalls > ~2 days, take the Electron fallback (§A). - For the future cloud backend, add the network-client entitlement and a Keychain-access-group then — not in the POC.

Defer (explicitly out of POC scope): any cloud backend implementation; in-process mlx-swift; RL objectives (DPO/GRPO via mlx-tune) beyond plain SFT/LoRA; gpt-oss-20b as a training base; App Store / sandbox hardening; the SwiftUI front-end swap; multi-OS (Windows/Linux) packaging. Do not defer: the ComputeBackend interface, streaming infer, and the signed-WorkOrder/verified-receipt envelope — the forward-compat spine, cheap now.

First-week sequence (inferred, for velocity): (1) uv venv + MLX, pull HF Qwen3-8B, validate mlx_lm.convert → mlx_lm.lora → mlx_lm.fuse → eval end-to-end from the CLI before any app code; (2) stand up hinetd FastAPI with ComputeBackend/LocalBackend + WorkOrder envelope on loopback; (3) Tauri shell spawning the sidecar with the bearer-token handshake; (4) attack the notarization pipeline early (it's the schedule risk, not the code).

Sources: mlx-lm LORA.md · mlx-lm SERVER.md · mlx-lm#361 (gpt-oss NaN) · mlx-tune (MoE per-expert LoRA) · Qwen3.5 MLX guide · Fine-tune on Mac 2026 · Apple ML: LLMs in MLX on M5 · Tauri v2 sidecar · tauri#11992 (externalBin notarization, open) · Tauri macOS signing · Tauri+Python sidecar example · NVIDIA confidential computing H100 · Google Cloud Attestation · Azure AI Confidential Inferencing · az-cgpu-onboarding (CMK/BYOK)

Next: Technical spec →  ·  All documentation →