Documentation  /  Personalization engine

HiNet — Personalization Engine & Hardware Tiers

Status: design options to research + test (documented 2026-06-21, per decision to "document the possibilities, then research and test them"). Companion to Foundational-Model-Plan §4, Personal-Data-and-Sovereignty, Node-Packaging-Addendum. Consolidated in the Whitepaper.

This doc captures how an iCore is built from a person's data — the training-method spectrum, the depth↔composability trade-off, and the hardware tiers — so we can research and test the open questions rather than assume.

1. What the research established (grounding)

(Sources in Whitepaper §9 / project memory.) - Fine-tuning teaches voice/behavior/skill; RAG teaches facts. (Ovadia "FT vs Retrieval"; OPPU EMNLP-2024 — the canonical per-user system is explicitly hybrid; Stanford FineTuneBench: commercial FT APIs ~37% on new facts.) - 4-bit quantization caps knowledge capacity >2× (Physics-of-LMs 3.3: ~2 → ~0.7 bits/param). Use bf16/int8 for the knowledge/deep stage; reserve 4-bit for inference. - Extractable knowledge requires diverse augmentation — varied phrasings, sentence-shuffling, and especially Self-QA (Physics-of-LMs 3.1: 1 phrasing → 9.7%; 5 diverse → 96.6%). Train in QA/extraction format, not raw docs. Inject bidirectionally (Reversal Curse). - Overtraining new facts increases hallucination (Gekhman) → replay + regularize-to-base + eval-gate. - LoRA was a feasibility default, not a requirement. It absorbs less new knowledge than full FT (Biderman); rank/target-modules matter more than people assume. We want deeper mutation — "a genuinely new model," not a thin layer — which is achievable with heavier methods + the overnight compute budget.

Therefore the iCore is hybrid: vault + RAG (the person's facts/memory) + tuned weights (the person's voice/skill/reasoning), with the weights tuned as deeply as the hardware allows.

2. The personalization-method spectrum (to test)

Required regardless of method (the recipe that fixes our earlier failure): bf16/int8 (not 4-bit) for the deep stage · diverse Self-QA augmentation · QA/extraction format · bidirectional facts · replay general data + regularize toward base · eval-gate (promote/rollback, watch the hallucination cliff) · RAG for facts/freshness alongside.

3. Depth ⟷ composability (the core tension) + reconciliation

The fractal "compose iCores into a bigger model" mechanism works because iCores stay near a shared base (linear mode connectivity — merges don't interfere). Deep, independent full-FT drives models apart → naive merging breaks. Depth and easy composability pull opposite ways.

Reconciliation (Branch-Train-Merge / federated paradigm): - Shared base + deep local fine-tune + compose via task vectors — TIES/DARE/model-soup operate on the full delta W_user − W_base, not just LoRA, so full-FT models are still mergeable relative to the shared base… - …as long as drift between syncs is bounded by periodic anchor re-basing (the slow communal clock). Re-basing is precisely how "a new network-level foundational model is built from the personally-tailored iCores." Deep nightly local mutation; bounded, consolidated communal re-basing.

Open question to test: how far can independently fine-tuned iCores drift and still merge cleanly — i.e., the relationship between fine-tune depth, re-basing cadence, and merge interference.

E1 confirmed the tension empirically (2026-06-21). Merging 3 heavily-overfit bf16 full-FT specialists cost ~28%/pack (each ~0.93 solo → ~0.63 merged; macro 0.667). The composed single model still beat the best single (+11pp) — so composition works — but sat well below the routing ceiling (0.934). Resolution → route vs merge (full mechanism: Foundational-Model-Plan §3.3.1–3.3.2): for distinct specialists, route the minimal sufficient set per prompt (≈ ceiling, full fidelity) and reserve merging for when a single persistent low-latency model is needed — then keep deltas bounded (DoRA/lower-rank/less-overfit) + TIES/DARE + re-basing to shrink the tax. "More isn't better" validated: routing the right specialist (0.934) beat merging all (0.667).

4. Hardware tiers — the same human, multiple coherent iCore builds

The iCore is not one binary. The same person runs different-depth iCore instances on different hardware they own, kept coherent by one identity + the shared vault.

Coherence across tiers (one mind, many bodies): - One identity (Ed25519); one vault — data + deltas replicate via VaultReplication (ciphertext, user-held keys). - Learning flows down the tiers by distillation: the cloud full-FT model is the deepest; it distills into the laptop's DoRA/high-rank adapter, which distills into the pocket small-model. Heavier tiers teach lighter tiers. - ComputeBackend abstracts where a job runs; the person's "true iCore" is the family of synced instances, deepest in the cloud, most-sovereign in the pocket. - Why the pocket/Jetson tier matters: it's the strongest self-custody story — the person physically holds a running iCore on an owned device, independent of laptop or cloud. A sovereign distribution form factor worth planning for.

5. Open research + test questions

Research (verify before/while testing): - 2026 state of robust personalization fine-tuning (DoRA/high-rank/full-FT) and continued-pretraining-with-replay recipes. - Drift-vs-merge limits: how far can independently fine-tuned models diverge and still merge (task-vector interference vs re-basing cadence; Git-Re-Basin/permutation alignment at scale). - Exact pocket-device target: Jetson Orin Nano (edge) vs DGX-Spark-class pocket AI box — specs, which base/method fits, on-device FT feasibility. (Confirm which the operator means.) - Full-FT memory limits per tier (30B full-FT ≈ 240–480 GB optimizer state → not on a 128 GB laptop; 7–8B full-FT feasible locally).

Test (on our hardware): 1. Robust single-iCore re-attempt — bf16 + DoRA/high-rank + diverse Self-QA + replay + eval-gate: does deep personalization hold without collapse? (Fixes the earlier failed run.) 2. Laptop method bake-off — full-FT-7B vs high-rank-LoRA/DoRA-on-30B: which personalizes better on quality × depth × composability? 3. Composition via task-vectors of independently-tuned models + a re-basing step. 4. Tier coherence — cloud→laptop→pocket distillation fidelity (later).

5a. Research results (2026-06-21) — recipes, the rule of thumb, hardware verdicts

Web-verified; sources in project memory. These convert §2–§5 from "options" to "settled recipes."

Method (Q1): DoRA ≈ best depth-for-budget (beats LoRA +2.9–4.4 pts at equal params, ~full-FT learning pattern, zero inference overhead post-merge). High-rank LoRA (r=256) matches full-FT for style/voice, lags for knowledge (gap persists even at r=256; full-FT perturbations are 10–100× higher rank). LoRA/DoRA forget less than full-FT — decisive for nightly continual FT. The "low-regret regime" (PEFT == full-FT) holds iff (a) target ALL linear layers incl. MLP/experts, not attention-only, and (b) not capacity-constrained. - Recipe: DoRA (or high-rank LoRA) · all-linear (q,k,v,o,gate,up,down) · r=16–32 voice / r=128–256 knowledge · α=2r (or rsLoRA α/√r at high rank) · lr ~2e-4 (~10× full-FT) · 1 epoch (multi-epoch overfits small corpora) · effective batch ~16 · dropout 0. - Knowledge stage (durable facts): bf16/int8 (NOT 4-bit — int4 cuts capacity >2×) · train embed_tokens+lm_head · rsLoRA at r≥256 · diverse Self-QA augmentation + bidirectional facts. - Anti-forgetting (overnight continual): ~5% replay of general data (even 1% helps; raise to ~25% for instruction streams) + LR re-warm/re-decay each night (replay counters re-warm forgetting) + optional EWCLoRA.

Composability (Q2): Merging works in a regime. 2 shared-base fine-tunes merge ~98–99%; degrades with #models (≈98%@2 → 87%@7 → 82%@11). SFT deltas are ~80–99% redundant (TIES keeps top-20% by magnitude; DARE drops 90–99%). Full-FT task-vectors merge better than LoRA (LoRA injects off-spectrum "intruder dimensions", low CKA → SVD-align/KnOTS first, or use r≥64). Boundary: CPT-magnitude deltas (~0.03 vs SFT ~0.002) or a different base → merging collapses. Git-Re-Basin is irrelevant (shared init ⇒ one basin ⇒ plain arithmetic works). DiLoCo: H≈500 inner steps → ~500× less comm, validated at LLM scale → the re-sync cadence.

Hardware (Q3–Q4): memory ladder — full-FT ≈16 B/param · 8-bit-Adam ≈10 · QLoRA ≈0.5 + adapter. | Device | FT verdict | |---|---| | 128 GB Apple Silicon (MLX) | PEFT up to 70B comfortably; full-FT 7–8B borderline (8-bit opt + batch1 + grad-ckpt, slow); full-FT 30B no | | DGX Spark (GB10, 128 GB, ~$4k) | QLoRA ≤70B, LoRA 8B, full-FT ~3B — desk appliance, NOT pocket | | Jetson Orin Nano Super (8 GB, $249) | ~8B 4-bit inference; only 1–3B tiny QLoRA | | Flagship phone (12–24 GB) | 1–8B 4-bit inference + LoRA adapter-swap only, no real training |

Cross-tier (Q5): 30B + DoRA/high-rank wins on quality×depth; full-FT-7B wins on composability/serving-simplicity. Since HiNet's north-star is composing many iCores, resolve toward small/clean mergeable deltas on a bigger shared base — depth from base size + training nights, not from violent FT.

THE RULE OF THUMB: Keep every iCore on one frozen shared base; keep its personal delta in the SFT-magnitude, ~80–99%-redundant zone (DoRA/LoRA r≤~256, ≥5% replay, not CPT-scale drift). Then 2–4 iCores merge near-losslessly and ~8–10 merge gracefully via TIES/DARE. Go deeper via a bigger base + more nights, not a more violent fine-tune; re-sync drifting replicas on a DiLoCo cadence.

5b. Merge-interference: methods, re-basing, per-tier recipe (research 2026-06-22)

Web-grounded synthesis complementing the M5-009 sweep. Sources at end.

Root cause: interference scales with delta MAGNITUDE / parameter-space displacement, not the averaging rule. L2 displacement >~50 from base correlates with catastrophic forgetting [In-the-Wild 2511.21437]. So: bound the delta near the base, then close the residual in function space — swapping sum↔TIES↔DARE alone buys only single digits.

Merge methods to add, ranked (all MLX-implementable): 1. Distillation-into-merge (SAMerging) — the big lever. After any weight merge, short KL-distill each iCore's soft outputs into the merged weights on a small unlabeled probe set (~16–256/iCore). ~96.5% of the fine-tuned ceiling [SAMerging 2512.21288] — published evidence a merged single model can approach the routing ceiling. Not a merge algorithm — a post-merge function-space pass. Highest ROI. 2. Localize-and-Stitch — stitch only ~1% disjoint skill regions per iCore onto the frozen base; top MergeBench performer on real LLMs (≫ TIES/DARE) [2408.13656 / 2505.10833]. 3. RegMean — closed-form per-layer least-squares matching each specialist's activations; co-top on LLMs when you have per-iCore data (we do). 4. TSV-Merge / Iso-C — subspace orthogonalization; SOTA in our (distinct-task, shared-base) regime on vision, unverified on LLMs → use an L2-displacement guardrail (alarm ~50) [TSV CVPR 2025]. 5. Model Breadcrumbs — clip outlier + negligible deltas; aimed at the full-FT-heavy case. 6. DELLA — strongest no-training baseline (control to beat) [2406.11617]. - Skip: Git Re-Basin (shared base ⇒ one basin, plain arithmetic works), Fisher merging (underperforms on LLMs). - If averaging LoRA factors: FedEx-LoRA exact aggregation (naive mean(A)·mean(B) ≠ mean(AB)) [ACL 2025].

Re-basing / iterative protocol (DiLoCo cadence): run merging as federated rounds — merge → outer Nesterov step (lr≈0.7, mom 0.9) → fold consensus into base (re-base) → each member retrains a SMALL BOUNDED delta → distill-into-merge → re-merge. H≈500 local steps (H=1000 costs only +2.9% ppl) [DiLoCo 2311.08105]. Magnitude-bound each delta (early-stop/norm-cap; accept a small per-person dip to lift merged macro). At scale, fight 1/T dilution: never flat-merge many iCores — cluster by embedding (T≤8) → merge+distill within cluster → ROUTE across clusters (the fractal-holographic composition story).

Per-tier recipe + canonical artifact. Canonical mergeable artifact = the full task vector τ = θ_ft − θ_base on ONE byte-identical frozen base, stored sparsified (produced at the cloud tier; every tier converts into it). Full-FT has the higher mergeability ceiling but only once magnitude is tamed (regularize + sparsify + SVD-align) — raw overfit full-FT is the E1 worst case; don't ban full-FT, bound it. - LOW (on-device macOS, inference + tiny): bounded LoRA + DARE-TIES + per-layer coeffs (shipped, ≤2–3 packs); else a router over pre-merged cluster-models. No distillation budget. Re-sync at app-update cadence. - MID (workstation / the 4B testbed): DoRA-r64 bounded + DARE-TIES + a light KL distill pass + RegMean + TSV (with guardrail). Where to chase the 0.667 → ~0.85+ jump. - HIGH (cloud, ≥8–9B base): full-FT/DoRA + OSRM orthogonal-subspace FT + DARE-TIES/TSV + full SAMerging distillation + DiLoCo H=500 re-basing + cross-cluster routing. Closest to 0.93.

Structural ceiling caveat: merge recovery is ~80% at 2–3B vs >90% at 8–9B, and instruction-tuned bases merge far better — so part of E1's 0.667 gap is the 4B testbed itself. For the ceiling, the honest path is TALL-masks (localized per-iCore binary masks at inference = soft routing, >99% single-task recovery) — the middle ground between merge and route.

Updated rule of thumb (supersedes §5a's): Interference ∝ delta magnitude, not the averaging rule. Bound every personal delta near the shared frozen base, then close the residual in function space — a short KL distill-into-merge reaches ~96.5% of ceiling. Localize (~1% disjoint regions) + SVD-orthogonalize to make deltas near-additive. As packs grow, signal dilutes ~1/T: cluster (T≤8) → merge+distill → route across clusters. Re-base on a DiLoCo cadence (H≈500). One byte-identical frozen base is non-negotiable; the canonical artifact is τ = θ_ft − θ_base. A static small model is structurally capped below the 8–9B band — for the ceiling, ship localized masks (soft routing), not a flat merge.

Sources: SAMerging 2512.21288 · DiLoCo 2311.08105 · MergeBench/L&S 2505.10833 / 2408.13656 · TSV CVPR2025 2412.00081 · In-the-Wild 2511.21437 · Will-it-Merge 2601.06672 · DELLA 2406.11617 · OSRM ACL2025 · FedEx-LoRA ACL2025 · TALL-masks · KnOTS 2410.19735.

5c. M5-009 sweep result + the fact/skill composition split (2026-06-22)

The interference sweep (4B testbed, synthetic FACT packs) measured retention = merged/ceiling:

Findings: (1) merge-algo buys only single digits (DARE≈sum > TIES) — confirms interference ∝ magnitude, not the rule; (2) LoRA/DoRA under-inject facts (0.478 ceiling) — low-rank bounding backfires for facts; full-FT is needed for fact injection; (3) full-FT-light under-injects too; (4) best = full-FT-heavy+DARE 0.678/73%, still far from 0.934 — the delta×algo axes cannot close it.

The deeper conclusion — composition has the same fact/skill split as storage: pure independent facts are the worst case for weight-merging — once a fact is erased by sign conflict, it cannot be recovered by distilling on other facts (facts don't generalize from a probe set; SAMerging's ~96.5% is on skills/tasks, not memorized facts). Our 73% sits at the ~80% small-model structural recovery band. Therefore:

Facts → compose by RETRIEVAL union (lossless, scales to any N — merge the RAG stores). Skills/behaviors → compose by weight-merge + distill (~ceiling reachable). Weight-merging facts is inherently lossy and is not the mechanism. This is the storage hybrid (RAG=facts, weights=voice/skill) extended to composition.

Implication for the per-tier recipe / M5-010: the canonical mergeable artifact carries skill/voice deltas; factual breadth across iCores is a federated-retrieval concern, not a weight-merge concern.

M5-011 update (2026-06-22) — the variable is CONFLICT, not fact-vs-skill

The skill-composition test (3 substitution-cipher specialists, full-FT, generation eval) upgraded the conclusion. Results (per-char acc): solos ~0.56/0.58/0.68 (ceiling 0.61), but merged sum 0.04 / DARE 0.03 / TIES 0.18 — i.e. 6–29% retention, far worse than facts' 73% — while routing the (tagged) specialists hits the 0.61 ceiling, ~15× the merge.

Why: the 3 ciphers are directly conflicting functions — same input letters → contradictory outputs — so summing their deltas pulls shared weights three contradictory ways (maximal sign-conflict). "Skill" wasn't the point; conflict was. Refined model of mergeability:

Composition mechanism is chosen by delta CONFLICT / orthogonality, not by fact-vs-skill: - Complementary capabilities (different tasks/domains, non-overlapping circuits) → MERGE works (task-arithmetic ~98% @2 in the literature). - Conflicting capabilities (contradictory maps on the same input; competing styles) → MERGE fails → ROUTE (M5-011: routing 0.61 ≫ merge 0.04). - Independent facts → partial collision → RETRIEVAL-union. The dynamic selector should detect conflict and pick merge / route / RAG accordingly — this is exactly the §3.3 spectrum router, now with an empirical conflict signal.

Caveat: the cipher was a doubly-bad choice — hard to learn (solos only ~0.6) and conflicting. A clean merge-positive case needs easy + complementary skills (different task types), which the literature already establishes (~98%); reproducing it locally is optional (M5-011b).

6. Relationship to the rest of HiNet

• Replaces the implicit "frozen anchor + thin LoRA" assumption with a depth spectrum (LoRA → high-rank → DoRA → full-FT) and composition via task-vectors + re-basing — preserving the fractal thesis while allowing genuinely new per-person models.
• Δ carries voice/skill; the vault+RAG carries facts (the hybrid correction). The "coupled twin" Δ_public is the shareable, generalized projection of the deep personal weights.
• Hardware tiers attach to the existing ComputeBackend (where compute runs) and VaultReplication (where the owned weights/data live) seams.

6a. Evaluated, not adopted — Soup (the trainer, not Model Soups)

Soup (Makazhan, 2026; Apache-2.0) is a one-YAML PEFT/TRL CLI whose headline trick is layer streaming: LoRA-train an 8B on a 4 GB CUDA laptop by paging the frozen base one decoder layer at a time (claimed bit-exact vs a resident run; paper doi:10.5281/zenodo.21918325). It is not Wortsman et al. 2022 Model Soups (weight-averaging of fine-tunes) — that paper is already in our merge toolkit.

Verdict (2026-08-24): do not vendor, do not switch the nightly clock to it. Take discipline, not code.

Steal these, implement in our MLX loop (already specified, not yet all built):

1. Eval-gated ship — soup ship refuses a tune that wins the task and quietly regresses tool-calling / refusal / JSON. This is our promote/rollback eval-gate; make the gate a real multi-leg scorer, not a single loss number.
2. Provenance-bound evidence — bind the ship verdict to the exact recipe + data digest (stale evidence → fail). Same spirit as our competence-attestation "can't backdate" rule, applied to the sleep clock.
3. Refuse silent wrong-training — Soup's own postmortem (assistant-only mask trained on tokenizer key strings, healthy loss, zero tokens; flags validated then read by nothing) is the failure mode that matters overnight. Every training flag we expose must have a test that the trainer actually reads it; a healthy loss curve is not evidence of learning.

Layer streaming stays a footnote for a future CUDA ComputeBackend if a user-owned low-VRAM box ever becomes a training tier. It is not on the macOS or iCore Prime path.

Next: RAG architecture →  ·  All documentation →