Documentation / The sim-to-real gap
HiNet — Sim-to-Real Gap (how far our experiments are from real user data)
Status: honest assessment (2026-06-22). Companion to E1-Experiment-Design and Personalization-Engine. Linked from the Whitepaper.
One-line answer: our experiments validate mechanisms (does fine-tuning inject owner data? does composition work, and how — fact vs skill?) at a small, clean, controlled, single-shot scale. They do not yet validate the real-data reality — messy mixed data, the fact/skill split in practice, real volume, continual learning/forgetting, privacy/leakage, production model size, routing-at-scale, and (the biggest) whether the result actually feels like you and recalls your life. The gap on that last axis is large, and it's exactly what Track 2 / E2-on-real-data is built to start closing.
1. What the sims DID validate (high confidence, mechanism-level)
- Fine-tuning injects owner data — bf16 full-FT learned invented facts to ~0.97 recall (4-bit can't; ≥8B/cloud does it better). [E1 §6]
- Composition works — independently fine-tuned iCores merge into one model that beats the best single (+11 pp), and routing reaches a higher ceiling. [E1 §6]
- The fact/skill composition split — weight-merging facts is structurally lossy (~73%); skills (generalizing rules) merge far better (M5-011). → facts compose by retrieval-union, skills by weight-merge+distill. [Personalization-Engine §5b–§5c]
- The depth↔composability trade-off & dynamic-selection mechanism — right-size → route vs merge → re-base, with knobs (delta magnitude, merge method). [Plan §3.3]
These are real, transferable conclusions. They tell us what to build. They do not tell us it works on a real person's data.
2. The gaps to real user data
| # | Gap | Our sim | Real user data | Severity | Closing experiment |
|---|---|---|---|---|---|
| 1 | Data messiness | clean, uniform synthetic facts/cipher words | emails/chats/notes/docs/code — noisy, threaded, redundant, contradictory, multi-format, multilingual, OCR errors | High | the ingestion→normalize→training-views pipeline on a real corpus (Track 2) |
| 2 | Fact ⟷ skill entanglement | facts and skills tested in isolation | one messy stream mixing facts (contacts, events), voice/style, preferences, tacit context — not pre-labeled | High | a classifier that routes content to RAG (facts) vs weights (skill); test on real data |
| 3 | Eval validity | exact-match accuracy on clean tasks | "is it more me? does it recall my life? more helpful to me?" — open-ended, subjective, no ground truth | Highest | owner-in-the-loop preference eval + personal-recall probes (E2); the metric problem is the hardest |
| 4 | Continual / forgetting | one-shot training | nightly continual updates over months → catastrophic forgetting, drift, staleness | High | multi-cycle sleep-cycle sim (SuRe fast/slow+replay+EMA) over a simulated/real 30–90-day stream |
| 5 | Privacy / leakage | no real PII | sensitive personal data → public-twin derivation, leakage gate, scrubbing, DP all untested | High | leakage-gate eval (canaries + membership-inference) on real-style data before any sharing |
| 6 | Volume / long-tail | 40 facts / 400 words | GBs, millions of tokens, long-tailed (few important, vast mundane) | Medium | injection/forgetting scaling on a realistic-size personal corpus |
| 7 | Headroom realism | engineered chance-level anchor | base already knows general knowledge; personalization is a small delta over a strong prior — harder to detect | Medium | measure personalization lift on real owner-specific queries, not invented facts |
| 8 | Routing at scale | 3 orthogonal, hand-tagged domains; oracle routing for the ceiling | thousands of overlapping/correlated people; the router must infer scope from messy queries | High | build + eval the real competence-signature router (M5-005) — now specified (Network-Routing-and-Aggregation); build + eval still pending, routing accuracy the live open risk |
| 9 | Model size / structure | 4B testbed (structural recovery cap) | 30B / cloud production base (>90% recovery band) | Medium | re-run the key results at 30B / production scale |
| 10 | Heterogeneity / drift | one base, one snapshot | iCores across hardware tiers, base versions, anchor drift over time | Medium | cross-version composition + anchor re-basing migration tests |
| 11 | Adversarial / quality | correct synthetic data | owner errors, contradictions, stale/wrong info → model can learn wrong things | Medium | eval-gate robustness on noisy/contradictory real data |
3. The three gaps that matter most
- Eval validity (no ground truth for "is it me"). Every sim number is exact-match on a task with a known answer. Real personalization has no such answer key. Until we have an owner-in-the-loop preference/recall eval, we cannot claim the product works — only that the mechanism does. This is the single most important thing to build.
- Fact/skill entanglement in messy data. Our clean conclusion (facts→RAG, skills→weights) assumes we can tell them apart. Real data doesn't come labeled. The classifier/router that decides "this goes to retrieval, this goes to the weight-delta" is unbuilt and untested.
- Continual learning / forgetting. A real iCore trains every night for months. We've only done one-shot runs. The sleep-cycle dynamics (forgetting, drift, replay sufficiency) are where a personalized model most plausibly degrades over time — and we have zero real evidence yet.
4. Bottom line + what closes the gap
- Trust the sims for: architecture decisions — that fine-tuning injects, that composition works, how fact vs skill differ, the trade-off knobs. These are mechanism truths.
- Do NOT yet trust: any claim about real-world personalization quality, continual stability, privacy safety, or routing accuracy at scale. The sims are silent on these.
- The first real-data probe (E2-on-real-data, Track 2): run the hybrid iCore on the developer's own Mac data (files + mail) — RAG for facts, a style-LoRA for voice — with (a) an owner-judged eval, (b) a multi-cycle sleep loop, and (c) the leakage gate. That single experiment touches gaps #1–5 at once and is the highest-value next real-world step.
Honest framing for any external claim: "We have validated the mechanisms of owned, composable personal models at small scale; we have not yet validated them on real personal data, and the eval/continual/privacy realities are open." Say that, not more.
Next: Node packaging & compute backend → · All documentation →