Documentation  /  Canonical record (HCR)

HiNet Canonical Record (HCR) — the shared normalization format

Status: v1 draft (2026-06-25). The single shape every connector normalizes into and the vault / RAG / training pipeline consume. Formalizes (and supersedes the prose of) SourceRecord in Track 2 spec §7. Implemented in app/osx/hinetd/hinetd/canonical.py (pydantic v2) so it is enforced, not just described — Connector.normalize(raw) -> CanonicalItem.

Design goal: lossless-enough that nothing meaningful from any source is dropped (anything without a typed home lands in source_meta), uniform-enough that downstream code never branches on the source.

1. The envelope — CanonicalItem

2. Sub-models

Participant — display_name?, handles: list[Handle], person_id? (resolved canonical human; None = unresolved), is_self: bool (the owner). Connectors emit handles; the entity-resolver fills person_id later (§5).

Handle — scheme: HandleScheme, value: str. The join key for identity resolution (e.g. two email handles with the same value across Slack + Gmail ⇒ same person_id).

Content — text: str (the canonical body), format: "markdown"|"plain", parts: list[ContentPart] (optional structured breakdown), lang?. Conventions: HTML email → markdown; Slack/Telegram mentions resolved to @DisplayName in text (raw kept in a part); quoted replies rendered as markdown blockquotes.

ContentPart — kind: "text"|"quote"|"code"|"mention"|"link"|"system", text, optional ref (e.g. mentioned handle, url). Lets RAG chunk smartly and training strip system noise.

Attachment — kind: "image"|"file"|"audio"|"video"|"link", filename?, mime?, size?, provider_ref? (source attachment id), vault_ref? (local path if imported), extracted_text? (OCR/text-extraction, opt-in).

ThreadRef — thread_id: str (canonical thread key), reply_to? (provider id of parent), root_id?. The conversation_id lives on the envelope; ThreadRef is the finer reply graph.

Reaction — emoji: str, by: list[Participant], count: int.

Origin — the human-readable where-from: account_label (e.g. me@work (Gmail)), space (Slack workspace / Drive "My Drive" or a shared-drive name), container (Slack #channel / Gmail label / Drive folder / chat title), item_name (file/doc/channel), path (full breadcrumb for display + retrieval), url (deep link back). See §4b.

3. Enums

• SourceType: slack | telegram | whatsapp | gmail | gdrive | local | other
• ItemType: message | email | file | doc | event
• Sensitivity: public | normal | personal | sensitive | secret
• Classification: fact | skill | voice | mixed | route (pipeline-assigned)
• HandleScheme: email | phone | slack_user | telegram_id | telegram_username | whatsapp_phone | google_account | display_name | other

4. Conversation / thread model (unifies chat + email)

• Chat (Slack/Telegram/WhatsApp): conversation_id = channel/chat/group. thread.thread_id = Slack thread_ts (or ts if flat) / Telegram reply-chain root / WhatsApp = conversation. thread.reply_to = parent message id (Telegram reply_to_message_id).
• Email (Gmail): conversation_id = threadId. thread.reply_to = the In-Reply-To Message-ID; thread.root_id = first id in References. Threading is the Message-ID graph.
• Files/docs (Drive): no thread; conversation_id = None.

4b. Provenance (Origin) — kept for RAG, generalized for training

Every record carries origin: the human-readable source location, not just the machine connection_id. It's load-bearing two ways: - RAG keeps it in full. Retrieval is metadata-filtered by origin ("only my #eng Slack", "only work Gmail", "files under Legal/Contracts", "since March") and answers cite it ("from your #eng Slack in Acme, 2024-01-15, Alice said…"). Multiple Slack workspaces / Google accounts stay distinguishable. - Training generalizes or strips it. DeltaView learns the owner's voice/skill, not where it came from — so origin is dropped or mapped to a coarse descriptor ("work chat", "personal email"). Provenance-specific facts stay in RAG, per the fact→RAG / skill→Δ split (Track 2 §4.3): the model never memorizes "this lives in #eng," while RAG can still cite it.

Origin per source:

Multi-space / multi-account is first-class: each Slack workspace and each Google account is a distinct account_label + connection_id, so "work Slack #eng" never collides with "side-project Slack #eng".

5. Identity & entity resolution

Connectors emit Participant.handles (never invent a person_id). A separate resolver (P2/P4) clusters participants into canonical people by strong handles (email, phone) first, then best-effort (display name within a source). is_self is set by matching the owner's known handles (configured at onboarding). Resolution is advisory metadata — records remain usable unresolved.

6. Stable ID + dedupe

• provider_id present → id = "hcr_" + sha256(f"{source}|{account_id}|{provider_id}")[:32].
• provider_id absent (WhatsApp .txt) → synthesize provider_id = sha256(f"{conversation_id}|{source_epoch}|{author_key}|{text}")[:24], then id as above — deterministic so re-imports don't duplicate.
• content_hash = sha256(content.text + "␟" + sorted(attachment provider_refs)). Used for cross-account / cross-source dedupe (e.g. the same email in two Gmail accounts, or a forwarded message) — same content_hash, different id; the vault keeps one canonical + links.

7. Per-source mapping tables (source field → canonical)

Slack (conversations.history message) | source | canonical | note | |---|---|---| | ts | provider_id ({channel}:{ts}) | ts is the per-channel id | | channel | conversation_id | C/D/G id | | thread_ts | thread.thread_id / reply_to | flat msg ⇒ thread_id = ts | | user / bot_id | author.handles[slack_user] (+ email via users.info) | | | text (mrkdwn) | content.text | resolve <@U…>, <#C…|n>, <url|t> | | blocks/rich_text | source_meta.blocks | structured passthrough | | subtype | type=message+source_meta.subtype | service subtypes → ContentPart(kind=system) or skip | | edited | edited_at | | | reactions[] | reactions[] | | | files[] | attachments[] | url_private = provider_ref |

Telegram (export JSON message / Telethon Message) | source | canonical | note | |---|---|---| | id | provider_id ({chat}:{id}) | | | chat id/name | conversation_id | | | reply_to_message_id | thread.reply_to | | | from / from_id | author.handles[telegram_id, telegram_username] + display_name | | | text or text_entities | content.text (+ parts) | flatten entities to text + keep formatting in parts | | date_unixtime | source_epoch→timestamp | prefer epoch over the ISO string | | forwarded_from | source_meta.forwarded_from | | | type=="service" | type=event or ContentPart(system) | joins/pins/etc. | | media + mime_type | attachments[] | metadata; file if opted in |

WhatsApp (Export Chat .txt) | source | canonical | note | |---|---|---| | [date, time] line | timestamp (+ tz_offset unknown) | parse locale variants (DD/MM vs MM/DD, 12h/24h, separators) | | Sender: | author.handles[whatsapp_phone or display_name] | number or name | | message text (multi-line) | content.text | join continuation lines | | <Media omitted> / IMG-….jpg (file attached) | attachments[] | placeholder; real media only if zip imported | | system lines (e2e notice, joins) | ContentPart(system) / type=event, author=None | | | (no native id) | synthesized provider_id | see §6 | | chat/export | conversation_id | per export file |

Gmail (users.messages resource) | source | canonical | note | |---|---|---| | Message-ID header | provider_id | stable across accounts (better than id for dedupe/threading) | | threadId | conversation_id | | | In-Reply-To / References | thread.reply_to / root_id | Message-ID graph | | From / To / Cc | author / participants[email] | | | Subject | subject | | | internalDate (epoch ms) | timestamp | prefer over Date header (tz/spoof) | | payload MIME: text/plain | content.text | if only text/html → convert to markdown | | labelIds[] | labels[] | | | parts w/ filename+attachmentId | attachments[] | | | full headers[] | source_meta.headers | passthrough |

Google Drive (files v3 resource) | source | canonical | note | |---|---|---| | id | provider_id | | | name | subject (title) | | | mimeType | type (doc if native/text-extracted, else file) | application/vnd.google-apps.* → export | | native export / local extract | content.text | Docs→md, Sheets→csv, Slides→text; PDF/docx→extracted | | parents[] | labels (folder path) / conversation_id=None | resolve ids → path | | owners[].emailAddress | author.handles[google_account/email] | | | modifiedTime | timestamp | RFC3339 | | md5Checksum | feeds content_hash (binary) | native docs hash exported text |

8. source_meta — the escape hatch

Any source field without a typed home goes here verbatim (Slack blocks, Gmail full headers, Telegram text_entities, Drive md5Checksum/webViewLink). Stored, not interpreted — so we never lose data and can promote a field to a typed slot later without re-ingesting.

9. How it's used

• Connectors (hinetd plugins): normalize(raw) -> CanonicalItem. They set everything except classification (pipeline) and person_id (resolver).
• Vault: validates against the pydantic model, stores encrypted, indexes content.text / parts for RAG, dedupes by content_hash.
• Training views: DeltaView reads content + classification; RAGView reads chunks.

10. Open decisions (for the owner / next iteration)

1. HTML email — convert to markdown (default) vs keep a raw html part in source_meta? (proposed: markdown body + raw in source_meta.)
2. Media text-extraction default — off (metadata only) until the owner opts a source in. (proposed: off by default.)
3. Entity-resolution aggressiveness — auto-merge on email/phone only (safe) vs also fuzzy display-name within a source. (proposed: strong-handles auto, name fuzzy = suggest-only.)
4. Reactions/edits — store for chat sources, or drop as noise for v1? (proposed: store; cheap and useful for "voice".)

All documentation →