# YachtMemory Source Catalog — protocol v2 (draft 0.1.0)

**Scope:** Design and reference implementation, not yet a production wire contract. No changes to installed v1 receiver/controller, schema or onboard source discovery.

## Identity and records

- `vessel_id`: stable server-assigned UUID; receiver MUST bind it to the authenticated origin and to the registered telemetry vessel.
- `snapshot_id`: newly allocated random UUID per discovery cycle, stable across retries.
- `revision`: monotonic per vessel, allocated transactionally and persisted on board before outbox publishing; do not reuse a revision for different bytes.
- `discovered_at_utc`: UTC timestamp; not a substitute for revision ordering.
- Snapshot scope: `current-selected-sources`; inventory coverage: `selected-only`. These may be expanded **only by future protocol versions**. All chunks arriving is **transport completeness**, NOT proof of all physically connected sensors.
- Records represent `path`, `observation` and `source`. Unknown paths are accepted. Source metadata and path metadata are optional; `null` unit is meaningful. This first draft supports value types `number|string|boolean|position|object|array|null|unknown`; type detection must not assume all objects are positions.
- Paths describe measurement metadata; observations describe path→source; source records describe interface/device metadata. Device provenance and competing sources are future extensions.

## Envelope

A manifest (`header`) is an outbox entity `signalk_catalog_manifest_v2` and each chunk is a separate entity `signalk_catalog_chunk_v2`. This is a **proposed** wire layout; existing production receiver supports neither.

Manifest required fields: `protocol`, `vessel_id`, `snapshot_id`, `revision`, `discovered_at_utc`, `scope`, `inventory_coverage`, `transport_complete`, `chunk_count`, `record_count`, `document_sha256`.

Chunk required fields: `protocol`, `vessel_id`, `snapshot_id`, `revision`, `chunk_index` (zero based), `records`, `payload_sha256`.

Canonical payload: UTF-8 JSON, keys lexicographically sorted, compact separators, NaN forbidden. `payload_sha256` is the SHA-256 hex of the canonical JSON `{ "records": [...] }` of **that chunk**. `document_sha256` is SHA-256 of the same JSON shape containing all records in canonical global order, not of the manifest or its timestamps. Do not hash the raw MySQL JSON representation: MySQL may reserialize it. Preserve the original canonical text or bytes in the Outbox and verify before insertion.

Hard draft limits: 256 chunks; 100 records per chunk; 16,384 bytes per canonical chunk *body*; 25,600 records per snapshot. The complete outbox wrapper must also fit the existing 32,768-byte receiver request cap. No silent truncation: oversized observations fail closed until explicitly supported.

## Completion, retries, and atomic activation

1. Onboard writer must construct and persist the immutable manifest, complete chunk set and outbox entries in a **single database transaction**. It MUST NOT emit a complete manifest if it has seen any scan truncation or pagination uncertainty. With MySQL/MariaDB DDL operations, separate migration checks are required; no transaction claim is made for DDL.
2. Chunks and manifest may arrive in any order. Server accepts chunks only for the claimed vessel/snapshot and checks per-chunk hashes, indices and consistent revision. Missing chunks leave snapshot in `staging`; the currently active catalog is unchanged.
3. Retried identical message + matching outbox receipt => `already_received`; duplicate `(snapshot_id,chunk_index)` with different content => conflict, no receipt. Chunk receipt acknowledges **durable staging**, not activation. Server may need retryable reconciliation if crash occurs between final receipt and activation.
4. A separate resumable `finalize` operation checks exact index set, record count, per-chunk and document hash, uniqueness, schema-validity and source-path integrity. It may run repeatedly. The finalization must serialize per vessel (e.g. vessel row lock), reject same-revision conflicts, and disregard older revisions without overwriting the active catalog.
5. Activation writes path/observation/source inventory and flips active pointer **inside one InnoDB transaction**. Prior first/last-seen histories are preserved. For coverage `selected-only`, `currently_observed` may only describe *currently selected* Signal K paths/sources, never physically absent equipment. An empty metadata list is not evidence that devices were removed.
6. ACK or SSH loss after server commit is harmless: resends reuse same content hashes. A failure after staging but before activation is recovered via replay/finalize. Unrecognized entity types are **not sent to the existing production receiver** until controller and receiver support them.
7. Exhaustion, aging, cleanup, rate limits, secure authorization, and snapshot garbage collection require a separate reviewed policy before production. Data volume must be bounded and the protocol must reject unknown extra capabilities (such as all-sources coverage) until explicitly tested.

## Example

See `example_header.json` and `example_chunks.json`, generated from the reference codec. Examples have **synthetic** UUIDs/records. No credentials.

## Compatibility

The existing source catalog v1 tables (`signalk_catalog_state`, `signalk_discovery_snapshots`, `signalk_discovered_paths`, `signalk_discovered_sources`) remain unchanged. The v2 schema uses `_v2` suffix, deliberately avoids `USE yachtmemory_telemetry` and must only be executed in a dedicated test schema. The previous v1 integration tests remain valid for v1 but do not prove v2 production readiness.