Data Structure¶

This page describes the database schema, how incoming payloads are processed, and the shallow-merge behaviour that governs how successive events for the same identifier are combined.

Database tables¶

records: One row per identifier. data_json is a JSONB column that accumulates the merged payload across all events received for that identifier. A GIN index on data_json supports efficient containment and existence queries (@>, ?). updated_at is refreshed on every write.
identifier_groups: Post-hoc tag table. Stores (identifier, tag_key, tag_value) triples. The composite primary key (identifier, tag_key) enforces one value per tag key per identifier. Foreign-keyed to records; rows are removed automatically when the parent record is purged.

The full schema is in examples/archiver/database/provision.sql and is applied automatically by docker-compose on first start.

Payload ingestion¶

Each incoming event carries a (payload, metadata) pair. Before the database write, the archiver builds a single merged dict:

default_metadata (from BOSS config) is merged first.
The event metadata overwrites it.
The payload overwrites that (payload keys always win).

The value at identifier_key (default "identifier") is extracted from this merged dict and used as the row key. If the key is absent the record is dropped with an error log.

Numpy arrays found anywhere in the merged dict are extracted, replaced by artifact reference stubs (see Artifact references below), and stored in the configured S3 backend before the database write.

Shallow merge¶

Each event is merged into the existing data_json row using the PostgreSQL || operator:

SET data_json = COALESCE(records.data_json, '{}'::jsonb) || EXCLUDED.data_json

|| is a top-level shallow merge: keys from the incoming payload overwrite matching keys in the stored row; keys absent from the incoming payload are preserved. Nested dicts are replaced wholesale, not recursively merged.

Example: three events for the same identifier and the accumulated state after each write.

# event 1
{"a": 1, "b": 2}
# -> data_json: {"a": 1, "b": 2}

# event 2  (b overlaps, c is new)
{"b": 99, "c": 3}
# -> data_json: {"a": 1, "b": 99, "c": 3}

# event 3  (nested dict is new)
{"nested": {"x": 1, "y": 2}}
# -> data_json: {"a": 1, "b": 99, "c": 3, "nested": {"x": 1, "y": 2}}

If event 4 sends {nested: {x: 10}}, the result is {..., nested: {x: 10}}. The y key inside nested is removed because the inner dict is replaced rather than merged.

A mapping that contains several frames is also one top-level value. A later event for that key replaces the complete mapping. To retain frames across independent events, use one top-level key per frame. If frames are sent in one mapping, each later update must include every frame that remains in the record.

Note

Use distinct top-level keys across events to accumulate data without collision (e.g. value_0, value_1, …). Use a single overlapping key deliberately to keep only the latest value (e.g. last_event).

Artifact references¶

When a dict payload contains a numpy array at any depth, the archiver replaces it with a lightweight reference stub before writing to the database:

{
  "identifier": "shot_0001",
  "temperature": 23.4,
  "image": {
    "__type__": "artifact_ref",
    "artifact_id": "a7f3e2c1-...",
    "size_bytes": 8388608,
    "bucket": "artifacts-2026-08-04",
    "shape": [2048, 2048],
    "dtype": "<u2"
  }
}

The artifact_id is a UUID generated at ingest time. size_bytes is the decoded NumPy element-data size, shape is the decoded array shape, and dtype is array.dtype.str. New artifacts are written to a UTC daily bucket named <configured-bucket>-YYYY-MM-DD; the stub stores that exact bucket. Set use_single_bucket=True to store new artifacts in the configured static bucket instead. Absent legacy bucket fields use the configured static bucket. The array is PUT to the S3 store under its UUID key. A later event with the top-level key "image" overwrites the stub with a new artifact_id. The previous S3 object becomes unreferenced. Normal record purge removes only artifacts referenced by the final record state; orphan collection is outside the archiver contract.

The dot-separated key path within the original payload is recorded as the artifact_name (e.g. "image", "sensor.spectrum", "readings.0"). This name is not stored in the database; it is only relevant if you inspect the S3 objects directly.

To lazily retrieve a stub discovered through get(..., resolve_artifacts=False), call get_artifact(artifact_id, bucket) on the archiver. get_artifacts([(artifact_id, bucket), ...]) retrieves at most 100 locations per call by default; configure max_artifacts_per_batch on the artifact storage to change this operational bound.