==============
Data Structure
==============

This page describes the database schema, how incoming payloads are processed,
and the shallow-merge behaviour that governs how successive events for the same
identifier are combined.


Database tables
---------------

``records``
    One row per identifier. ``data_json`` is a JSONB column that accumulates
    the merged payload across all events received for that identifier. A GIN
    index on ``data_json`` supports efficient containment and existence queries
    (``@>``, ``?``). ``updated_at`` is refreshed on every write.

``identifier_groups``
    Post-hoc tag table. Stores ``(identifier, tag_key, tag_value)`` triples.
    The composite primary key ``(identifier, tag_key)`` enforces one value per
    tag key per identifier. Foreign-keyed to ``records``; rows are removed
    automatically when the parent record is purged.

The full schema is in ``examples/archiver/database/provision.sql`` and is
applied automatically by docker-compose on first start.


Payload ingestion
-----------------

Each incoming event carries a ``(payload, metadata)`` pair. Before the
database write, the archiver builds a single merged dict:

1. ``default_metadata`` (from BOSS config) is merged first.
2. The event ``metadata`` overwrites it.
3. The ``payload`` overwrites that (payload keys always win).

The value at ``identifier_key`` (default ``"identifier"``) is extracted from
this merged dict and used as the row key. If the key is absent the record is
dropped with an error log.

Numpy arrays found anywhere in the merged dict are extracted, replaced by
artifact reference stubs (see `Artifact references`_ below), and stored in the
configured S3 backend before the database write.


Shallow merge
-------------

Each event is merged into the existing ``data_json`` row using the PostgreSQL
``||`` operator:

.. code:: sql

    SET data_json = COALESCE(records.data_json, '{}'::jsonb) || EXCLUDED.data_json

``||`` is a **top-level shallow merge**: keys from the incoming payload
overwrite matching keys in the stored row; keys absent from the incoming
payload are preserved. Nested dicts are replaced wholesale, not recursively
merged.

Example: three events for the same identifier and the accumulated state after
each write.

.. code:: python

    # event 1
    {"a": 1, "b": 2}
    # -> data_json: {"a": 1, "b": 2}

    # event 2  (b overlaps, c is new)
    {"b": 99, "c": 3}
    # -> data_json: {"a": 1, "b": 99, "c": 3}

    # event 3  (nested dict is new)
    {"nested": {"x": 1, "y": 2}}
    # -> data_json: {"a": 1, "b": 99, "c": 3, "nested": {"x": 1, "y": 2}}

If event 4 sends ``{nested: {x: 10}}``, the result is
``{..., nested: {x: 10}}``. The ``y`` key inside ``nested`` is removed because
the inner dict is replaced rather than merged.

A mapping that contains several frames is also one top-level value. A later
event for that key replaces the complete mapping. To retain frames across
independent events, use one top-level key per frame. If frames are sent in one
mapping, each later update must include every frame that remains in the record.

.. note::

   Use distinct top-level keys across events to accumulate data without
   collision (e.g. ``value_0``, ``value_1``, ...). Use a single overlapping
   key deliberately to keep only the latest value (e.g. ``last_event``).


Artifact references
-------------------

When a dict payload contains a numpy array at any depth, the archiver replaces
it with a lightweight reference stub before writing to the database:

.. code:: json

    {
      "identifier": "shot_0001",
      "temperature": 23.4,
      "image": {
        "__type__": "artifact_ref",
        "artifact_id": "a7f3e2c1-...",
        "size_bytes": 8388608,
        "bucket": "artifacts-2026-08-04",
        "shape": [2048, 2048],
        "dtype": "<u2"
      }
    }

The ``artifact_id`` is a UUID generated at ingest time. ``size_bytes`` is the
decoded NumPy element-data size, ``shape`` is the decoded array shape, and
``dtype`` is ``array.dtype.str``. New artifacts are written to a UTC daily
bucket named ``<configured-bucket>-YYYY-MM-DD``; the stub stores that exact
bucket. Set ``use_single_bucket=True`` to store new artifacts in the configured
static bucket instead. Absent legacy bucket fields use the configured static
bucket. The array is PUT to the S3 store under its UUID key. A later event with
the top-level key ``"image"`` overwrites the stub with a new
``artifact_id``. The previous S3 object becomes unreferenced.
Normal record purge removes only artifacts referenced by the final record state;
orphan collection is outside the archiver contract.

The dot-separated key path within the original payload is recorded as the
``artifact_name`` (e.g. ``"image"``, ``"sensor.spectrum"``,
``"readings.0"``). This name is not stored in the database; it is only relevant
if you inspect the S3 objects directly.

To lazily retrieve a stub discovered through ``get(...,
resolve_artifacts=False)``, call ``get_artifact(artifact_id, bucket)`` on the
archiver. ``get_artifacts([(artifact_id, bucket), ...])`` retrieves at most 100
locations per call by default; configure ``max_artifacts_per_batch`` on the
artifact storage to change this operational bound.