==========================
PostgreSQL Archiver Setup
==========================

This page describes how to run the full archiver stack: a PostgreSQL database
for record storage, a RustFS instance as the S3-compatible artifact store, and
the ``PostgresArchiver`` service, all managed by a single
``docker compose`` file.

RustFS is an S3-compatible object store. Its S3 API is used by
``S3ArtifactStorage`` to persist numpy arrays. Configure
``artifact_storage_kwargs`` for another S3 endpoint such as AWS S3.


Stack overview
--------------

``db``
    PostgreSQL 16. Stores all scalar record data and artifact reference stubs
    in the ``records`` and ``identifier_groups`` tables. Schema is applied
    automatically from ``provision.sql`` on first start.

``rustfs``
    RustFS S3-compatible object store. Numpy arrays are uploaded as ``.npy``
    objects keyed by UUID and downloaded on ``get()`` calls.

``archiver``
    The ``PostgresArchiver`` HEROS actor, started via ``boss.starter``.
    Subscribes to data events on the Zenoh bus, writes to ``db``, and stores
    arrays in ``rustfs``.


Environment
-----------

Both ``db`` and ``rustfs`` read credentials from a shared ``.env`` file:

.. literalinclude:: ../../../../examples/archiver/database/.env
   :language: bash


Persistent artifact storage
---------------------------

The RustFS ``data`` volume must reside on a local, exclusively owned disk or
filesystem on the RustFS host. Do not mount CIFS/SMB or NFS as the live RustFS
data directory. Use a local filesystem for the running service and copy data to
network storage through the S3 API for backup instead. RustFS recommends local
SSD/NVMe storage and explicitly advises against network filesystems because of
their locking and write-semantics risks; see the `RustFS storage recommendations
<https://docs.rustfs.com/installation/docker/>`_.


Provisioning with Docker Compose
--------------------------------

The Compose project requires these files in the same directory:

- ``docker-compose.yml``
- ``.env``
- ``provision.sql``

.. literalinclude:: ../../../../examples/archiver/database/docker-compose.yml
   :language: yaml

Start the full stack:

.. code:: bash

    cd examples/archiver/database/
    docker compose up -d

The PostgreSQL health check must pass before the schema is considered ready.
Watch the database log until ``database system is ready to accept connections``
appears:

.. code:: bash

    docker compose logs -f db

In Arcane, upload all three files to the same Compose-project workspace. The
read-only bind mount applies ``provision.sql`` when PostgreSQL initializes its
data directory.

``provision.sql`` is idempotent (``CREATE TABLE IF NOT EXISTS``,
``CREATE OR REPLACE FUNCTION``), so re-creating the container with the same
database volume is safe.


BOSS configuration
------------------

A single ``PostgresArchiver`` actor handles both database writes and artifact
storage. The example configuration:

.. literalinclude:: ../../../../examples/archiver/database/archiver_postgres.json
   :language: json

The event flow is:

1. An upstream HERO publishes ``emit_data``.
2. ``archiver`` receives the payload, stores scalar fields to the DB, and
   writes numpy arrays directly to the RustFS artifact store keyed by UUID.

**Parameters**

``db_url``
    psycopg connection string, e.g. ``postgresql://user:pass@host/dbname``.

``identifier_key`` *(default: "identifier")*
    Key looked up in the merged payload to determine the record identifier.
    Metadata is shallow-merged into the payload before the lookup (payload wins
    on key collision), so the identifier may come from either source.

``allow_purge`` *(default: False)*
    Must be set to ``True`` to enable the
    :py:meth:`~herostools.actor.archiver.PostgresDatabase.purge` method.

``artifact_storage_kwargs`` *(default: local RustFS)*
    Keyword arguments forwarded to ``S3ArtifactStorage``. Omit to connect to
    the local RustFS instance defined in ``docker-compose.yml`` (credentials
    from ``.env``). Pass ``null`` to use ``InMemoryArtifactStorage`` (no
    persistence -- useful for local testing without a running S3 backend).
    For AWS S3 pass ``{"endpoint_url": null, "bucket": "...", "access_key":
    "...", "secret_key": "..."}``. For S3-backed storage, new artifacts are
    written to UTC daily buckets named ``<bucket>-YYYY-MM-DD``. The configured
    ``bucket`` remains the fallback for legacy artifact references that do not
    contain a bucket name. The storage credential must be permitted to create
    daily buckets, unless they are provisioned externally before their first
    write.

``use_single_bucket`` *(default: False)*
    Store newly written artifacts in the configured static ``bucket`` rather
    than UTC daily buckets. Set this to ``true`` for S3 deployments where a
    single bucket is the preferred operational model. Artifact references that
    already contain a bucket are always read and purged from that bucket.


.. _auto-cleanup:

Automated cleanup
-----------------

Records accumulate indefinitely unless pruned. The cleanup compose stack runs
:py:meth:`~herostools.actor.archiver.PostgresDatabase.cleanup` once a day via
`ofelia <https://github.com/mcuadros/ofelia>`_. The scheduler executes the job
in the running container, so no dedicated cron daemon is required.

The stack consists of two services:

``cleanup``
    Runs the HEROS image with ``sleep infinity``. ofelia execs the cleanup
    script into this container daily.

``scheduler``
    The ofelia container. Reads job definitions from Docker labels on the
    ``cleanup`` container and triggers them on schedule.

Both services use ``network_mode: host`` so they reach the Zenoh router and the
running ``PostgresArchiver`` without extra network configuration. The Python
script is written into the container at startup through the ``command``
heredoc. No external script file or volume mount is required.

.. literalinclude:: ../../../../examples/archiver/cleanup/docker-compose.yml
   :language: yaml

Start the stack:

.. code:: bash

    cd examples/archiver/cleanup/
    HERO_NAME=my-postgres-archiver RETENTION_DAYS=30 docker compose up -d