PostgreSQL Archiver Setup¶

This page describes how to run the full archiver stack: a PostgreSQL database for record storage, a RustFS instance as the S3-compatible artifact store, and the PostgresArchiver service, all managed by a single docker compose file.

RustFS is an S3-compatible object store. Its S3 API is used by S3ArtifactStorage to persist numpy arrays. Configure artifact_storage_kwargs for another S3 endpoint such as AWS S3.

Stack overview¶

db: PostgreSQL 16. Stores all scalar record data and artifact reference stubs in the records and identifier_groups tables. Schema is applied automatically from provision.sql on first start.
rustfs: RustFS S3-compatible object store. Numpy arrays are uploaded as .npy objects keyed by UUID and downloaded on get() calls.
archiver: The PostgresArchiver HEROS actor, started via boss.starter. Subscribes to data events on the Zenoh bus, writes to db, and stores arrays in rustfs.

Environment¶

Both db and rustfs read credentials from a shared .env file:

POSTGRES_USER=heros
POSTGRES_PASSWORD=heros
POSTGRES_DB=herosdb

RUSTFS_ACCESS_KEY=heros
RUSTFS_SECRET_KEY=heros

Persistent artifact storage¶

The RustFS data volume must reside on a local, exclusively owned disk or filesystem on the RustFS host. Do not mount CIFS/SMB or NFS as the live RustFS data directory. Use a local filesystem for the running service and copy data to network storage through the S3 API for backup instead. RustFS recommends local SSD/NVMe storage and explicitly advises against network filesystems because of their locking and write-semantics risks; see the RustFS storage recommendations.

Provisioning with Docker Compose¶

The Compose project requires these files in the same directory:

docker-compose.yml
.env
provision.sql

services:
  db:
    image: postgres:16
    restart: unless-stopped
    env_file: .env
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./provision.sql:/docker-entrypoint-initdb.d/provision.sql:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 10

  rustfs:
    image: rustfs/rustfs
    restart: unless-stopped
    env_file: .env
    ports:
      - "9000:9000"   # S3 API
      - "9001:9001"   # web console
    volumes:
      - rustfsdata:/data

  archiver:
    image: registry.gitlab.com/atomiq-project/herostools
    restart: unless-stopped
    network_mode: host
    environment:
      - >
        BOSS1=

        {
          "_id": "archiver_atomiq_data",
          "classname": "herostools.actor.archiver.PostgresArchiver",
          "tags": [
            "experiment:archiver"
          ],
          "arguments": {
            "db_url": "postgresql://heros:heros@localhost/herosdb",
            "identifier_key": "identifier",
            "object_selector": "*",
            "event_name": "emit_data",
            "max_retries": 3,
            "default_metadata": {},
            "artifact_storage_kwargs": {
              "endpoint_url": "http://localhost:9000",
              "bucket": "artifacts",
              "access_key": "heros",
              "secret_key": "heros"
            }
          }
        }
    command: python -m boss.starter -e BOSS1

volumes:
  pgdata:
  rustfsdata:

Start the full stack:

cd examples/archiver/database/
docker compose up -d

The PostgreSQL health check must pass before the schema is considered ready. Watch the database log until database system is ready to accept connections appears:

docker compose logs -f db

In Arcane, upload all three files to the same Compose-project workspace. The read-only bind mount applies provision.sql when PostgreSQL initializes its data directory.

provision.sql is idempotent (CREATE TABLE IF NOT EXISTS, CREATE OR REPLACE FUNCTION), so re-creating the container with the same database volume is safe.

BOSS configuration¶

A single PostgresArchiver actor handles both database writes and artifact storage. The example configuration:

{
  "_id": "archiver_postgres",
  "classname": "herostools.actor.archiver.PostgresArchiver",
  "tags": [
    "experiment:archiver"
  ],
  "arguments": {
    "db_url": "postgresql://heros:heros@localhost/herosdb",
    "identifier_key": "identifier",
    "object_selector": "foobar",
    "event_name": "test_data",
    "max_retries": 3,
    "default_metadata": {},
    "allow_purge": false,
    "artifact_storage_kwargs": {
      "endpoint_url": "http://localhost:9000",
      "bucket": "artifacts",
      "access_key": "heros",
      "secret_key": "heros"
    }
  }
}

The event flow is:

An upstream HERO publishes emit_data.
archiver receives the payload, stores scalar fields to the DB, and writes numpy arrays directly to the RustFS artifact store keyed by UUID.

Parameters

db_url: psycopg connection string, e.g. postgresql://user:pass@host/dbname.
identifier_key (default: “identifier”): Key looked up in the merged payload to determine the record identifier. Metadata is shallow-merged into the payload before the lookup (payload wins on key collision), so the identifier may come from either source.
allow_purge (default: False): Must be set to True to enable the purge() method.
artifact_storage_kwargs (default: local RustFS): Keyword arguments forwarded to S3ArtifactStorage. Omit to connect to the local RustFS instance defined in docker-compose.yml (credentials from .env). Pass null to use InMemoryArtifactStorage (no persistence – useful for local testing without a running S3 backend). For AWS S3 pass {"endpoint_url": null, "bucket": "...", "access_key": "...", "secret_key": "..."}. For S3-backed storage, new artifacts are written to UTC daily buckets named <bucket>-YYYY-MM-DD. The configured bucket remains the fallback for legacy artifact references that do not contain a bucket name. The storage credential must be permitted to create daily buckets, unless they are provisioned externally before their first write.
use_single_bucket (default: False): Store newly written artifacts in the configured static bucket rather than UTC daily buckets. Set this to true for S3 deployments where a single bucket is the preferred operational model. Artifact references that already contain a bucket are always read and purged from that bucket.

Automated cleanup¶

Records accumulate indefinitely unless pruned. The cleanup compose stack runs cleanup() once a day via ofelia. The scheduler executes the job in the running container, so no dedicated cron daemon is required.

The stack consists of two services:

cleanup: Runs the HEROS image with sleep infinity. ofelia execs the cleanup script into this container daily.
scheduler: The ofelia container. Reads job definitions from Docker labels on the cleanup container and triggers them on schedule.

Both services use network_mode: host so they reach the Zenoh router and the running PostgresArchiver without extra network configuration. The Python script is written into the container at startup through the command heredoc. No external script file or volume mount is required.

services:

  cleanup:
    image: registry.gitlab.com/atomiq-project/heros
    environment:
      HERO_NAME: ${HERO_NAME:-my-postgres-archiver}
      RETENTION_DAYS: ${RETENTION_DAYS:-30}
    # write the script once at container startup, then wait for ofelia to exec it
    command:
      - sh
      - -c
      - |
        cat > /cleanup.py << 'PYEOF'
        import logging, os, sys
        from heros import RemoteHERO

        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s %(levelname)s %(message)s",
            stream=sys.stdout,
        )
        log = logging.getLogger(__name__)

        hero_name = sys.argv[1] if len(sys.argv) > 1 else os.environ["HERO_NAME"]
        retention_days = int(os.environ.get("RETENTION_DAYS", "30"))

        log.info(f"connecting to hero '{hero_name}'")
        remote = RemoteHERO(hero_name)

        expired = remote.cleanup(retention_days=retention_days, dry_run=False)

        log.info(f"purged {len(expired)} record(s)")
        for identifier in expired:
            log.info(f"  purged: {identifier}")
        PYEOF
        sleep infinity
    labels:
      ofelia.enabled: "true"
      ofelia.job-exec.cleanup.schedule: "@every 24h"
      ofelia.job-exec.cleanup.command: /usr/local/bin/python /cleanup.py
    network_mode: host
    restart: unless-stopped

  scheduler:
    image: mcuadros/ofelia:latest
    # required: without this ofelia exits with "no config file" on startup
    command: daemon --docker
    depends_on:
      - cleanup
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    network_mode: host
    restart: unless-stopped

Start the stack:

cd examples/archiver/cleanup/
HERO_NAME=my-postgres-archiver RETENTION_DAYS=30 docker compose up -d