herostools.actor.archiver.artifact_storage¶

Attributes¶

DEFAULT_MAX_ARTIFACTS_PER_BATCH

Exceptions¶

ArtifactReadError

Raised when an artifact cannot be read from its configured storage location.

Classes¶

`ArtifactStorage`	Abstract base for artifact key-value stores.
`InMemoryArtifactStorage`	Dict-backed in-memory ArtifactStorage — useful for testing and dry runs.
`S3ArtifactStorage`	ArtifactStorage backed by any S3-compatible object store.

Module Contents¶

herostools.actor.archiver.artifact_storage.DEFAULT_MAX_ARTIFACTS_PER_BATCH = 100¶

exception herostools.actor.archiver.artifact_storage.ArtifactReadError(artifact_id: str, bucket: str | None)[source]¶

Bases: RuntimeError

Raised when an artifact cannot be read from its configured storage location.

Parameters:

artifact_id – Identifier of the artifact that could not be read.
bucket – Effective bucket used for the read.

artifact_id¶

bucket¶

class herostools.actor.archiver.artifact_storage.ArtifactStorage(max_artifacts_per_batch: int = DEFAULT_MAX_ARTIFACTS_PER_BATCH)[source]¶

Bases: abc.ABC

Abstract base for artifact key-value stores.

Implementations must support put/get/remove by string artifact_id and optionally list all stored IDs. All mutating methods accept an optional bucket override; when omitted they use the backend’s default bucket.

_max_artifacts_per_batch = 100¶

property default_bucket: str | None¶: Default bucket name for this storage backend, or None if not applicable.

abstractmethod put(artifact_id: str, data: numpy.ndarray, bucket: str | None = None) → str | None[source]¶

Store a numpy array under the given artifact_id.

Parameters:

artifact_id – Unique key (UUID string).
data – Array to store.
bucket – Bucket override; uses backend default when None.

Returns:

The effective bucket name used for storage, or None for backends without a bucket concept (e.g. in-memory).

abstractmethod get(artifact_id: str, bucket: str | None = None) → numpy.ndarray[source]¶

Retrieve a numpy array by artifact_id.

Parameters:

artifact_id – Key to look up.
bucket – Bucket override; uses backend default when None.

Returns:

The stored numpy array.

abstractmethod remove(artifact_id: str, bucket: str | None = None) → None[source]¶

Delete the artifact with the given artifact_id.

Parameters:

artifact_id – Key to delete.
bucket – Bucket override; uses backend default when None.

abstractmethod list(bucket: str | None = None) → list[str][source]¶

Return all artifact_ids currently stored.

Parameters:: bucket – Bucket override; uses backend default when None.
Returns:: List of artifact_id strings.

get_many(artifact_ids: list[str], bucket: str | None = None) → list[numpy.ndarray][source]¶

Retrieve multiple arrays by artifact_id, in the same order as the input.

Parameters:

artifact_ids – Keys to look up.
bucket – Bucket override forwarded to each get() call.

Returns:

List of numpy arrays in the same order as artifact_ids.

get_artifact(artifact_id: str, bucket: str | None = None) → numpy.ndarray[source]¶

Retrieve one decoded artifact by its ID and optional bucket.

Parameters:

artifact_id – Identifier of the artifact to retrieve.
bucket – Bucket override; uses default_bucket when None.

Returns:

The decoded numpy array.

Raises:

ArtifactReadError – If the artifact cannot be read.

get_artifacts(locations: collections.abc.Sequence[tuple[str, str | None]]) → collections.abc.Mapping[str, numpy.ndarray][source]¶

Retrieve a bounded set of artifacts addressed by ID and bucket.

The returned mapping is keyed only by artifact ID. An ID may therefore occur at one effective bucket only within a request. Identical locations are read once and represented once in the result.

Parameters:

locations – Artifact ID and optional bucket pairs.

Returns:

Mapping of artifact IDs to decoded numpy arrays.

Raises:

ValueError – If the request exceeds the batch limit or one artifact ID is requested from multiple effective buckets.
ArtifactReadError – If an artifact cannot be read.

ensure_bucket(bucket: str | None) → None[source]¶

Ensure a bucket is available for writes when the backend requires it.

Parameters:: bucket – Bucket to ensure, or None for backends without buckets.

put_many(artifact_ids: list[str], arrays: list[numpy.ndarray], bucket: str | None = None) → None[source]¶

Store multiple arrays under their respective artifact_ids.

Parameters:

artifact_ids – Unique keys.
arrays – Arrays to store, paired with artifact_ids by position.
bucket – Bucket override forwarded to each put() call.

remove_many(artifact_ids: list[str], bucket: str | None = None) → None[source]¶

Delete multiple artifacts by artifact_id.

Parameters:

artifact_ids – Keys to delete.
bucket – Bucket override forwarded to each remove() call.

class herostools.actor.archiver.artifact_storage.InMemoryArtifactStorage(max_artifacts_per_batch: int = DEFAULT_MAX_ARTIFACTS_PER_BATCH)[source]¶

Bases: ArtifactStorage

Dict-backed in-memory ArtifactStorage — useful for testing and dry runs.

The bucket parameter is accepted on all methods for interface compatibility but is ignored; all artifacts share a single in-memory dict regardless of bucket.

_store: dict[str, numpy.ndarray]¶

put(artifact_id: str, data: numpy.ndarray, bucket: str | None = None) → str | None[source]¶

Store array in memory.

Parameters:

artifact_id – Unique key.
data – Array to store.
bucket – Ignored.

Returns:

None (no bucket concept for in-memory storage).

get(artifact_id: str, bucket: str | None = None) → numpy.ndarray[source]¶

Retrieve array from memory.

Parameters:

artifact_id – Key to look up.
bucket – Ignored.

Returns:

The stored numpy array.

remove(artifact_id: str, bucket: str | None = None) → None[source]¶

Remove array from memory.

Parameters:

artifact_id – Key to delete.
bucket – Ignored.

list(bucket: str | None = None) → list[str][source]¶

Return all stored keys.

Parameters:: bucket – Ignored.
Returns:: List of artifact_id strings.

class herostools.actor.archiver.artifact_storage.S3ArtifactStorage(endpoint_url: str | None = 'http://localhost:9000', bucket: str = 'artifacts', access_key: str = 'heros', secret_key: str = 'heros', max_workers: int = 10, max_artifacts_per_batch: int = DEFAULT_MAX_ARTIFACTS_PER_BATCH, create_buckets: bool = True)[source]¶

Bases: ArtifactStorage

ArtifactStorage backed by any S3-compatible object store.

Numpy arrays are serialised via np.save/np.load into BytesIO and stored as S3 objects keyed by artifact_id (UUID string). The bucket is created on first use if it does not already exist.

Defaults point to the RustFS instance defined in docker-compose.yml (endpoint http://localhost:9000, credentials rustfs/rustfsdev).

To use AWS S3, pass endpoint_url=None and supply real IAM credentials:

S3ArtifactStorage(
    endpoint_url=None,
    bucket="my-bucket",
    access_key="MY_ACCESS_KEY",
    secret_key="MY_SECRET_KEY",
)

Parameters:

endpoint_url – S3 API URL for self-hosted backends, e.g. "http://localhost:9000" for local RustFS. Pass None to use the default AWS S3 endpoint.
bucket – Default bucket name.
access_key – Access key id (RUSTFS_ACCESS_KEY for RustFS, AWS access key id for AWS S3).
secret_key – Secret access key (RUSTFS_SECRET_KEY for RustFS, AWS secret access key for AWS S3).
create_buckets – Create missing buckets on construction and before writes. Set False for read-only clients.

_bucket = 'artifacts'¶

_max_workers = 10¶

_create_buckets = True¶

_known_buckets: set[str]¶

_bucket_lock¶

_client¶

property default_bucket: str¶: Default bucket name configured at construction time.

ensure_bucket(bucket: str | None) → None[source]¶

Ensure an S3 bucket exists, tolerating concurrent creation.

Parameters:: bucket – Bucket to ensure. None uses default_bucket.
Raises:: ClientError – If the bucket cannot be inspected or created.

put(artifact_id: str, data: numpy.ndarray, bucket: str | None = None) → str[source]¶

Serialise array to npy bytes and upload as an S3 object.

Parameters:

artifact_id – S3 object key.
data – Array to store.
bucket – Bucket override; uses default_bucket when None.

Returns:

The effective bucket name used for storage.

get(artifact_id: str, bucket: str | None = None) → numpy.ndarray[source]¶

Download and deserialise an S3 object to a numpy array.

Parameters:

artifact_id – S3 object key.
bucket – Bucket override; uses default_bucket when None.

Returns:

The stored numpy array.

remove(artifact_id: str, bucket: str | None = None) → None[source]¶

Delete an S3 object.

Parameters:

artifact_id – S3 object key to delete.
bucket – Bucket override; uses default_bucket when None.

list(bucket: str | None = None) → list[str][source]¶

Return all object keys in the bucket via paginated listing.

Parameters:: bucket – Bucket override; uses default_bucket when None.
Returns:: List of artifact_id strings.

get_many(artifact_ids: list[str], bucket: str | None = None) → list[numpy.ndarray][source]¶

Download and deserialise multiple S3 objects in parallel.

Parameters:

artifact_ids – S3 object keys to fetch.
bucket – Bucket override forwarded to each get() call.

Returns:

List of numpy arrays in the same order as artifact_ids.

put_many(artifact_ids: list[str], arrays: list[numpy.ndarray], bucket: str | None = None) → None[source]¶

Serialise and upload multiple arrays to S3 in parallel.

Parameters:

artifact_ids – S3 object keys.
arrays – Arrays to store, paired with artifact_ids by position.
bucket – Bucket override forwarded to each put() call.

remove_many(artifact_ids: list[str], bucket: str | None = None) → None[source]¶

Delete multiple S3 objects using the native batch delete API.

Sends at most 1000 keys per request as required by the S3 API.

Parameters:

artifact_ids – S3 object keys to delete.
bucket – Bucket override; uses default_bucket when None.