fleche.storage.bagofholding_file

Attributes

logger
VersionValidator
_DEFAULT_PREFIX_LENGTH

Classes

BagOfHoldingH5FileBackend	File-based storage backend using pickle.
ValueBagOfHoldingH5File	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.
CallBagOfHoldingH5File	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

Functions

_validate_prefix_length(→ None)
_observed_prefix_lengths(→ set[int])	Prefix lengths of all fleche-written files in root (0 = per-key).

Module Contents

fleche.storage.bagofholding_file.logger

fleche.storage.bagofholding_file.VersionValidator

fleche.storage.bagofholding_file._DEFAULT_PREFIX_LENGTH = 2

fleche.storage.bagofholding_file._validate_prefix_length(prefix_length: Any) → None

fleche.storage.bagofholding_file._observed_prefix_lengths(root: pathlib.Path) → set[int]

Prefix lengths of all fleche-written files in root (0 = per-key).

Multi-bag files are named {prefix}.h5 and per-key files by the full digest, so the prefix length a file was written with can be read off its name. Files this backend never writes (locks, dotfiles, anything else) are ignored.

class fleche.storage.bagofholding_file.BagOfHoldingH5FileBackend

Bases: fleche.storage.file.FileStorage

File-based storage backend using pickle.

Stores objects on the filesystem.

version_validator: VersionValidator | None = None

prefix_length: int | None = 2

check_consistency: dataclasses.InitVar[bool] = True

__post_init__(check_consistency: bool = True)

_infer_prefix_length() → int

_check_prefix_consistency() → None

Raise ValueError if files already in root were written with a different prefix length.

refix(prefix_length: int) → BagOfHoldingH5FileBackend

Copy every stored entry into a new prefix-length layout.

Originals are evicted as soon as their entries are copied, so the transient extra disk usage stays bounded by a single entry (per-key mode) or a single bag file (multi-bag mode — the old bag is unlinked by _evict() the moment its last entry goes, and HDF5 files do not shrink before that anyway). self is left untouched: it keeps addressing the old — afterwards empty — layout, and the returned storage addresses the new one.

The migration is not atomic, but resumable: entries already present in the target layout are skipped, so re-running never re-copies work already done, and consolidate() repairs a root left with both layouts by an interrupted or aborted run (no data is lost either way).

Parameters:

prefix_length – target prefix length, between 0 (one file per key) and DIGEST_LENGTH. Must be explicit — None is not accepted.

Returns:

a storage of the same type at the same

root addressing the new layout; self when prefix_length already matches.

Return type:

BagOfHoldingH5FileBackend

Raises:

• ValueError – if prefix_length is not an integer in range.

• RuntimeError – if an entry cannot be read — silently skipping it would make the next instantiation fail its consistency check instead. Migration aborts immediately, leaving both layouts present.

_refix_one(target: BagOfHoldingH5FileBackend, key: fleche.digest.Digest) → None

Copy one entry into target’s layout, skipping entries a previous (aborted) migration already moved.

classmethod consolidate(root: pathlib.Path | str, prefix_length: int = _DEFAULT_PREFIX_LENGTH, **kwargs) → BagOfHoldingH5FileBackend

Open root regardless of which prefix lengths it contains, migrate everything to prefix_length, and return the resulting storage.

This is the repair constructor for roots holding several layouts at once — e.g. after an interrupted refix(), or after entries were written with different prefix_length settings. Every other prefix length found in root is converted via refix().

Parameters:

• root – storage directory to open.

• prefix_length – target prefix length, between 0 (one file per key) and DIGEST_LENGTH.

• **kwargs – forwarded to the constructor (e.g. lock_timeout, version_validator).

Returns:

a consistency-checked storage at

root with every entry stored under prefix_length.

Return type:

BagOfHoldingH5FileBackend

_to_file(value: Any, path: pathlib.Path) → None

_from_file(path: pathlib.Path) → Any

_bag_file(key: str) → pathlib.Path

The HDF5 file backing key in multi-bag mode: root/{prefix}.h5.

_path(key: str) → pathlib.Path

Path to hand to H5Bag: the plain per-key file, or the composite file.h5/{key} group path in multi-bag mode.

_lock_path(key: str) → pathlib.Path

_contains(key: fleche.digest.Digest) → bool

_evict(key: fleche.digest.Digest) → None

list() → Iterable[fleche.digest.Digest]

rebag(version_validator: VersionValidator = 'none') → None

Re-open and re-save all bags using the given version validator.

Useful when bags were created with an older library version and would otherwise fail strict version checking on load.

class fleche.storage.bagofholding_file.ValueBagOfHoldingH5File

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.destructuring.DestructuringMixin, fleche.storage.base.ValueMixin, BagOfHoldingH5FileBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

class fleche.storage.bagofholding_file.CallBagOfHoldingH5File

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.base.CallMixin, BagOfHoldingH5FileBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...