fleche.storage

Storage subpackage public API.

This module re-exports the primary storage interfaces and implementations for backward compatibility with from fleche.storage import … imports.

Submodules

• fleche.storage.bagofholding_file
• fleche.storage.base
• fleche.storage.destructuring
• fleche.storage.file
• fleche.storage.memory
• fleche.storage.pickle_file
• fleche.storage.sql
• fleche.storage.thread_safe
• fleche.storage.void

Exceptions

SaveError	Common base class for all non-exit exceptions.
AmbiguousDigestError	Inappropriate argument value (of correct type).

Classes

Intent	Describes the kind of operation being performed on storage.
OperationContext	Minimal base that exposes the _operation_context() hook.
KeyManagement	Abstract base providing key-management helpers for any keyed storage.
StorageBackend	Primitive backend interface for key-value storage.
ValueStorage	Abstract domain interface for value storage.
ValueMixin	Bridges ValueStorage with StorageBackend primitives.
CallStorage	Abstract domain interface for call storage.
CallMixin	Bridges CallStorage with StorageBackend primitives.
DestructuringMixin	Mixin that recursively destructures collections on save/load.
ValueMemory	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.
CallMemory	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.
ValueVoid	Bridges ValueStorage with StorageBackend primitives.
CallVoid	Bridges CallStorage with StorageBackend primitives.
FileStorage	File-based storage backend using pickle.
ValuePickleFile	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.
CallPickleFile	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.
ValueBagOfHoldingH5File	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.
CallBagOfHoldingH5File	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.
Sql	SQLAlchemy-backed CallStorage with JSON metadata and DB-backed expand().
SerializingMixin	Mixin that serializes all storage operations behind a single reentrant lock.
PerKeyLockMixin	Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

Functions

register_destructurer(→ None)

Register a custom container destructurer.

Package Contents

exception fleche.storage.SaveError

Bases: Exception

Common base class for all non-exit exceptions.

exception fleche.storage.AmbiguousDigestError

Bases: ValueError

Inappropriate argument value (of correct type).

class fleche.storage.Intent

Bases: enum.StrEnum

Describes the kind of operation being performed on storage.

Mixins may use this to choose between exclusive and shared locks.

WRITE = 'write'

class fleche.storage.OperationContext

Bases: abc.ABC

Minimal base that exposes the _operation_context() hook.

Both KeyManagement (storage layer) and BaseCache (cache layer) inherit from this class so that the same thread-safety mixins (SerializingMixin, PerKeyLockMixin) can attach to either layer without duplication.

_operation_context(key: fleche.digest.Digest | str, *, intent: Intent = Intent.WRITE)

Context manager entered around every operation on key.

The base implementation is a no-op. Override in a mixin to inject any resource scoped to the operation — a threading lock, a SQLAlchemy session, an open file handle, a decompression stream, etc.

Receiving key lets implementations choose between a single global resource (ignore the key) or per-key resources (e.g. a striped lock table or a key-specific file handle).

intent describes the kind of operation being performed. Mixins may use it to choose between exclusive and shared locks. Currently the only defined value is Intent.WRITE (the default).

Composing multiple mixins: use super() to chain so that every mixin in the MRO gets to wrap the operation:

@contextlib.contextmanager
def _operation_context(self, key, *, intent=Intent.WRITE):
    with self._lock:                   # this mixin's resource
        with super()._operation_context(key, intent=intent):
            yield

class fleche.storage.KeyManagement

Bases: OperationContext

Abstract base providing key-management helpers for any keyed storage.

Subclasses must implement list, _evict, and _contains. The concrete helpers evict, contains, expand, and shrink are implemented here once and inherited by all storage classes.

Every public operation enters _operation_context() around the compound work it performs, so mixins can inject an operation-scoped resource (e.g. a threading lock, a SQLAlchemy session, a file handle) without overriding every method individually.

abstractmethod list() → Iterable[fleche.digest.Digest]

abstractmethod _evict(key: fleche.digest.Digest) → None

abstractmethod _contains(key: fleche.digest.Digest) → bool

evict(key: fleche.digest.Digest | str) → None

Removes the entry corresponding to the key from the storage.

contains(key: fleche.digest.Digest | str) → bool

expand(key: fleche.digest.Digest | str) → fleche.digest.Digest

Expands a short-hand digest to the full length one.

shrink(key: fleche.digest.Digest | str, /) → fleche.digest.Digest

shrink(key: fleche.digest.Digest | str, /, *keys: fleche.digest.Digest | str) → tuple[Digest, ...]

Find the shortest substring(s) that unambiguously reference each key.

With a single key, returns one Digest. With multiple keys, returns a tuple of Digest in the same order as the inputs; the batched form fetches list() once instead of per-key, which matters on backends where listing is expensive (e.g. SQL, filesystem).

_shrink_one(key: Digest | str, sorted_all: Sequence[str]) → fleche.digest.Digest

_normalize_key(key: fleche.digest.Digest | str) → fleche.digest.Digest

Expand a short digest prefix to a full key, or wrap a full key as Digest.

class fleche.storage.StorageBackend

Bases: KeyManagement

Primitive backend interface for key-value storage.

Backends implement the low-level put/get/_evict/list operations. Higher-level classes (ValueMixin, CallMixin) add domain-specific logic on top.

abstractmethod put(value: Any, key: fleche.digest.Digest) → fleche.digest.Digest

abstractmethod get(key: fleche.digest.Digest) → Any

_contains(key: fleche.digest.Digest) → bool

class fleche.storage.ValueStorage

Bases: KeyManagement

Abstract domain interface for value storage.

abstractmethod save(value: Any, key: fleche.digest.Digest | None = None) → fleche.digest.Digest

abstractmethod load(key: fleche.digest.Digest | str) → Any

class fleche.storage.ValueMixin

Bases: ValueStorage, StorageBackend

Bridges ValueStorage with StorageBackend primitives.

Implements save and load using put and get. Concrete classes inherit from this and a StorageBackend implementation to get a fully functional value storage.

save(value: Any, key: fleche.digest.Digest | None = None) → fleche.digest.Digest

load(key: fleche.digest.Digest | str) → Any

class fleche.storage.CallStorage

Bases: KeyManagement

Abstract domain interface for call storage.

abstractmethod save(call: fleche.call.DigestedCall) → fleche.digest.Digest

abstractmethod load(key: fleche.digest.Digest | str) → fleche.call.DigestedCall

abstractmethod query(template: fleche.call.QueryCall) → Iterable[fleche.call.DigestedCall]

transform(func: Callable[[fleche.call.DigestedCall], fleche.call.DigestedCall] | None = None) → None

Applies a transformation function to all DigestedCall objects in the storage.

Parameters:

func – A function that takes a DigestedCall and returns a transformed one. If None, the identity is used (useful for re-calculating keys).

class fleche.storage.CallMixin

Bases: CallStorage, StorageBackend

Bridges CallStorage with StorageBackend primitives.

Implements save, load, and query using put and get, deriving the storage key from the call’s lookup key. transform is inherited from CallStorage.

Concrete classes inherit from this and a StorageBackend implementation to get a fully functional call storage.

save(call: fleche.call.DigestedCall) → fleche.digest.Digest

load(key: fleche.digest.Digest | str) → fleche.call.DigestedCall

query(template: fleche.call.QueryCall) → Iterable[fleche.call.DigestedCall]

Find cached calls that ‘match’ the template.

Returns all calls where the given arguments, results or metadata match exactly the stored ones. Values may be given either as they are or as Digest.

Parameters:

template (Call) – specification for calls to return; use None as wildcard.

Returns:

an iterable over all matching digested call objects

Return type:

Iterable[DigestedCall]

class fleche.storage.DestructuringMixin

Bases: fleche.storage.base.ValueStorage

Mixin that recursively destructures collections on save/load.

Place before a ValueMixin in the MRO to add destructuring behavior. Lists, tuples, and dicts are broken apart so each element is stored independently; on load the original structure is reassembled.

Example

>>> from fleche.storage.base import ValueMixin
>>> from fleche.storage.memory import MemoryBackend
>>> @dataclass(frozen=True)
... class MyValueStorage(DestructuringMixin, ValueMixin, MemoryBackend): ...
>>> vm = MyValueStorage(storage={})
>>> key = vm.save([1, [2, 3]])
>>> vm.load(key) == [1, [2, 3]]
True

remaining_depth: int = 0

static _is_trojan_tuple(value)

_intern_rec(value: Any, key: fleche.digest.Digest | None = None) → tuple[Any, int | float]

Post-order traversal: recurse to leaves, decide inline-vs-store on the way back up.

Returns (result, depth) where result is the plain value when depth < remaining_depth (the element is inlined in its parent’s Digested wrapper) or a Digest when the element was written to storage separately. Every node in the structure is visited exactly once (O(n)), unlike a separate depth-counting pass.

save(value: Any, key: fleche.digest.Digest | None = None) → fleche.digest.Digest

load(key: fleche.digest.Digest | str) → Any

_raw_sub_digests(raw: Any) → set[fleche.digest.Digest]

Direct digest children of a raw stored entry.

A raw entry is what super().load returns — i.e. what was written to the underlying backend before mend() rewires sub-digests back into their parent container. Only Digested wrappers carry child references; scalars and plain (non-destructured) containers return an empty set.

child_digests(key: fleche.digest.Digest | str) → set[fleche.digest.Digest]

Direct digest children of the raw entry stored at key.

Bypasses mend(), so destructured sub-references are returned as opaque Digest keys rather than being followed. Intended for reference-graph traversals (GC, debugging) where loading the mended value would flatten the structure we need to inspect.

Raises:

KeyError – if key is not present in the underlying backend.

count_reuses() → collections.Counter[fleche.digest.Digest]

Return a counter of how many times each stored key is referenced as a sub-component.

Scans every raw entry and tallies Digest back-references found inside DigestedIterable and DigestedDict wrappers. A count of 0 means the key is not pointed to by any other stored value (i.e. a top-level entry). A count greater than 1 indicates a sub-value shared between multiple parent containers.

Returns:

A Counter mapping each Digest key to the number of times it is referenced by other stored entries.

Example

>>> from fleche.storage.memory import ValueMemory
>>> ds = ValueMemory(storage={})
>>> shared = [2, 3]
>>> _ = ds.save([1, shared])
>>> _ = ds.save([4, shared])
>>> hits = ds.count_reuses()
>>> hits[ds.save(shared)]  # [2, 3] is referenced by both outer lists
2

fleche.storage.register_destructurer(pred: Callable[[Any], bool], fn: Callable) → None

Register a custom container destructurer.

pred(value) should return True for values this destructurer handles. fn must accept (intern, value) where intern is DestructuringMixin._intern_rec(). Entries are appended after the built-in ones; first match wins, so registering a handler for an entirely new container type is safe without displacing list/dict/dataclass/attrs. Call before any DestructuringMixin instance is used.

class fleche.storage.ValueMemory

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.destructuring.DestructuringMixin, fleche.storage.base.ValueMixin, MemoryBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

__hash__

class fleche.storage.CallMemory

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.base.CallMixin, MemoryBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

__hash__

class fleche.storage.ValueVoid

Bases: fleche.storage.base.ValueMixin, VoidBackend

Bridges ValueStorage with StorageBackend primitives.

Implements save and load using put and get. Concrete classes inherit from this and a StorageBackend implementation to get a fully functional value storage.

class fleche.storage.CallVoid

Bases: fleche.storage.base.CallMixin, VoidBackend

Bridges CallStorage with StorageBackend primitives.

Implements save, load, and query using put and get, deriving the storage key from the call’s lookup key. transform is inherited from CallStorage.

Concrete classes inherit from this and a StorageBackend implementation to get a fully functional call storage.

class fleche.storage.FileStorage

Bases: fleche.storage.base.StorageBackend

File-based storage backend using pickle.

Stores objects on the filesystem.

root: pathlib.Path

lock_timeout: float = 1.0

__post_init__() → None

_path(key: str) → pathlib.Path

list() → Iterable[fleche.digest.Digest]

_evict(key: fleche.digest.Digest) → None

put(value: Any, key: fleche.digest.Digest) → fleche.digest.Digest

get(key: fleche.digest.Digest) → Any

abstractmethod _to_file(value: Any, path: pathlib.Path) → None

abstractmethod _from_file(path: pathlib.Path) → Any

_contains(key: fleche.digest.Digest) → bool

class fleche.storage.ValuePickleFile

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.destructuring.DestructuringMixin, fleche.storage.base.ValueMixin, PickleFileBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

class fleche.storage.CallPickleFile

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.base.CallMixin, PickleFileBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

class fleche.storage.ValueBagOfHoldingH5File

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.destructuring.DestructuringMixin, fleche.storage.base.ValueMixin, BagOfHoldingH5FileBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

class fleche.storage.CallBagOfHoldingH5File

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.base.CallMixin, BagOfHoldingH5FileBackend

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

class fleche.storage.Sql

Bases: fleche.storage.thread_safe.PerKeyLockMixin, fleche.storage.base.CallStorage

SQLAlchemy-backed CallStorage with JSON metadata and DB-backed expand().

url: str | None = None

echo: bool = False

engine: Any

session: Any

_local: threading.local

__post_init__() → None

__reduce__()

_session_context()

_operation_context(key, *, intent: fleche.storage.base.Intent = Intent.WRITE)

Context manager entered around every operation on key.

The base implementation is a no-op. Override in a mixin to inject any resource scoped to the operation — a threading lock, a SQLAlchemy session, an open file handle, a decompression stream, etc.

Receiving key lets implementations choose between a single global resource (ignore the key) or per-key resources (e.g. a striped lock table or a key-specific file handle).

intent describes the kind of operation being performed. Mixins may use it to choose between exclusive and shared locks. Currently the only defined value is Intent.WRITE (the default).

Composing multiple mixins: use super() to chain so that every mixin in the MRO gets to wrap the operation:

@contextlib.contextmanager
def _operation_context(self, key, *, intent=Intent.WRITE):
    with self._lock:                   # this mixin's resource
        with super()._operation_context(key, intent=intent):
            yield

_persist_call(call: fleche.call.DigestedCall, key: fleche.digest.Digest) → fleche.digest.Digest

_fetch_call(key: fleche.digest.Digest) → fleche.call.DigestedCall

_contains(key: fleche.digest.Digest) → bool

list() → Iterable[fleche.digest.Digest]

expand(key: fleche.digest.Digest | str) → fleche.digest.Digest

Expands a short-hand digest to the full length one.

_evict(key: fleche.digest.Digest) → None

save(call: fleche.call.DigestedCall) → fleche.digest.Digest

load(key: fleche.digest.Digest | str) → fleche.call.DigestedCall

_normalize_value(v: Any) → str

Return the stored form used in SQL for argument/result matching.

We must match the generic CallStorage.query semantics which compare digest(template_value) == digest(stored_call_value). In this backend, stored argument/result values are hex-digest strings, and digest(Digest(x)) == x. Therefore we should always compare Arg.value/CallModel.result to str(digest(template_value)).

_build_call_conditions(template: fleche.call.QueryCall) → List[Any]

_apply_argument_filters(stmt: Any, arguments: dict[str, Any] | None) → Any

_apply_metadata_filters(stmt: Any, meta_specs: dict[str, dict[str, Any]] | None) → Any

query(template: fleche.call.QueryCall) → Iterable[fleche.call.DigestedCall]

Find cached calls matching a template using SQL-side filtering.

Semantics match CallStorage.query: - Fields set to None are wildcards. - Arguments and result are compared by digest(template_value) == digest(stored_value). - Metadata can be filtered by providing template.metadata as a mapping of

metadata name -> dict of key/value filters. An empty dict for a given name means “presence of that metadata name”. Filters with simple types (str, bool, int, float) are pushed down to SQL via JSON-extract expressions; other types (e.g., lists) or None values fall back to client-side checks after loading.

This method builds a SELECT over calls, joining the arguments table and metadata table as needed to reduce candidate rows, then loads the resulting calls and performs any remaining client-side validation.

Parameters:

template – A Call used as a template. None-valued fields are wildcards.

Yields:

Call – Matching calls including their decoded metadata.

class fleche.storage.SerializingMixin

Bases: fleche.storage.base.OperationContext

Mixin that serializes all storage operations behind a single reentrant lock.

Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class SerializingValueMemory(SerializingMixin, ValueMemory): ...

_lock: _PicklableRLock

_operation_context(key, *, intent: fleche.storage.base.Intent = Intent.WRITE)

Context manager entered around every operation on key.

The base implementation is a no-op. Override in a mixin to inject any resource scoped to the operation — a threading lock, a SQLAlchemy session, an open file handle, a decompression stream, etc.

Receiving key lets implementations choose between a single global resource (ignore the key) or per-key resources (e.g. a striped lock table or a key-specific file handle).

intent describes the kind of operation being performed. Mixins may use it to choose between exclusive and shared locks. Currently the only defined value is Intent.WRITE (the default).

Composing multiple mixins: use super() to chain so that every mixin in the MRO gets to wrap the operation:

@contextlib.contextmanager
def _operation_context(self, key, *, intent=Intent.WRITE):
    with self._lock:                   # this mixin's resource
        with super()._operation_context(key, intent=intent):
            yield

class fleche.storage.PerKeyLockMixin

Bases: fleche.storage.base.OperationContext

Mixin that locks per-key so concurrent ops on different keys proceed in parallel.

A lightweight threading.Lock guards the lock-table itself; once the per-key RLock is obtained the table lock is released, so two threads operating on different keys never block each other. Operations on the same key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. expand inside load).

Instances must be hashable. Place before the concrete storage class in the MRO:

@dataclass(frozen=True)
class PerKeyValuePickle(PerKeyLockMixin, ValuePickleFile): ...

_get_key_lock(key: fleche.digest.Digest | str) → threading.RLock

_operation_context(key, *, intent: fleche.storage.base.Intent = Intent.WRITE)

Context manager entered around every operation on key.

The base implementation is a no-op. Override in a mixin to inject any resource scoped to the operation — a threading lock, a SQLAlchemy session, an open file handle, a decompression stream, etc.

Receiving key lets implementations choose between a single global resource (ignore the key) or per-key resources (e.g. a striped lock table or a key-specific file handle).

intent describes the kind of operation being performed. Mixins may use it to choose between exclusive and shared locks. Currently the only defined value is Intent.WRITE (the default).

Composing multiple mixins: use super() to chain so that every mixin in the MRO gets to wrap the operation:

@contextlib.contextmanager
def _operation_context(self, key, *, intent=Intent.WRITE):
    with self._lock:                   # this mixin's resource
        with super()._operation_context(key, intent=intent):
            yield