fleche.storage ============== .. py:module:: fleche.storage .. autoapi-nested-parse:: Storage subpackage public API. This module re-exports the primary storage interfaces and implementations for backward compatibility with `from fleche.storage import ...` imports. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/fleche/storage/bagofholding_file/index /autoapi/fleche/storage/base/index /autoapi/fleche/storage/destructuring/index /autoapi/fleche/storage/file/index /autoapi/fleche/storage/memory/index /autoapi/fleche/storage/pickle_file/index /autoapi/fleche/storage/sql/index /autoapi/fleche/storage/thread_safe/index /autoapi/fleche/storage/void/index Exceptions ---------- .. autoapisummary:: fleche.storage.SaveError fleche.storage.AmbiguousDigestError Classes ------- .. autoapisummary:: fleche.storage.KeyManagement fleche.storage.StorageBackend fleche.storage.ValueStorage fleche.storage.ValueMixin fleche.storage.CallStorage fleche.storage.CallMixin fleche.storage.DestructuringMixin fleche.storage.ValueMemory fleche.storage.CallMemory fleche.storage.ValueVoid fleche.storage.CallVoid fleche.storage.FileStorage fleche.storage.ValuePickleFile fleche.storage.CallPickleFile fleche.storage.ValueBagOfHoldingH5File fleche.storage.CallBagOfHoldingH5File fleche.storage.Sql fleche.storage.SerializingMixin fleche.storage.PerKeyLockMixin Package Contents ---------------- .. py:exception:: SaveError Bases: :py:obj:`Exception` Common base class for all non-exit exceptions. .. py:exception:: AmbiguousDigestError Bases: :py:obj:`ValueError` Inappropriate argument value (of correct type). .. py:class:: KeyManagement Bases: :py:obj:`abc.ABC` Abstract base providing key-management helpers for any keyed storage. Subclasses must implement ``list``, ``_evict``, and ``_contains``. The concrete helpers ``evict``, ``contains``, ``expand``, and ``shrink`` are implemented here once and inherited by all storage classes. Every public operation enters :meth:`_operation_context` around the compound work it performs, so mixins can inject an operation-scoped resource (e.g. a threading lock, a SQLAlchemy session, a file handle) without overriding every method individually. .. py:method:: _operation_context(key: fleche.digest.Digest | str) Context manager entered around every operation on ``key``. The base implementation is a no-op. Override in a mixin to inject any resource scoped to the operation — a threading lock, a SQLAlchemy session, an open file handle, a decompression stream, etc. Receiving ``key`` lets implementations choose between a single global resource (ignore the key) or per-key resources (e.g. a striped lock table or a key-specific file handle). **Composing multiple mixins**: use ``super()`` to chain so that every mixin in the MRO gets to wrap the operation:: @contextlib.contextmanager def _operation_context(self, key): with self._lock: # this mixin's resource with super()._operation_context(key): yield .. py:method:: list() -> Iterable[fleche.digest.Digest] :abstractmethod: .. py:method:: _evict(key: fleche.digest.Digest) -> None :abstractmethod: .. py:method:: _contains(key: fleche.digest.Digest) -> bool :abstractmethod: .. py:method:: evict(key: fleche.digest.Digest | str) -> None Removes the entry corresponding to the key from the storage. .. py:method:: contains(key: fleche.digest.Digest | str) -> bool .. py:method:: expand(key: fleche.digest.Digest | str) -> fleche.digest.Digest Expands a short-hand digest to the full length one. .. py:method:: shrink(key: fleche.digest.Digest | str) -> fleche.digest.Digest Find the shortest substring that is still an unambiguous reference to the same value. .. py:class:: StorageBackend Bases: :py:obj:`KeyManagement` Primitive backend interface for key-value storage. Backends implement the low-level ``put``/``get``/``_evict``/``list`` operations. Higher-level classes (:class:`ValueMixin`, :class:`CallMixin`) add domain-specific logic on top. .. py:method:: put(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest :abstractmethod: .. py:method:: get(key: fleche.digest.Digest) -> Any :abstractmethod: .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:class:: ValueStorage Bases: :py:obj:`KeyManagement` Abstract domain interface for value storage. .. py:method:: save(value: Any, key: fleche.digest.Digest | None = None) -> fleche.digest.Digest :abstractmethod: .. py:method:: load(key: fleche.digest.Digest | str) -> Any :abstractmethod: .. py:class:: ValueMixin Bases: :py:obj:`ValueStorage`, :py:obj:`StorageBackend` Bridges :class:`ValueStorage` with :class:`StorageBackend` primitives. Implements ``save`` and ``load`` using ``put`` and ``get``. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional value storage. .. py:method:: save(value: Any, key: fleche.digest.Digest | None = None) -> fleche.digest.Digest .. py:method:: load(key: fleche.digest.Digest | str) -> Any .. py:class:: CallStorage Bases: :py:obj:`KeyManagement` Abstract domain interface for call storage. .. py:method:: save(call: fleche.call.Call) -> fleche.digest.Digest :abstractmethod: .. py:method:: load(key: fleche.digest.Digest | str) -> fleche.call.Call :abstractmethod: .. py:method:: query(template: fleche.call.QueryCall) -> Iterable[fleche.call.Call] :abstractmethod: .. py:method:: transform(func: Callable[[fleche.call.Call], fleche.call.Call] | None = None) -> None Applies a transformation function to all Call objects in the storage. :param func: A function that takes a Call and returns a transformed Call. If None, the identity function is used (useful for re-calculating keys). :type func: Callable[[Call], Call] | None .. py:class:: CallMixin Bases: :py:obj:`CallStorage`, :py:obj:`StorageBackend` Bridges :class:`CallStorage` with :class:`StorageBackend` primitives. Implements ``save``, ``load``, and ``query`` using ``put`` and ``get``, deriving the storage key from the call's lookup key. ``transform`` is inherited from :class:`CallStorage`. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional call storage. .. py:method:: save(call: fleche.call.Call) -> fleche.digest.Digest .. py:method:: load(key: fleche.digest.Digest | str) -> fleche.call.Call .. py:method:: query(template: fleche.call.QueryCall) -> Iterable[fleche.call.Call] Find cached calls that 'match' the template. Returns all calls where the given arguments, results or metadata match exactly the stored ones. Values may be given either as they are or as :class:`Digest`. :param template: specification for calls to return; use `None` as wildcard. :type template: Call :returns: an iterable over all matching call objects :rtype: Iterable[Call] .. py:class:: DestructuringMixin Bases: :py:obj:`fleche.storage.base.StorageBackend` Mixin that recursively destructures collections on save/load. Place before a concrete :class:`StorageBackend` in the MRO to add destructuring behavior. Lists, tuples, and dicts are broken apart so each element is stored independently; on load the original structure is reassembled. Example:: @dataclass(frozen=True) class ValueMemory(ValueMixin, DestructuringMixin, MemoryBackend): ... vm = ValueMemory(storage={}) key = vm.save([1, [2, 3]]) assert vm.load(key) == [1, [2, 3]] .. py:attribute:: remaining_depth :type: int :value: 0 .. py:method:: _is_trojan_tuple(value) :staticmethod: .. py:method:: _intern_rec(value: Any, key: fleche.digest.Digest | None = None) -> tuple[Any, int | float] Post-order traversal: recurse to leaves, decide inline-vs-store on the way back up. Returns ``(result, depth)`` where *result* is the plain value when ``depth < remaining_depth`` (the element is inlined in its parent's :class:`Digested` wrapper) or a :class:`Digest` when the element was written to storage separately. Every node in the structure is visited exactly once (O(n)), unlike a separate depth-counting pass. .. py:method:: put(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: get(key: fleche.digest.Digest | Any) -> Any .. py:class:: ValueMemory Bases: :py:obj:`fleche.storage.base.ValueMixin`, :py:obj:`fleche.storage.destructuring.DestructuringMixin`, :py:obj:`MemoryBackend` Bridges :class:`ValueStorage` with :class:`StorageBackend` primitives. Implements ``save`` and ``load`` using ``put`` and ``get``. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional value storage. .. py:class:: CallMemory Bases: :py:obj:`fleche.storage.base.CallMixin`, :py:obj:`MemoryBackend` Bridges :class:`CallStorage` with :class:`StorageBackend` primitives. Implements ``save``, ``load``, and ``query`` using ``put`` and ``get``, deriving the storage key from the call's lookup key. ``transform`` is inherited from :class:`CallStorage`. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional call storage. .. py:class:: ValueVoid Bases: :py:obj:`fleche.storage.base.ValueMixin`, :py:obj:`VoidBackend` Bridges :class:`ValueStorage` with :class:`StorageBackend` primitives. Implements ``save`` and ``load`` using ``put`` and ``get``. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional value storage. .. py:class:: CallVoid Bases: :py:obj:`fleche.storage.base.CallMixin`, :py:obj:`VoidBackend` Bridges :class:`CallStorage` with :class:`StorageBackend` primitives. Implements ``save``, ``load``, and ``query`` using ``put`` and ``get``, deriving the storage key from the call's lookup key. ``transform`` is inherited from :class:`CallStorage`. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional call storage. .. py:class:: FileStorage Bases: :py:obj:`fleche.storage.base.StorageBackend` File-based storage backend using pickle. Stores objects on the filesystem. .. py:attribute:: root :type: pathlib.Path .. py:attribute:: lock_timeout :type: float :value: 1.0 .. py:attribute:: lock_wait_start :type: float :value: 0.001 .. py:method:: __post_init__() -> None .. py:method:: _path(key: str) -> pathlib.Path .. py:method:: list() -> Iterable[fleche.digest.Digest] .. py:method:: _evict(key: fleche.digest.Digest) -> None .. py:method:: put(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: get(key: fleche.digest.Digest) -> Any .. py:method:: _to_file(value: Any, path: pathlib.Path) -> None :abstractmethod: .. py:method:: _from_file(path: pathlib.Path) -> Any :abstractmethod: .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:class:: ValuePickleFile Bases: :py:obj:`fleche.storage.base.ValueMixin`, :py:obj:`fleche.storage.destructuring.DestructuringMixin`, :py:obj:`PickleFileBackend` Bridges :class:`ValueStorage` with :class:`StorageBackend` primitives. Implements ``save`` and ``load`` using ``put`` and ``get``. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional value storage. .. py:class:: CallPickleFile Bases: :py:obj:`fleche.storage.base.CallMixin`, :py:obj:`PickleFileBackend` Bridges :class:`CallStorage` with :class:`StorageBackend` primitives. Implements ``save``, ``load``, and ``query`` using ``put`` and ``get``, deriving the storage key from the call's lookup key. ``transform`` is inherited from :class:`CallStorage`. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional call storage. .. py:class:: ValueBagOfHoldingH5File Bases: :py:obj:`fleche.storage.base.ValueMixin`, :py:obj:`fleche.storage.destructuring.DestructuringMixin`, :py:obj:`BagOfHoldingH5FileBackend` Bridges :class:`ValueStorage` with :class:`StorageBackend` primitives. Implements ``save`` and ``load`` using ``put`` and ``get``. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional value storage. .. py:class:: CallBagOfHoldingH5File Bases: :py:obj:`fleche.storage.base.CallMixin`, :py:obj:`BagOfHoldingH5FileBackend` Bridges :class:`CallStorage` with :class:`StorageBackend` primitives. Implements ``save``, ``load``, and ``query`` using ``put`` and ``get``, deriving the storage key from the call's lookup key. ``transform`` is inherited from :class:`CallStorage`. Concrete classes inherit from this and a :class:`StorageBackend` implementation to get a fully functional call storage. .. py:class:: Sql Bases: :py:obj:`fleche.storage.base.CallStorage` SQLAlchemy-backed CallStorage with JSON metadata and DB-backed expand(). .. py:attribute:: url :type: str | None :value: None .. py:attribute:: echo :type: bool :value: False .. py:attribute:: engine :type: Any .. py:attribute:: session :type: Any .. py:method:: __post_init__() -> None .. py:method:: __getstate__() .. py:method:: __setstate__(state) .. py:method:: put(call: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: get(key: fleche.digest.Digest) -> fleche.call.Call .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:method:: list() -> Iterable[fleche.digest.Digest] .. py:method:: expand(key: fleche.digest.Digest | str) -> fleche.digest.Digest Expands a short-hand digest to the full length one. .. py:method:: _evict(key: fleche.digest.Digest) -> None .. py:method:: save(call: fleche.call.Call) -> fleche.digest.Digest .. py:method:: load(key: fleche.digest.Digest | str) -> fleche.call.Call .. py:method:: _normalize_value(v: Any) -> str Return the stored form used in SQL for argument/result matching. We must match the generic CallStorage.query semantics which compare digest(template_value) == digest(stored_call_value). In this backend, stored argument/result values are hex-digest strings, and digest(Digest(x)) == x. Therefore we should always compare Arg.value/CallModel.result to str(digest(template_value)). .. py:method:: _build_call_conditions(template: fleche.call.QueryCall) -> List[Any] .. py:method:: _apply_argument_filters(stmt: Any, arguments: dict[str, Any] | None) -> Any .. py:method:: _apply_metadata_filters(stmt: Any, meta_specs: dict[str, dict[str, Any]] | None) -> Any .. py:method:: query(template: fleche.call.QueryCall) -> Iterable[fleche.call.Call] Find cached calls matching a template using SQL-side filtering. Semantics match CallStorage.query: - Fields set to None are wildcards. - Arguments and result are compared by digest(template_value) == digest(stored_value). - Metadata can be filtered by providing template.metadata as a mapping of metadata name -> dict of key/value filters. An empty dict for a given name means "presence of that metadata name". Filters with simple types (str, bool, int, float) are pushed down to SQL via JSON-extract expressions; other types (e.g., lists) or None values fall back to client-side checks after loading. This method builds a SELECT over calls, joining the arguments table and metadata table as needed to reduce candidate rows, then loads the resulting calls and performs any remaining client-side validation. :param template: A Call used as a template. None-valued fields are wildcards. :Yields: *Call* -- Matching calls including their decoded metadata. .. py:class:: SerializingMixin Bases: :py:obj:`fleche.storage.base.KeyManagement` Mixin that serializes all storage operations behind a single reentrant lock. Place before the concrete storage class in the MRO:: @dataclass(frozen=True) class SerializingValueMemory(SerializingMixin, ValueMemory): ... .. py:attribute:: _lock :type: threading.RLock .. py:method:: _operation_context(key) Context manager entered around every operation on ``key``. The base implementation is a no-op. Override in a mixin to inject any resource scoped to the operation — a threading lock, a SQLAlchemy session, an open file handle, a decompression stream, etc. Receiving ``key`` lets implementations choose between a single global resource (ignore the key) or per-key resources (e.g. a striped lock table or a key-specific file handle). **Composing multiple mixins**: use ``super()`` to chain so that every mixin in the MRO gets to wrap the operation:: @contextlib.contextmanager def _operation_context(self, key): with self._lock: # this mixin's resource with super()._operation_context(key): yield .. py:class:: PerKeyLockMixin Bases: :py:obj:`fleche.storage.base.KeyManagement` Mixin that locks per-key so concurrent ops on different keys proceed in parallel. A lightweight ``threading.Lock`` guards the lock-table itself; once the per-key ``RLock`` is obtained the table lock is released, so two threads operating on *different* keys never block each other. Operations on the *same* key are serialized by the per-key lock, which is reentrant to allow nested calls (e.g. ``expand`` inside ``load``). Place before the concrete storage class in the MRO:: @dataclass(frozen=True) class PerKeyValueMemory(PerKeyLockMixin, ValueMemory): ... .. py:attribute:: _key_locks :type: weakref.WeakValueDictionary[fleche.digest.Digest | str, threading.RLock] .. py:attribute:: _meta_lock :type: threading.Lock .. py:method:: _get_key_lock(key: fleche.digest.Digest | str) -> threading.RLock .. py:method:: _operation_context(key) Context manager entered around every operation on ``key``. The base implementation is a no-op. Override in a mixin to inject any resource scoped to the operation — a threading lock, a SQLAlchemy session, an open file handle, a decompression stream, etc. Receiving ``key`` lets implementations choose between a single global resource (ignore the key) or per-key resources (e.g. a striped lock table or a key-specific file handle). **Composing multiple mixins**: use ``super()`` to chain so that every mixin in the MRO gets to wrap the operation:: @contextlib.contextmanager def _operation_context(self, key): with self._lock: # this mixin's resource with super()._operation_context(key): yield