fleche.storage ============== .. py:module:: fleche.storage .. autoapi-nested-parse:: Storage subpackage public API. This module re-exports the primary storage interfaces and implementations for backward compatibility with `from fleche.storage import ...` imports. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/fleche/storage/bagofholding_file/index /autoapi/fleche/storage/base/index /autoapi/fleche/storage/file/index /autoapi/fleche/storage/memory/index /autoapi/fleche/storage/pickle_file/index /autoapi/fleche/storage/sql/index /autoapi/fleche/storage/void/index Exceptions ---------- .. autoapisummary:: fleche.storage.SaveError fleche.storage.AmbiguousDigestError Classes ------- .. autoapisummary:: fleche.storage.Storage fleche.storage.CallStorage fleche.storage.CallStorageAdapter fleche.storage.DestructuringMixin fleche.storage.DestructuringStorage fleche.storage.Memory fleche.storage.Void fleche.storage.FileStorage fleche.storage.PickleFile fleche.storage.BagOfHoldingH5File fleche.storage.Sql Package Contents ---------------- .. py:exception:: SaveError Bases: :py:obj:`Exception` Common base class for all non-exit exceptions. .. py:exception:: AmbiguousDigestError Bases: :py:obj:`ValueError` Inappropriate argument value (of correct type). .. py:class:: Storage Bases: :py:obj:`StorageBase` Abstract base class for defining storage mechanisms. .. py:method:: save(value: Any, key: fleche.digest.Digest | None = None) -> fleche.digest.Digest .. py:method:: _save(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest :abstractmethod: .. py:method:: load(key: fleche.digest.Digest | str) -> Any .. py:method:: _load(key: fleche.digest.Digest) -> Any :abstractmethod: .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:class:: CallStorage Bases: :py:obj:`StorageBase` Special storage for saving :class:`Call` instances. .. py:method:: save(call: fleche.call.Call) -> fleche.digest.Digest .. py:method:: _save(call: fleche.call.Call) -> fleche.digest.Digest :abstractmethod: .. py:method:: load(key: str | fleche.digest.Digest) -> fleche.call.Call .. py:method:: _load(key: fleche.digest.Digest) -> fleche.call.Call :abstractmethod: .. py:method:: transform(func: Callable[[fleche.call.Call], fleche.call.Call] | None = None) -> None Applies a transformation function to all Call objects in the storage. :param func: A function that takes a Call and returns a transformed Call. If None, the identity function is used (useful for re-calculating keys). :type func: Callable[[Call], Call] | None .. py:method:: query(template: fleche.call.QueryCall) -> Iterable[fleche.call.Call] Find cached calls that 'match' the template. Returns all calls where the given arguments, results or metadata match exactly the stored ones. Values may be given either as they are or as :class:`Digest`. :param template: specification for calls to return; use `None` as wildcard. :type template: Call :returns: an iterable over all matching call objects :rtype: Iterable[Call] .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:class:: CallStorageAdapter Bases: :py:obj:`CallStorage` Implement a CallStorage from a generic Storage. .. py:attribute:: storage :type: Storage .. py:method:: _save(call: fleche.call.Call) -> fleche.digest.Digest .. py:method:: _load(key: fleche.digest.Digest) -> fleche.call.Call .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:method:: _evict(key: fleche.digest.Digest) -> None .. py:method:: list() -> Iterable[fleche.digest.Digest] .. py:class:: DestructuringMixin Bases: :py:obj:`Storage` Mixin that recursively destructures collections on save/load. Place before a concrete :class:`Storage` in the MRO to add destructuring behavior. Lists, tuples, and dicts are broken apart so each element is stored independently; on load the original structure is reassembled. Example:: class DestructuringMemory(DestructuringMixin, Memory): pass # ``storage`` here is the backing dict required by ``Memory``, not a # Storage instance. dm = DestructuringMemory(storage={}) key = dm.save([1, [2, 3]]) assert dm.load(key) == [1, [2, 3]] .. py:attribute:: remaining_depth :type: int :value: 0 .. py:method:: _intern_rec(value: Any) -> tuple[Any, int | float] Post-order traversal: recurse to leaves, decide inline-vs-store on the way back up. Returns ``(result, depth)`` where *result* is the plain value when ``depth < remaining_depth`` (the element is inlined in its parent's :class:`Digested` wrapper) or a :class:`Digest` when the element was written to storage separately. Every node in the structure is visited exactly once (O(n)), unlike a separate depth-counting pass. .. py:method:: _save(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: _load(key: fleche.digest.Digest | Any) -> Any .. py:class:: DestructuringStorage Bases: :py:obj:`DestructuringMixin`, :py:obj:`_DelegatingStorage` Storage wrapper that recursively destructures collections. This is a convenience class combining :class:`DestructuringMixin` with delegation to a wrapped storage. Prefer using :class:`DestructuringMixin` directly as a mixin with a concrete storage class when possible. .. py:attribute:: remaining_depth :type: int :value: 0 .. py:method:: __post_init__() .. py:class:: Memory Bases: :py:obj:`fleche.storage.base.Storage` A concrete implementation of Storage that stores values in an in-memory dictionary. .. py:attribute:: storage :type: dict[fleche.digest.Digest, Any] .. py:method:: _save(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: _load(key: fleche.digest.Digest) -> Any .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:method:: list() -> Iterable[fleche.digest.Digest] .. py:method:: _evict(key: fleche.digest.Digest) -> None .. py:class:: Void Bases: :py:obj:`fleche.storage.base.Storage` A concrete implementation of Storage that does not store anything. .. py:method:: _save(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: _load(key: fleche.digest.Digest) -> Any .. py:method:: list() -> Iterable[fleche.digest.Digest] .. py:method:: _evict(key: fleche.digest.Digest) -> None .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:class:: FileStorage Bases: :py:obj:`fleche.storage.base.Storage` File-based storage backend using pickle. Stores objects on the filesystem. .. py:attribute:: root :type: pathlib.Path .. py:attribute:: lock_timeout :type: float :value: 1.0 .. py:attribute:: lock_wait_start :type: float :value: 0.001 .. py:method:: __post_init__() -> None .. py:method:: _path(key: str) -> pathlib.Path .. py:method:: list() -> Iterable[fleche.digest.Digest] .. py:method:: _evict(key: fleche.digest.Digest) -> None .. py:method:: save(value: Any, key: fleche.digest.Digest | None = None) -> fleche.digest.Digest .. py:method:: load(key: fleche.digest.Digest | str) -> Any .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:class:: PickleFile Bases: :py:obj:`fleche.storage.file.FileStorage` Store values as files on the filesystem using a serialization module. .. py:attribute:: secret_key :type: list[bytes] :value: [] .. py:attribute:: serializer :type: Any .. py:attribute:: compress :type: bool :value: False .. py:method:: __post_init__() .. py:method:: with_pickle(*args, **kwargs) :classmethod: Construct a PickleFile using the standard pickle module. .. py:method:: with_cloudpickle(*args, **kwargs) :classmethod: Construct a PickleFile using the cloudpickle module. .. py:method:: with_dill(*args, **kwargs) :classmethod: Construct a PickleFile using the dill module. .. py:method:: __getstate__() .. py:method:: __setstate__(state) .. py:method:: _save(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: _load(key: fleche.digest.Digest) -> Any .. py:class:: BagOfHoldingH5File Bases: :py:obj:`fleche.storage.file.FileStorage` File-based storage backend using pickle. Stores objects on the filesystem. .. py:method:: __post_init__() .. py:method:: _save(value: Any, key: fleche.digest.Digest) -> fleche.digest.Digest .. py:method:: _load(key: fleche.digest.Digest) -> Any .. py:class:: Sql Bases: :py:obj:`fleche.storage.base.CallStorage` SQLAlchemy-backed CallStorage with JSON metadata and DB-backed expand(). .. py:attribute:: url :type: str | None :value: None .. py:attribute:: echo :type: bool :value: False .. py:attribute:: engine :type: Any .. py:attribute:: session :type: Any .. py:method:: __post_init__() -> None .. py:method:: __getstate__() .. py:method:: __setstate__(state) .. py:method:: _save(call: fleche.call.Call) -> fleche.digest.Digest .. py:method:: _load(key: fleche.digest.Digest) -> fleche.call.Call .. py:method:: _contains(key: fleche.digest.Digest) -> bool .. py:method:: list() -> Iterable[fleche.digest.Digest] .. py:method:: expand(key: fleche.digest.Digest | str) -> fleche.digest.Digest Expands a short-hand digest to the full length one. .. py:method:: _evict(key: fleche.digest.Digest) -> None .. py:method:: _normalize_value(v: Any) -> str Return the stored form used in SQL for argument/result matching. We must match the generic CallStorage.query semantics which compare digest(template_value) == digest(stored_call_value). In this backend, stored argument/result values are hex-digest strings, and digest(Digest(x)) == x. Therefore we should always compare Arg.value/CallModel.result to str(digest(template_value)). .. py:method:: _build_call_conditions(template: fleche.call.QueryCall) -> List[Any] .. py:method:: _apply_argument_filters(stmt: Any, arguments: dict[str, Any] | None) -> Any .. py:method:: _apply_metadata_filters(stmt: Any, meta_specs: dict[str, dict[str, Any]] | None) -> Any .. py:method:: query(template: fleche.call.QueryCall) -> Iterable[fleche.call.Call] Find cached calls matching a template using SQL-side filtering. Semantics match CallStorage.query: - Fields set to None are wildcards. - Arguments and result are compared by digest(template_value) == digest(stored_value). - Metadata can be filtered by providing template.metadata as a mapping of metadata name -> dict of key/value filters. An empty dict for a given name means "presence of that metadata name". Filters with simple types (str, bool, int, float) are pushed down to SQL via JSON-extract expressions; other types (e.g., lists) or None values fall back to client-side checks after loading. This method builds a SELECT over calls, joining the arguments table and metadata table as needed to reduce candidate rows, then loads the resulting calls and performs any remaining client-side validation. :param template: A Call used as a template. None-valued fields are wildcards. :Yields: *Call* -- Matching calls including their decoded metadata.