fleche.config ============= .. py:module:: fleche.config .. autoapi-nested-parse:: Configuration system for fleche. Storage type names ------------------ The ``type`` key in a storage config dict is case-sensitive and uses the following **lowercase** identifiers: ``"memory"`` In-memory dict (:class:`~fleche.storage.ValueMemory` / :class:`~fleche.storage.CallMemory`). No required keys. Optional (value backend): ``remaining_depth`` (int, default ``0``). ``"void"`` No-op — discards all data (:class:`~fleche.storage.ValueVoid` / :class:`~fleche.storage.CallVoid`). No required keys. ``"pickle"`` Filesystem backend serialised with the standard ``pickle`` module (:class:`~fleche.storage.ValuePickleFile` / :class:`~fleche.storage.CallPickleFile`). Required: ``root`` (path to storage directory). Optional: ``compress`` (bool, default ``False``) — gzip-compress files. Optional: ``lock_timeout`` (float, default ``1.0``) — file-lock acquisition timeout (s). Optional: ``secret_key`` (list of hex strings) — HMAC-SHA256 signing keys; each element is a hex-encoded byte string (same format as ``FLECHE_SECRET_KEY``). If omitted, falls back to the ``FLECHE_SECRET_KEY`` environment variable. Optional (value backend): ``remaining_depth`` (int, default ``0``). ``"cloudpickle"`` Filesystem backend serialised with ``cloudpickle`` — handles more complex Python objects than ``pickle``. Required: ``root``. Optional: ``compress`` (bool, default ``False``) — gzip-compress files. Optional: ``lock_timeout`` (float, default ``1.0``) — file-lock acquisition timeout (s). Optional: ``secret_key`` (list of hex strings) — same as ``"pickle"``. Optional (value backend): ``remaining_depth`` (int, default ``0``). ``"dill"`` Filesystem backend serialised with ``dill``. Required: ``root``. Optional: ``compress`` (bool, default ``False``) — gzip-compress files. Optional: ``lock_timeout`` (float, default ``1.0``) — file-lock acquisition timeout (s). Optional: ``secret_key`` (list of hex strings) — same as ``"pickle"``. Optional (value backend): ``remaining_depth`` (int, default ``0``). ``"bagofholding_hdf"`` HDF5-backed storage via the ``bagofholding`` library (:class:`~fleche.storage.ValueBagOfHoldingH5File` / :class:`~fleche.storage.CallBagOfHoldingH5File`). Required: ``root``. Optional: ``lock_timeout`` (float, default ``1.0``) — file-lock acquisition timeout (s). Optional: ``version_validator`` (str, default omitted) — version validation strategy passed to :meth:`bagofholding:bagofholding.h5.bag.H5Bag.load`. One of ``"exact"``, ``"semantic-minor"``, ``"semantic-major"``, ``"none"``. When omitted, bagofholding's default applies. Optional (value backend): ``remaining_depth`` (int, default ``0``). ``"sql"`` SQL database via SQLAlchemy (:class:`~fleche.storage.Sql`). *Call storage only.* Required: ``url`` (SQLAlchemy connection URL, e.g. ``"sqlite:///~/.fleche/calls.db"``). Optional: ``echo`` (bool, default ``False``) — log SQL statements. Example fleche.toml ------------------- :: [default] cache = "persistent" metadata = ["Runtime"] [persistent] values.type = "cloudpickle" values.root = "~/.fleche/values" calls.type = "cloudpickle" calls.root = "~/.fleche/calls" [fast] values.type = "memory" calls.type = "memory" [with_sql_calls] values.type = "cloudpickle" values.root = "~/.fleche/values" calls.type = "sql" calls.url = "sqlite:///~/.fleche/calls.db" # SizeLimitedCache — evicts oldest entries once 100 entries are stored [limited] values.type = "memory" calls.type = "memory" max_size = 100 # ReadOnlyCache — loads from storage but never writes new results [readonly] values.type = "cloudpickle" values.root = "~/.fleche/values" calls.type = "cloudpickle" calls.root = "~/.fleche/calls" read_only = true # CacheStack — TOML array-of-tables; saves to the bottom layer, # loads top-down and back-fills hits to the bottom [[mystack]] values.type = "memory" calls.type = "memory" [[mystack]] values.type = "cloudpickle" values.root = "~/.fleche/values" calls.type = "cloudpickle" calls.root = "~/.fleche/calls" # SshCache — share results with another machine over SSH. The remote # runs `python -m fleche remote --serve` and proxies into its own # configured cache. Compose with a local cache by stacking two # entries (saves go to the first entry; reads fall back to the SSH # remote and back-fill hits into the local layer). [[shared]] values.type = "cloudpickle" values.root = "~/.fleche/values" calls.type = "cloudpickle" calls.root = "~/.fleche/calls" [[shared]] type = "ssh" host = "user@bigpc.example.com" cache_name = "shared" # optional: named cache on remote python = "python3" # optional: remote python interpreter ssh_options = ["-o", "ControlMaster=auto", "-o", "ControlPath=~/.ssh/cm-%r@%h:%p", "-o", "ControlPersist=10m"] setup_commands = ["module load python/3.11", # optional: shell snippets "source ~/.venv/bin/activate"] # run before the server workdir = "~/project" # optional: cd here before launching # the server, so the remote can import # the project's local modules Config file discovery --------------------- When the active cache or default metadata is loaded, fleche walks from the current working directory upward, picking up every ``fleche.toml`` it encounters. The walk stops at ``$HOME`` (inclusive) or at the filesystem root, whichever comes first. ``$XDG_CONFIG_HOME/fleche/cache.toml`` (defaulting to ``~/.config/fleche/cache.toml`` per the XDG base directory spec when ``XDG_CONFIG_HOME`` is unset or empty) is appended as a final lowest-priority layer. All discovered files are **shallow-merged** at the top level: files closer to the CWD win, and a closer file's top-level table fully replaces the same key in a farther file (tables are *not* recursively merged). Attributes ---------- .. autoapisummary:: fleche.config.logger fleche.config._live_caches fleche.config._STORAGE_NAME_MAPPING fleche.config._STORAGE_CLASS_TO_NAME Functions --------- .. autoapisummary:: fleche.config._load_config fleche.config._collect_config_paths fleche.config._load_merged_config fleche.config.load_default_metadata fleche.config.storage_from_config fleche.config._asdict_init_only fleche.config.storage_to_config fleche.config.cache_from_config fleche.config.cache_to_config fleche.config._default_memory_cache fleche.config.load_cache_config Module Contents --------------- .. py:data:: logger .. py:data:: _live_caches :type: dict[str | None, fleche.caches.BaseCache] .. py:function:: _load_config(path: pathlib.Path) -> dict[str, Any] .. py:function:: _collect_config_paths() -> list[pathlib.Path] Return config paths in priority order (closest first, lowest last). Walks from the current working directory up to ``$HOME`` (inclusive), collecting any ``fleche.toml`` files encountered. If the walk reaches the filesystem root without crossing ``$HOME`` (or ``$HOME`` is unset), it stops at the root. Finally, ``$XDG_CONFIG_HOME/fleche/cache.toml`` is appended as the lowest-priority fallback. Per the XDG base directory spec, an unset or empty ``XDG_CONFIG_HOME`` defaults to ``$HOME/.config``. .. py:function:: _load_merged_config() -> dict[str, Any] Load and shallow-merge all config files on the walk path. Files closer to the CWD override files farther away. Top-level keys from the closest file fully replace the same key from any farther file (no recursive table merging). .. py:function:: load_default_metadata() Load the default metadata from the merged configuration files. .. py:data:: _STORAGE_NAME_MAPPING .. py:data:: _STORAGE_CLASS_TO_NAME :type: dict[type, str] .. py:function:: storage_from_config(d: dict[str, Any], type: Literal['call']) -> fleche.storage.CallStorage storage_from_config(d: dict[str, Any], type: Literal['value']) -> fleche.storage.ValueStorage Construct a :class:`~fleche.storage.StorageBackend` from a config dict. The dict must contain a ``"type"`` key (case-sensitive, lowercase) and any additional parameters required by that storage backend. The input dict is **not** mutated. Supported type values and their parameters: * ``{"type": "memory"}`` * ``{"type": "void"}`` * ``{"type": "pickle", "root": ""}`` — optional: ``compress``, ``lock_timeout``, ``secret_key`` (list of hex strings), ``remaining_depth`` (value only) * ``{"type": "cloudpickle", "root": ""}`` — same optional keys as ``"pickle"`` * ``{"type": "dill", "root": ""}`` — same optional keys as ``"pickle"`` * ``{"type": "bagofholding_hdf", "root": ""}`` — optional: ``lock_timeout``, ``version_validator``, ``remaining_depth`` (value only) * ``{"type": "sql", "url": ""}`` *(call storage only)* — optional: ``echo`` See the module docstring for full descriptions of each key. .. py:function:: _asdict_init_only(obj) -> dict[str, Any] Like ``dataclasses.asdict()`` but restricted to ``init=True`` fields. ``init=False`` fields are internal state (locks, caches) that must not appear in serialised config. .. py:function:: storage_to_config(s: fleche.storage.ValueStorage | fleche.storage.CallStorage) -> dict[str, Any] Convert a Storage instance to a config dict (inverse of ``storage_from_config``). The returned dict contains a ``"type"`` key and any additional parameters needed to reconstruct the storage via :func:`storage_from_config`. .. py:function:: cache_from_config(d: dict[str, Any] | list[dict[str, Any]]) -> fleche.caches.BaseCache Construct a :class:`~fleche.caches.BaseCache` from a config dict or list. The cache type is determined **implicitly** from the shape of the input: - A **list** of dicts is treated as a :class:`~fleche.caches.CacheStack`, with each element processed recursively. - A **dict** containing a ``max_size`` key creates a :class:`~fleche.caches.SizeLimitedCache`. - A **dict** containing ``read_only: true`` wraps the resulting cache in a :class:`~fleche.caches.ReadOnlyCache`. - Otherwise a plain :class:`~fleche.caches.Cache` is created. The input dict is **not** mutated. .. rubric:: Examples >>> c = cache_from_config({"values": {"type": "memory"}, "calls": {"type": "memory"}}) >>> type(c).__name__ 'Cache' >>> c = cache_from_config({"values": {"type": "memory"}, "calls": {"type": "memory"}, "max_size": 100}) >>> isinstance(c, caches.SizeLimitedCache) True >>> c = cache_from_config({"values": {"type": "memory"}, "calls": {"type": "memory"}, "read_only": True}) >>> isinstance(c, caches.ReadOnlyCache) True >>> c = cache_from_config([{"values": {"type": "memory"}, "calls": {"type": "memory"}}, {"values": {"type": "void"}, "calls": {"type": "void"}}]) >>> isinstance(c, caches.CacheStack) True .. py:function:: cache_to_config(c: fleche.caches.BaseCache) -> dict[str, Any] | list[dict[str, Any]] Convert a :class:`~fleche.caches.BaseCache` to a config dict or list. This is the inverse of :func:`cache_from_config`. The output can be round-tripped back via ``cache_from_config(cache_to_config(cache))``. - :class:`~fleche.caches.Cache` → dict with ``"values"`` and ``"calls"`` - :class:`~fleche.caches.SizeLimitedCache` → same dict plus ``"max_size"`` - :class:`~fleche.caches.ReadOnlyCache` wrapping a ``Cache`` or ``SizeLimitedCache`` → inner cache dict with ``"read_only": True`` - :class:`~fleche.caches.CacheStack` → list of dicts :raises ValueError: for unsupported cache types or unsupported ``ReadOnlyCache`` inner types. .. py:function:: _default_memory_cache(name: str | None, reason: str | None = None) -> fleche.caches.Cache Return (and intern) a fresh in-memory cache, optionally logging the fallback reason. .. py:function:: load_cache_config(name: str | None = None) -> fleche.caches.BaseCache Load a cache from the configuration file. If name is None, the default cache is loaded. The names 'memory', 'void', and 'default' are special-cased: 'memory' and 'void' return transient backends; 'default' resolves to whichever cache the config file designates as the default (equivalent to calling this function with ``name=None``). Note: The `Tags` metadata cannot be configured from the config file.