Source code for osprey_connectors.simulation.composite

"""The composite: one LUME model serving a simulator view's every channel.

:class:`Composite` reads the simulator view a render writes under
``data/simulator/`` and serves its channels through two kinds of child:

* one **physics child** per served model whose engine is not ``texture``,
  built by the engine plug-in the ``osprey.simulation.engines`` entry-point
  group names, through ``build(model, wiring, deck, settings, active=...)``;
* the **texture**, a :class:`~osprey_connectors.simulation.texture.TextureModel`
  holding every other channel and the motion of all of them.

``supported_variables`` is the channel addresses of ``addresses.json`` and its
status addresses, ``<code>:SIM:<model>:STATUS``, one per physics child. Each
child's own variables that no channel names (faults, optics) are reached as
``<model>/<name>`` through :meth:`Composite.model_get` and
:meth:`Composite.model_set` only.

**IOC health.** ``supported_variables`` also holds the view's ``ioc_health``
addresses, ``<code>:SIM:<model>:<signal>``, read-only and served for every model
the view lists whether or not it is built: ``HEARTBEAT`` counts the whole seconds
since the composite started, ``UPTIME`` is those seconds, both on its clock, and
``ENGINEER`` is a fixed demo string. No read of one reaches a child.

**Reads.** A physics readback of type float reads ``truth + motion(address,
t)``, passed through the child's ``readout`` when the engine exports one, then
clamped; the readout always sees both readings of a monitor, so a read of one
plane reads its partner too. A texture channel reads as the texture serves it.
A reading the active scenarios still has no motion: a physics readback reads
its truth through readout and clamp, a texture channel its held value.
:meth:`Composite.get` returns each value as its variable declares it, a
waveform as an array of the variable's dtype and shape.
:meth:`Composite.held` reads every channel without motion or readout.

**Writes.** :meth:`Composite.set` coerces each value to its channel's
``value_type``, then sets the physics children in their order (see
**Inputs**) and the texture last. When any child refuses, every earlier child gets its previous inputs back
and the refusal is raised as a ``WriteRefusedError`` (a ``ValueError``) carrying
the child's text and, when one channel was refused, its address. A
setpoint the active scenarios hold (``stuck`` or ``disconnected`` in
``channel_faults``) accepts a write, reads the value written, and forwards none
of it; its readback shows the model where it was.

**Channel faults.** A ``frozen`` reading reads the value it served when its
scenario became active. A ``disconnected`` reading reads not-a-number when it
is a float, else the value it served when its scenario became active; a
disconnected setpoint's readback is that reading, and a setpoint that is its
own readback reads its demand. :meth:`Composite.output_severity` names every
disconnected reading with the ``udf`` condition. A channel fault acts on live
reads, writes and severity only; :meth:`Composite.readings` serves history
without it.

**A model with no deck.** A model whose record names no deck and wires no
channel is not built: there is nothing for an engine to serve, and the
facility describes such a machine so that its channels are the texture's. Its
status is ``no deck`` (:data:`~osprey_connectors.simulation.status.STATUS_NO_DECK`),
its log records an ``unbuilt`` event, and it is no failure.

**A failed child.** A child whose engine raises while it is built or read is
failed, with the engine's error text as its status (``ok`` otherwise), capped
at :data:`_STATUS_MAX_BYTES` UTF-8 bytes. It holds the inputs written to it and
never raises: its setpoints read those inputs, its float outputs NaN, any other
output its last good value or its type's zero, and
:meth:`Composite.output_severity` names each of its channels. A refused write
leaves a child as it was. A failed child is rebuilt on :meth:`Composite.reset`
and on a change of the active scenarios.

**Scenarios.** The active set is the ``active_scenarios`` file of the state
directory, re-read when its modification time changes. A change rebuilds every
physics child with the active scenarios' writes and fault seeds as its start
state, and hands the texture the active writes it owns and the active drivers,
couplings and noise replacements; session writes are dropped. The readings
any active scenario's ``still`` list names, ``nominal`` included, are stilled.
A set whose scenarios touch one target twice is served without its
scenarios.

**Evolution.** A composite built with ``evolve=True`` drives the children
whose engine states the time contract (``advance``, ``next_deadline``,
``emitted``, ``client_write``), each looked up on the engine the way
``readout`` is: it calls ``advance(model, t)`` at its clock's reading right
after it builds or rebuilds a child, right after :meth:`Composite.reset`, and
before every read and write; :meth:`Composite.next_deadline` is the earliest
of the children's deadlines and :meth:`Composite.emitted` the union of what
they emitted, drained. Only a write through :meth:`Composite.set` that reaches
a child's model is a client write, handed to ``client_write(model, values,
t)``; :meth:`Composite.restore`, the start state and the scenario writes are
not, and neither is a write a ``stuck`` or ``disconnected`` setpoint holds. A
rebuild discards what a child had pending. Built without it, the default, the
composite calls none of the four names.

**Inputs.** A model may declare inputs, each an ``{address, engine}`` record
naming a channel another model produces. The children are ordered so every
producer precedes its consumers, ties in name order, so a view without inputs
keeps name order and makes no delivery call. A consumer whose engine exports
``take_inputs(model, inputs, t_s, *, baseline)`` gets one ``(record, value)``
pair per declared input, the value its producer's plain read: when evolving,
immediately before each ``advance`` of it, so it reads what its producers
reached in the same pass; when not evolving, right after it is built or reset
and after every :meth:`Composite.set` or :meth:`Composite.restore`. A delivery
is a baseline when it is the first after the consumer or any of its producers
was built, rebuilt or reset, and always when the composite does not evolve. A
failed producer's inputs are left out; with none left no call is made. A raise
in ``take_inputs`` fails the consumer as any time-contract call does.

**The model log.** Each physics child appends JSON lines ``{instance, pid,
...}`` to ``var/simulator/<model>.log`` under the repo root of the loaded
config, one ``os.write`` per record of under :data:`_LOG_RECORD_MAX_BYTES`
bytes. Without a loaded config the records go to the process logger only.
"""

from __future__ import annotations

import json
import math
import os
import time
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, field
from importlib import metadata
from pathlib import Path
from typing import TYPE_CHECKING, Any, Protocol

import numpy as np
from lume.exceptions import ReadOnlyError
from lume.model import LUMEModel
from lume.variables import IntVariable, NDVariable, ScalarVariable, StrVariable, Variable

from osprey_connectors.config import default_config_path
from osprey_connectors.logger import get_logger
from osprey_connectors.simulation import values
from osprey_connectors.simulation.channel_faults import CHANNEL_FAULTS, ChannelFault
from osprey_connectors.simulation.state import (
    ACTIVE_SCENARIOS_FILENAME,
    composed_set,
    overlap_record,
    parse_active_state,
    resolve_active_scenarios,
    scenario_targets,
)
from osprey_connectors.simulation.status import STATUS_NO_DECK, STATUS_OK
from osprey_connectors.simulation.texture import (
    TEXTURE_OWNER,
    TextureModel,
    channel_value_type,
    channel_variable,
    is_float_channel,
)
from osprey_connectors.simulation.view import (
    ENGINEER,
    HEARTBEAT,
    MONITOR,
    SEEDS_FILE,
    SETPOINT,
    UPTIME,
    VARIABLES_FILE,
    Model,
    SimulatorView,
    ViewSchemaError,
)
from osprey_connectors.workspace import SIMULATOR_LOG_DIR_RELPATH, repo_root_for_config

if TYPE_CHECKING:  # pragma: no cover - typing only
    from osprey_connectors.simulation._records import (
        ChannelRecord,
        CouplingTerm,
        DriveRecord,
        NoiseRecord,
        ResolvedCoupling,
        ScenarioRecord,
    )

__all__ = [
    "Composite",
    "WriteRefusedError",
    "log_dir",
]

logger = get_logger("simulation_composite")

#: The entry-point group every engine plug-in registers under.
_ENGINE_GROUP = "osprey.simulation.engines"

#: The serving instances a model log record names.
_INSTANCES = ("virtual_accelerator", "live_standin", "inprocess")

#: The longest status text, in UTF-8 bytes.
_STATUS_MAX_BYTES = 1023

#: Every model log record, newline included, is shorter than this many bytes.
_LOG_RECORD_MAX_BYTES = 4096

#: The condition :meth:`Composite.output_severity` names for a failed child's channel.
_UDF = "udf"

#: The value ``ENGINEER`` serves.
_DEMO_ENGINEER = "demo engineer"

_ELLIPSIS = "…"
_FAULT_SEPARATOR = "/"
_MODEL_SEPARATOR = "/"
_LOG_MODE = 0o664
_MS_PER_S = 1000.0


def _cap_status(text: str, max_bytes: int = _STATUS_MAX_BYTES) -> str:
    """``text`` cut to at most ``max_bytes`` UTF-8 bytes, ending in ``…`` when cut.

    Args:
        text: The status text.
        max_bytes: The bound, in UTF-8 bytes.

    Returns:
        ``text`` itself when it fits; otherwise its longest prefix of whole
        characters that fits with ``…`` appended.
    """
    raw = text.encode("utf-8")
    if len(raw) <= max_bytes:
        return text
    room = max_bytes - len(_ELLIPSIS.encode("utf-8"))
    if room <= 0:
        return ""
    return raw[:room].decode("utf-8", errors="ignore") + _ELLIPSIS


[docs] def log_dir() -> Path | None: """The directory the model logs are appended in, or ``None`` without a loaded config. Returns: ``var/simulator`` under the repo root of the config this process loaded. """ config_path = default_config_path() if config_path is None: return None return repo_root_for_config(config_path) / SIMULATOR_LOG_DIR_RELPATH
def _default_log_dir() -> Path | None: """:func:`log_dir`, looked up at call time so a replaced module function is honoured.""" return log_dir() def _plain(value: object) -> object: """A child's output in its stored representation: a waveform as a flat list.""" if isinstance(value, np.ndarray): if value.ndim == 0: item: object = value.item() return item return [float(item) for item in value.reshape(-1)] if isinstance(value, np.generic): scalar: object = value.item() return scalar return value def _for_variable(variable: Variable, value: object) -> object: """A stored value as ``variable`` takes it: a waveform as an array of its dtype and shape.""" if isinstance(variable, NDVariable): return np.asarray(value, dtype=variable.dtype).reshape(variable.shape) return value def _ioc_health_variable(address: str, signal: str) -> Variable: """The read-only variable of one IOC health address. Raises: ViewSchemaError: ``signal`` is not an IOC health signal. """ if signal == HEARTBEAT: return IntVariable(name=address, default_value=0, read_only=True) if signal == UPTIME: return ScalarVariable(name=address, default_value=0.0, unit="s", read_only=True) if signal == ENGINEER: return StrVariable(name=address, default_value=_DEMO_ENGINEER, read_only=True) raise ViewSchemaError(f"{address} is not an IOC health address: rebuild with osprey build") def _producers_first(children: Mapping[str, _Child]) -> list[str]: """The children's names, every producer before its consumers, ties in name order. A cycle, which the facility build refuses, leaves its members in name order after every child that can be placed. """ waiting = { name: {producer for _, producer in child.takes if producer in children} - {name} for name, child in children.items() } order: list[str] = [] while waiting: ready = [name for name, producers in sorted(waiting.items()) if not producers] if not ready: order.extend(sorted(waiting)) break name = ready[0] order.append(name) del waiting[name] for producers in waiting.values(): producers.discard(name) return order class _Engine(Protocol): """An engine plug-in, as the ``osprey.simulation.engines`` group names it. Its optional functions (``error_text``, ``readout``, ``take_inputs`` and the time contract) are looked up by name. """ def build( self, model: str, wiring: Sequence[Mapping[str, object]], deck: Path | None, settings: Mapping[str, object], *, active: Mapping[str, object], ) -> LUMEModel: ...
[docs] class WriteRefusedError(ValueError): """A write the composite refused; ``address`` is the one channel refused, else ``None``."""
[docs] def __init__(self, message: str, *, address: str | None = None) -> None: super().__init__(message) self.address = address
@dataclass class _Child: """One physics child and what the composite keeps for it.""" name: str source: Model owned: list[str] defaults: dict[str, object] setpoints: frozenset[str] partners: dict[str, tuple[str, ...]] engine: _Engine | None = None model: LUMEModel | None = None status: str = STATUS_OK active: dict[str, object] = field(default_factory=dict) inputs: dict[str, values.StoredValue] = field(default_factory=dict) last_good: dict[str, object] = field(default_factory=dict) starts: dict[str, object] = field(default_factory=dict) #: The active scenarios that fault this child or override one of its #: setpoints, in activation order. scenarios: tuple[str, ...] = () #: The model inputs: each declared ``{address, engine}`` record with the #: child that produces its address, in declared order. Not ``inputs``, #: which holds the setpoint values written to the child. takes: tuple[tuple[Mapping[str, object], str], ...] = () #: The children that take an input from this one. consumers: tuple[str, ...] = () #: The next delivery is a baseline: nothing was delivered since this child #: or one of its producers was built, rebuilt or reset. baseline: bool = True
[docs] class Composite(LUMEModel): """Serve one simulator view through its physics children and the texture. Args: view: The simulator view, ``<render>/data/simulator``, opened or as its directory, which is opened with :meth:`SimulatorView.open`. state_dir: The directory holding the ``active_scenarios`` file; ``None`` serves ``nominal`` alone. instance: The serving instance the model log names, one of :data:`_INSTANCES`. clock: Returns the current instant in epoch seconds. model_log: ``False`` sends the model log records to the process logger only, for a composite no instance serves. log_dir: The directory the model logs are appended in; ``None`` takes :func:`log_dir` when ``model_log`` is set. evolve: Drive the children's time contract with ``clock``; ``False`` never calls it, so every channel holds its start state until written. Raises: ValueError: ``instance`` is not one of :data:`_INSTANCES`. NoSimulatorView: The directory holds no simulator view. ViewSchemaError: A view file is not the schema this OSPREY reads. """
[docs] def __init__( self, view: SimulatorView | Path | str, *, state_dir: Path | str | None = None, instance: str = "inprocess", clock: Callable[[], float] = time.time, model_log: bool = True, log_dir: Path | str | None = None, evolve: bool = False, ) -> None: if instance not in _INSTANCES: raise ValueError(f"instance is {instance!r}; use one of {list(_INSTANCES)}") if not isinstance(view, SimulatorView): view = SimulatorView.open(view) self._instance = instance self._clock = clock self._started_s = float(clock()) self._evolve = evolve self._state_path = ( None if state_dir is None else Path(state_dir) / ACTIVE_SCENARIOS_FILENAME ) self._log_dir = ( Path(log_dir) if log_dir is not None else (_default_log_dir() if model_log else None) ) self._active: list[str] = [] self._moving: frozenset[str] = frozenset() self._held_writes: frozenset[str] = frozenset() self._faulted: dict[str, ChannelFault] = {} self._fault_severity: dict[str, str] = {} self._snapshot: dict[str, object] = {} self._texture_demands: dict[str, object] = {} variables = view.document(VARIABLES_FILE) self._scenarios: dict[str, ScenarioRecord] = { str(scenario["name"]): scenario for scenario in view._scenarios() } self._channels: dict[str, ChannelRecord] = { str(channel["address"]): channel for channel in view._channel_entries() } models = {model.name: model for model in view.physics_models()} physics = sorted(models) self._texture = TextureModel(variables, view.document(SEEDS_FILE), clock=clock) self._owner: dict[str, str] = { address: (owner if (owner := str(record.get("owner"))) in physics else TEXTURE_OWNER) for address, record in self._channels.items() } children = {name: self._child(models[name]) for name in physics} for child in children.values(): for _, producer in child.takes: if producer in children and producer != child.name: source = children[producer] if child.name not in source.consumers: source.consumers = (*source.consumers, child.name) #: The physics children, every producer before its consumers. self._children: dict[str, _Child] = { name: children[name] for name in _producers_first(children) } self._status: dict[str, str] = { address: name for name, address in view.status_addresses().items() } self._ioc_health: dict[str, str] = { address: signal for signals in view.ioc_health_by_model().values() for signal, address in signals.items() } self._variables: dict[str, Variable] = {} for address in view.channels(): if self._owner[address] == TEXTURE_OWNER: self._variables[address] = self._texture.supported_variables[address] else: child = self._children[self._owner[address]] self._variables[address] = channel_variable( self._channels[address], self._start_value(child, address) ) for address in self._status: self._variables[address] = StrVariable( name=address, default_value=STATUS_OK, read_only=True ) for address, signal in self._ioc_health.items(): self._variables[address] = _ioc_health_variable(address, signal) self._signature: tuple[str, int] | None | bool = False self._refresh()
# -- construction -------------------------------------------------------- def _child(self, source: Model) -> _Child: name = source.name owned = sorted(address for address, owner in self._owner.items() if owner == name) wiring = [binding.record for binding in source.bindings] defaults = { str(entry["address"]): entry.get("default") for entry in wiring if str(entry["address"]) in owned } setpoints = frozenset( str(entry["address"]) for entry in wiring if entry.get("direction") == "write" and str(entry["address"]) in owned ) groups: dict[str, list[str]] = {} for binding in source.bindings: if binding.engine_role == MONITOR and binding.element is not None: groups.setdefault(str(binding.element), []).append(binding.address) partners = {address: tuple(sorted(group)) for group in groups.values() for address in group} takes = tuple( (record, self._owner.get(str(record["address"]), TEXTURE_OWNER)) for record in source.inputs ) return _Child( name=name, source=source, owned=owned, defaults=defaults, setpoints=setpoints, partners=partners, takes=takes, ) def _start_value(self, child: _Child, address: str) -> values.StoredValue: """A physics channel's start value: its active write, its wiring default, or zero.""" channel = self._channels[address] value_type = channel.get("value_type") if address in child.active: return values.coerce( child.active[address], value_type, channel.get("options"), channel.get("shape") ) default = child.defaults.get(address) if default is not None: return values.coerce(default, value_type, channel.get("options"), channel.get("shape")) return values.zero(value_type, channel.get("options"), channel.get("shape")) def _start_inputs(self, child: _Child) -> dict[str, values.StoredValue]: """A child's setpoints at their start values.""" return {address: self._start_value(child, address) for address in child.setpoints} @staticmethod def _engine(name: str) -> _Engine: """The plug-in module registered for engine ``name``. Raises: LookupError: No plug-in is registered under that name. """ found = metadata.entry_points(group=_ENGINE_GROUP, name=name) for entry in found: engine: _Engine = entry.load() return engine raise LookupError(f"no simulation engine {name!r} is registered in {_ENGINE_GROUP}") def _build(self, child: _Child) -> None: """Build a child at its active state; a raise leaves it failed with the engine's text. A child that names no deck and wires no channel is left unbuilt, with the ``no deck`` status. """ source = child.source self._mark_baseline(child) child.model = None child.engine = None child.starts = {} child.inputs = self._start_inputs(child) if source.deck is None and not source.bindings: child.status = STATUS_NO_DECK self._log(child.name, {"event": "unbuilt", "status": STATUS_NO_DECK}) return try: child.engine = self._engine(str(source.engine)) child.model = child.engine.build( child.name, [binding.record for binding in source.bindings], source.deck, source.settings, active=dict(child.active), ) writable = [ name for name, variable in child.model.supported_variables.items() if name not in self._channels and not variable.read_only ] child.starts = dict(child.model.get(writable)) if writable else {} except Exception as exc: self._fail(child, exc) return child.status = STATUS_OK self._log(child.name, {"event": "built", "status": STATUS_OK}) self._advance_child(child, float(self._clock()), start=True) def _fail(self, child: _Child, exc: BaseException) -> None: """Mark a child failed with the text its engine gives for ``exc``. A child some active scenario faults or overrides names those scenarios first, so a seed with no solve says which scenario seeded it. """ error_text = getattr(child.engine, "error_text", None) text = error_text(exc) if callable(error_text) else "" if not text: text = str(exc) or type(exc).__name__ if child.scenarios: plural = "s" if len(child.scenarios) > 1 else "" text = f"scenario{plural} {', '.join(child.scenarios)}: {text}" child.model = None child.status = _cap_status(text) self._log(child.name, {"event": "failed", "error": text}) # -- the time contract --------------------------------------------------- def _time_function(self, child: _Child, name: str) -> Callable[..., Any] | None: """A built child's time-contract function ``name``; ``None`` unless evolving.""" if not self._evolve or child.model is None: return None function = getattr(child.engine, name, None) return function if callable(function) else None def _call_time(self, child: _Child, name: str, *args: object) -> object: """Call ``child``'s time-contract function ``name`` on its model with ``args``. Returns what it returns, or ``None`` when the child states no such function or is not evolving; a raise fails the child and returns ``None``. """ function = self._time_function(child, name) if function is None: return None try: result: object = function(child.model, *args) except Exception as exc: self._fail(child, exc) return None return result def _advance_child(self, child: _Child, t_s: float, *, start: bool = False) -> None: """Deliver ``child``'s inputs, then apply every step due by ``t_s``; a raise fails it. Args: child: A physics child. t_s: The clock's reading. start: The child was just built or reset: a composite that does not evolve delivers its inputs too. """ if self._evolve or start: self._deliver(child, t_s) self._call_time(child, "advance", t_s) # -- model inputs -------------------------------------------------------- def _mark_baseline(self, child: _Child) -> None: """Make the next delivery to ``child`` and to each of its consumers a baseline.""" child.baseline = True for name in child.consumers: self._children[name].baseline = True def _input_value(self, producer: str, address: str) -> tuple[bool, object]: """A producer's plain value of ``address``: ``(False, None)`` when it failed.""" if producer == TEXTURE_OWNER: return True, _plain(self._texture.get([address])[address]) read = self._plain_get(self._children[producer], [address]) if read is None: return False, None return True, read[address] def _deliver(self, child: _Child, t_s: float) -> None: """Hand a built child its producers' values through ``take_inputs``. Not a time-contract call: a composite that does not evolve delivers too, always as a baseline. A failed producer's inputs are left out and no call is made when none remain; a raise fails the child. """ if not child.takes or child.model is None: return take = getattr(child.engine, "take_inputs", None) if not callable(take): return pairs: list[tuple[Mapping[str, object], object]] = [] for record, producer in child.takes: ok, value = self._input_value(producer, str(record["address"])) if ok: pairs.append((record, value)) if not pairs: return try: take(child.model, pairs, t_s, baseline=child.baseline or not self._evolve) except Exception as exc: self._fail(child, exc) return child.baseline = False def _advance(self, t_s: float) -> None: for child in self._children.values(): self._advance_child(child, t_s)
[docs] def advance(self) -> None: """Advance every time-driven child to the clock; nothing unless built to evolve.""" self._refresh() self._advance(float(self._clock()))
[docs] def next_deadline(self) -> float | None: """The earliest epoch second a time-driven child has a step due, else ``None``. ``None`` too when the composite does not evolve; a child whose engine raises is failed and has no deadline. """ self._refresh() deadlines: list[float] = [] for child in self._children.values(): deadline = self._call_time(child, "next_deadline") if deadline is not None: deadlines.append(values._real(deadline)) return min(deadlines) if deadlines else None
[docs] def emitted(self) -> frozenset[str]: """The addresses the time-driven children emitted since the last call, drained. Empty when the composite does not evolve; a child whose engine raises is failed and emits nothing. """ self._refresh() fired: set[str] = set() for child in self._children.values(): emitted = self._time_function(child, "emitted") if emitted is None: continue try: fired.update(str(address) for address in emitted(child.model)) except Exception as exc: self._fail(child, exc) return frozenset(fired)
def _client_write(self, forwarded: Mapping[str, Mapping[str, object]], t_s: float) -> None: """Hand each child the client writes that reached its model.""" for model, written in forwarded.items(): if written: self._call_time(self._children[model], "client_write", dict(written), t_s) # -- the active scenarios ------------------------------------------------ def _state_signature(self) -> tuple[str, int] | None: if self._state_path is None: return None try: return (str(self._state_path), self._state_path.stat().st_mtime_ns) except FileNotFoundError: return None def _refresh(self) -> None: """Rebuild when the ``active_scenarios`` file changed since the last look.""" signature = self._state_signature() if signature == self._signature: return self._signature = signature names: list[str] = [] if signature is not None and self._state_path is not None: try: text = self._state_path.read_text(encoding="utf-8") except FileNotFoundError: text = "" for name in parse_active_state(text)[0]: if name in self._scenarios: names.append(name) else: logger.warning(f"Unknown scenario {name!r} in {self._state_path}; ignoring") self._rebuild(resolve_active_scenarios(names)) def _rebuild(self, active: Sequence[str]) -> None: """Start every child at the composed state of the ``active`` scenarios.""" view = {name: scenario_targets(scenario) for name, scenario in self._scenarios.items()} active, overlaps = composed_set(view, list(active)) for overlap in overlaps: logger.error(str(overlap)) for name in self._children: self._log(name, overlap_record(overlap, instance=self._instance, pid=os.getpid())) self._active = list(active) overrides: dict[str, object] = {} faults: dict[str, dict[str, object]] = {} drivers: dict[str, DriveRecord] = {} couple: dict[str, list[CouplingTerm]] = {} noise: dict[str, NoiseRecord] = {} still: set[str] = set() channel_faults: dict[str, str] = {} faulted_by: dict[str, set[str]] = {} overridden_by: dict[str, set[str]] = {} for name in active: scenario = self._scenarios.get(name) if scenario is None: # ``nominal`` when the view does not list it: a scenario that writes nothing. continue overrides.update(scenario.get("overrides") or {}) overridden_by[str(name)] = {str(a) for a in scenario.get("overrides") or {}} for model, fault in (scenario.get("faults") or {}).items(): if not fault.get("inactive"): faults.setdefault(str(model), {}).update(fault.get("writes") or {}) faulted_by.setdefault(str(model), set()).add(str(name)) drivers.update(scenario.get("drivers") or {}) for address, terms in (scenario.get("couple") or {}).items(): couple.setdefault(str(address), []).extend(terms) noise.update(scenario.get("noise") or {}) still.update(str(address) for address in scenario.get("still") or ()) channel_faults.update(scenario.get("channel_faults") or {}) resolved: dict[str, list[ResolvedCoupling]] = {} for address, terms in couple.items(): for term in terms: drive = drivers.get(str(term["driver"])) if drive is None: logger.warning(f"{address} couples to undeclared driver {term['driver']!r}") continue resolved.setdefault(address, []).append({**term, "drive": drive}) self._texture.set_motion(resolved, noise, still=frozenset(still)) self._moving = frozenset( address for address in self._owner if address in self._variables and self._texture.has_motion(address) and (self._owner[address] == TEXTURE_OWNER or self._is_moving_readback(address)) ) self._texture.set_active( { address: value for address, value in overrides.items() if self._owner.get(address) == TEXTURE_OWNER and address in self._texture.supported_variables } ) for child in self._children.values(): child.active = self._child_active(child, overrides, faults) child.scenarios = tuple( str(name) for name in active if str(name) in faulted_by.get(child.name, ()) or overridden_by.get(str(name), set()) & child.setpoints ) child.last_good = {} self._build(child) self._apply_channel_faults(channel_faults) def _child_active( self, child: _Child, overrides: Mapping[str, object], faults: Mapping[str, Mapping[str, object]], ) -> dict[str, object]: """A child's active writes and fault seeds.""" active = { address: value for address, value in overrides.items() if address in child.setpoints } for key, value in (faults.get(child.name) or {}).items(): if isinstance(value, Mapping): for name, seed in value.items(): active[f"{key}{_FAULT_SEPARATOR}{name}"] = seed else: active[str(key)] = value return dict(sorted(active.items())) def _apply_channel_faults(self, channel_faults: Mapping[str, str]) -> None: """Hold the active channel faults and the values their readings served at activation. A setpoint whose fault holds writes joins the held-write set; a disconnected setpoint faults its readback, or, read back by itself, reports the condition on itself. The snapshot is each faulted reading as served, unfaulted, at this instant. """ held: set[str] = set() faulted: dict[str, ChannelFault] = {} derived: dict[str, ChannelFault] = {} severity: dict[str, str] = {} for address, word in sorted(channel_faults.items()): fault = CHANNEL_FAULTS.get(str(word)) channel = self._channels.get(str(address)) if fault is None or channel is None: logger.warning(f"Ignoring channel fault {word!r} on {address!r}") continue if channel.get("role") != SETPOINT: faulted[str(address)] = fault continue if fault.write: held.add(str(address)) pair = str(channel.get("pair") or address) if fault.read is None: continue if pair != address and pair in self._channels: derived[pair] = fault elif fault.severity is not None: severity[str(address)] = fault.severity faulted = {**derived, **faulted} severity.update( {address: fault.severity for address, fault in faulted.items() if fault.severity} ) self._held_writes = frozenset(held) self._faulted = dict(sorted(faulted.items())) self._fault_severity = severity self._moving = self._moving - frozenset(faulted) self._snapshot = self._served(list(self._faulted), float(self._clock())) self._texture_demands = {} # -- the LUME contract --------------------------------------------------- @property def supported_variables(self) -> dict[str, Variable]: """Every channel, status and IOC health address of the view, by address.""" return self._variables @property def models(self) -> list[str]: """The served physics models, sorted by name.""" return sorted(self._children) @property def active(self) -> list[str]: """The scenarios served: the active set, or ``nominal`` alone when its scenarios overlap.""" self._refresh() return list(self._active)
[docs] def moving(self) -> frozenset[str]: """The served channels whose motion can be non-zero under the active scenarios. The texture's own channels, and the physics readbacks the texture moves on their level, each when no active scenario stills, freezes or disconnects it and its seed drift, a held coupling or a non-zero noise term moves it. """ self._refresh() return self._moving
[docs] def texture_reach(self) -> dict[str, frozenset[str]]: """What a write to each settable texture channel can change, by address. The channel, the readback it echoes into, and the texture channels derived from either. No physics model reads a texture channel, so the write changes nothing else. """ return self._texture.reach()
[docs] def get(self, names: list[str] | str) -> dict[str, Any] | Any: """Read channels, each as its variable declares it; a waveform as an array of its shape. Args: names: One address or a list of addresses. Returns: The value of a single address, else the values by address. Raises: ValueError: A name is not a channel or status address of the view. """ single = isinstance(names, str) wanted = [names] if isinstance(names, str) else list(names) self._require(wanted) outputs = { name: _for_variable(self._variables[name], value) for name, value in self._get(wanted).items() } return outputs[wanted[0]] if single else outputs
def _require(self, names: Sequence[str]) -> None: for name in names: if name not in self._variables: raise ValueError(f"Variable '{name}' is not supported by the model.") def _partition( self, names: Sequence[str] ) -> tuple[dict[str, object], list[str], dict[str, list[str]]]: """Split names into the status and IOC health values, the texture's and each child's.""" statuses: dict[str, object] = {} texture: list[str] = [] physics: dict[str, list[str]] = {} for name in names: if name in self._status: statuses[name] = self._children[self._status[name]].status elif name in self._ioc_health: statuses[name] = self._ioc_health_value(self._ioc_health[name]) elif self._owner[name] == TEXTURE_OWNER: texture.append(name) else: physics.setdefault(self._owner[name], []).append(name) return statuses, texture, physics def _ioc_health_value(self, signal: str) -> str | int | float: """The value an IOC health signal serves at this instant.""" if signal == ENGINEER: return _DEMO_ENGINEER uptime = max(float(self._clock()) - self._started_s, 0.0) return int(uptime) if signal == HEARTBEAT else uptime def _get(self, names: list[str]) -> dict[str, object]: self._refresh() t_s = float(self._clock()) self._advance(t_s) outputs = self._served(names, t_s) outputs.update(self._held_demands(names)) for name in names: fault = self._faulted.get(name) if fault is not None and fault.read is not None: outputs[name] = fault.read( channel_value_type(self._channels[name]), self._snapshot.get(name) ) return {name: outputs[name] for name in names} def _served(self, names: list[str], t_s: float) -> dict[str, object]: """The channels as served at ``t_s`` without the channel faults.""" outputs, texture, physics = self._partition(names) if texture: outputs.update( {name: _plain(value) for name, value in self._texture.get(texture).items()} ) for model, owned in physics.items(): outputs.update(self._read(self._children[model], owned, t_s)) return {name: outputs[name] for name in names} def _is_moving_readback(self, address: str) -> bool: channel = self._channels[address] return channel.get("role") != SETPOINT and is_float_channel(channel) def _read(self, child: _Child, names: list[str], t_s: float) -> dict[str, object]: """A physics child's channels at ``t_s``: readbacks with motion, readout and clamp.""" readbacks = sorted( { partner for name in names if self._is_moving_readback(name) for partner in child.partners.get(name, (name,)) } ) truth = self._plain_get(child, sorted(set(names) | set(readbacks))) if truth is None: return self._failed_values(child, names) moved = self._moving_readbacks( child, {address: np.array([values._real(truth[address])]) for address in readbacks}, np.array([t_s], dtype=np.float64), ) for address in readbacks: truth[address] = float(moved[address][0]) return {name: truth[name] for name in names} def _held_demands(self, names: Sequence[str]) -> dict[str, object]: """The value last written to each of ``names`` the active scenarios hold. A physics setpoint's demand is its child's input; a texture setpoint's is the value the composite kept for it, absent until a write. """ demands: dict[str, object] = {} for name in names: if name not in self._held_writes: continue owner = self._owner[name] if owner == TEXTURE_OWNER: if name in self._texture_demands: demands[name] = self._texture_demands[name] elif name in (inputs := self._children[owner].inputs): demands[name] = inputs[name] return demands def _moving_readbacks( self, child: _Child, levels: Mapping[str, np.ndarray], times: np.ndarray ) -> dict[str, np.ndarray]: """A built physics child's moving readbacks: motion, then readout, then clamp. Args: child: A physics child whose model is built. levels: One float64 array shaped like ``times`` per readback, a monitor plane with every channel of its readout group. times: Epoch seconds, one-dimensional. Returns: One float64 array shaped like ``times`` per address of ``levels``. """ moved = { name: level + self._texture.motion(name, times, base=level) for name, level in levels.items() } readout = getattr(child.engine, "readout", None) if moved and callable(readout): for index, instant in enumerate(times): read = readout( child.model, {name: float(series_[index]) for name, series_ in moved.items()}, int(round(float(instant) * _MS_PER_S)), ) for name, series_ in moved.items(): series_[index] = float(read[name]) return { name: np.array( [self._texture.clamp(name, float(value)) for value in series_], dtype=np.float64 ) for name, series_ in moved.items() } def _plain_get(self, child: _Child, names: list[str]) -> dict[str, object] | None: """A built child's plain reads, or ``None`` once a read fails the child.""" if child.model is None: return None try: read = {name: _plain(value) for name, value in child.model.get(names).items()} except Exception as exc: self._fail(child, exc) return None child.last_good.update( {name: value for name, value in read.items() if not isinstance(value, float)} ) return read def _failed_values(self, child: _Child, names: Sequence[str]) -> dict[str, object]: """What a failed child's channels read: inputs, NaN, or the last good value.""" outputs: dict[str, object] = {} for name in names: channel = self._channels[name] value_type = channel_value_type(channel) if name in child.inputs: outputs[name] = child.inputs[name] elif value_type == "float": outputs[name] = math.nan elif name in child.last_good: outputs[name] = child.last_good[name] else: outputs[name] = values.zero( value_type, channel.get("options"), channel.get("shape") ) return outputs
[docs] def readout_group(self, address: str) -> tuple[str, ...]: """The channels a read of ``address`` reads with it, ``address`` among them. A monitor plane of a physics child is read with its partner planes; any other channel alone. Raises: ValueError: ``address`` is not a channel of the view. """ self._require([address]) owner = self._owner[address] if owner == TEXTURE_OWNER or not self._is_moving_readback(address): return (address,) return self._children[owner].partners.get(address, (address,))
[docs] def readings( self, levels: Mapping[str, float | np.ndarray], t_s: object ) -> dict[str, np.ndarray]: """Float channels as served at epoch seconds, from levels in place of held values. A moving channel, a texture float or a physics float readback, reads its level plus its motion on that level, through its child's ``readout`` when the engine exports one, then clamped; a physics setpoint reads its level. A failed child's readbacks read NaN. Args: levels: Float channel levels by address, each a scalar or an array shaped like ``t_s``; a monitor plane needs every channel of its :meth:`readout_group`. t_s: Epoch seconds, one-dimensional. Returns: One float64 array shaped like ``t_s`` per address of ``levels``. Raises: ValueError: A name is not a float channel of the view, or a monitor plane's partner has no level. """ names = sorted(levels) self._require(names) for name in names: channel = self._channels.get(name) if channel is None or not is_float_channel(channel): raise ValueError(f"{name} is not a float channel of the view") self._refresh() times = np.asarray(t_s, dtype=np.float64).reshape(-1) out: dict[str, np.ndarray] = {} readbacks: dict[str, dict[str, np.ndarray]] = {} for name in names: level = np.broadcast_to(np.asarray(levels[name], dtype=np.float64), times.shape) owner = self._owner[name] if owner == TEXTURE_OWNER: moved = level + self._texture.motion(name, times, base=level) out[name] = np.array( [self._texture.clamp(name, float(value)) for value in moved], dtype=np.float64, ) elif self._is_moving_readback(name): readbacks.setdefault(owner, {})[name] = level else: out[name] = np.array(level, dtype=np.float64) for model, owned in readbacks.items(): child = self._children[model] if child.model is None: out.update({name: np.full(times.shape, math.nan) for name in owned}) else: out.update(self._moving_readbacks(child, owned, times)) return {name: out[name] for name in names}
[docs] def held(self, names: Sequence[str]) -> dict[str, object]: """Each channel's held value: no motion, no readout, no clamp. A texture channel returns the value the texture holds; a physics channel its child's plain read; a setpoint the active scenarios hold the value last written to it; a faulted reading its unfaulted value; a status address its status; an IOC health address its value now. Args: names: Channel, status or IOC health addresses. Returns: The value by address, in its stored representation. Raises: ValueError: A name is not a channel or status address of the view. """ wanted = list(names) self._require(wanted) self._refresh() self._advance(float(self._clock())) outputs, texture, physics = self._partition(wanted) outputs.update(self._texture.held(texture)) for model, owned in physics.items(): child = self._children[model] read = self._plain_get(child, owned) if read is None: outputs.update(self._failed_values(child, owned)) else: outputs.update(read) outputs.update(self._held_demands(wanted)) return {name: outputs[name] for name in wanted}
[docs] def output_severity(self, names: Sequence[str]) -> dict[str, dict[str, str]]: """The condition of each named channel whose child failed or whose fault reports one. Args: names: Channel or status addresses. Returns: ``{name: {"condition": "udf"}}`` for each name a failed child owns, else ``{name: {"condition": <condition>}}`` for each name whose channel fault reports one; any other name is absent. """ self._refresh() severities: dict[str, dict[str, str]] = {} for name in names: child = self._children.get(self._owner.get(name, TEXTURE_OWNER)) if child is not None and child.model is None: severities[name] = {"condition": _UDF} elif name in self._fault_severity: severities[name] = {"condition": self._fault_severity[name]} return severities
[docs] def status(self, model: str) -> str: """A served physics model's status: ``ok`` or its engine's capped error text. Raises: ValueError: ``model`` is not a served physics model; the message names the served ones. """ if model not in self._children: raise ValueError(f"model {model!r} is not served; served: {self.models}") self._refresh() return self._children[model].status
[docs] def set(self, values_by_name: dict[str, Any]) -> None: """Write channels: physics children first, producers before consumers, the texture last. Args: values_by_name: Values by channel address; a waveform as a flat or nested list. Raises: WriteRefusedError: (a ``ValueError``) A name is not a channel of the view, a value is refused for its channel, or a child refuses the write; every earlier child has its previous inputs back. ``address`` names the channel when one channel was refused. ReadOnlyError: A name is not a settable channel. """ self._write(values_by_name, client=True)
[docs] def restore(self, values_by_name: dict[str, Any]) -> None: """Put channels back as :meth:`set` writes them, without a client write. For a rollback or a replay: the children's models take the values, and no engine is told a client wrote them. Raises: WriteRefusedError: As :meth:`set`. ReadOnlyError: As :meth:`set`. """ self._write(values_by_name, client=False)
def _write(self, values_by_name: Mapping[str, object], *, client: bool) -> None: self._refresh() if not values_by_name: return coerced: dict[str, values.StoredValue] = {} for name, value in values_by_name.items(): variable = self._variables.get(name) if variable is None: raise WriteRefusedError( f"Variable '{name}' is not supported by the model.", address=name ) if variable.read_only: raise ReadOnlyError(f"Variable '{name}' is read-only. Cannot be set.") channel = self._channels[name] try: coerced[name] = values.coerce( value.tolist() if isinstance(value, np.ndarray) else value, channel.get("value_type"), channel.get("options"), channel.get("shape"), ) variable.validate_value(_for_variable(variable, coerced[name])) except ValueError as exc: raise WriteRefusedError(str(exc), address=name) from exc t_s = float(self._clock()) self._advance(t_s) forwarded = self._forward(coerced) if not self._evolve: for child in self._children.values(): self._deliver(child, t_s) if client: self._client_write(forwarded, t_s) def _set(self, values_by_name: dict[str, Any]) -> None: self._forward(values_by_name) def _forward( self, values_by_name: Mapping[str, values.StoredValue] ) -> dict[str, dict[str, values.StoredValue]]: """Write the children, then the texture; returns what reached each child's model.""" batches: dict[str, dict[str, values.StoredValue]] = {} for name, value in values_by_name.items(): batches.setdefault(self._owner[name], {})[name] = value restores: list[Callable[[], None]] = [] forwarded: dict[str, dict[str, values.StoredValue]] = {} texture = batches.get(TEXTURE_OWNER) or {} demands = {name: value for name, value in texture.items() if name in self._held_writes} writing: Mapping[str, values.StoredValue] = {} try: for model, child in self._children.items(): if model in batches: writing = batches[model] restore, forwarded[model] = self._set_child(child, writing) restores.append(restore) forward = {name: value for name, value in texture.items() if name not in demands} if forward: writing = forward self._texture.set( { name: _for_variable(self._texture.supported_variables[name], value) for name, value in forward.items() } ) except Exception as exc: for restore in reversed(restores): restore() if isinstance(exc, ReadOnlyError): raise address = next(iter(writing)) if len(writing) == 1 else None raise WriteRefusedError(str(exc) or type(exc).__name__, address=address) from exc self._texture_demands.update(demands) return forwarded def _set_child( self, child: _Child, batch: Mapping[str, values.StoredValue] ) -> tuple[Callable[[], None], dict[str, values.StoredValue]]: """Write one child's batch; returns its undo and the values that reached the model. Every name of the batch lands in the child's inputs; only the names the active scenarios do not hold reach the model. """ forward = {name: value for name, value in batch.items() if name not in self._held_writes} previous = {name: child.inputs[name] for name in batch if name in child.inputs} model = child.model reached = dict(forward) if model is not None else {} if model is not None and forward: model.set( { name: _for_variable(model.supported_variables[name], value) for name, value in forward.items() } ) child.inputs.update(batch) undo = {name: value for name, value in previous.items() if name in forward} def restore() -> None: child.inputs.update(previous) if child.model is not None and undo: try: child.model.set( { name: _for_variable(child.model.supported_variables[name], value) for name, value in undo.items() } ) except Exception as exc: self._fail(child, exc) return restore, reached
[docs] def reset(self) -> None: """Return every child to its start state; a failed child is rebuilt. Setpoints return to their active writes or defaults, the texture to its nominals and active writes; session writes are dropped, the ones a channel fault holds among them. A faulted reading keeps the value it served when its fault became active. An evolving composite advances each child to the clock right after, which anchors its time anew. """ self._refresh() for child in self._children.values(): if child.model is None: self._build(child) continue try: child.model.reset() except Exception as exc: self._fail(child, exc) continue child.inputs = self._start_inputs(child) self._mark_baseline(child) self._advance_child(child, float(self._clock()), start=True) self._texture.reset() self._texture_demands = {}
# -- the children's own variables ---------------------------------------- def _model_variable(self, name: str) -> tuple[_Child, str]: model, separator, variable = name.partition(_MODEL_SEPARATOR) child = self._children.get(model) if not separator or child is None: raise ValueError( f"{name!r} names no model variable; use <model>/<name>, model in {self.models}" ) if child.model is None: if child.status == STATUS_NO_DECK: raise ValueError(f"model {model!r} is not built: it names no deck") raise ValueError(f"model {model!r} has failed: {child.status}") if variable in self._channels or variable not in child.model.supported_variables: raise ValueError(f"model {model!r} has no variable {variable!r}") return child, variable
[docs] def model_variables(self) -> dict[str, tuple[Variable, object]]: """Each built physics model's own variables, named ``<model>/<name>``. A model's own variables are the ones it declares beyond the view's channels. A failed model lists none. Returns: ``<model>/<name>`` -> ``(variable, start)``, models in name order and each model's variables in its declaration order. ``start`` is what the model held for a writable variable when it was built at the active scenarios; a read-only variable's is ``None``. """ self._refresh() listed: dict[str, tuple[Variable, object]] = {} for child in self._children.values(): if child.model is None: continue for name, variable in child.model.supported_variables.items(): if name not in self._channels: listed[f"{child.name}{_MODEL_SEPARATOR}{name}"] = ( variable, child.starts.get(name), ) return listed
[docs] def model_get(self, names: Sequence[str]) -> dict[str, object]: """Read the children's own variables, each named ``<model>/<name>``. Raises: ValueError: A name names no served model, a channel, an unknown variable, or a variable of a failed model. """ self._refresh() outputs: dict[str, object] = {} for name in names: child, variable = self._model_variable(name) assert child.model is not None outputs[name] = child.model.get(variable) return outputs
[docs] def model_set(self, values_by_name: Mapping[str, object]) -> None: """Write the children's own variables, each named ``<model>/<name>``. Raises: ValueError: A name names no served model, a channel, an unknown variable, or a variable of a failed model; or the child refuses the write. """ self._refresh() batches: dict[str, tuple[_Child, dict[str, object]]] = {} for name, value in values_by_name.items(): child, variable = self._model_variable(name) batches.setdefault(child.name, (child, {}))[1][variable] = value for child, batch in batches.values(): assert child.model is not None child.model.set(batch)
# -- the model log -------------------------------------------------------
[docs] def log_event(self, event: str, **fields: object) -> None: """Append one event record to every physics model's log. A view with no physics model logs nothing here. Args: event: The record's ``event``. **fields: Further fields of the record. """ for name in self._children: self._log(name, {**fields, "event": event})
def _log(self, model: str, record: Mapping[str, object]) -> None: """Append one record to ``model``'s log, and hand it to the process logger.""" entry = {"instance": self._instance, "pid": os.getpid(), "model": model, **record} line = self._log_line(entry) if record.get("event") == "failed": logger.warning(line) else: logger.info(line) if self._log_dir is None: return try: self._log_dir.mkdir(parents=True, exist_ok=True) fd = os.open( self._log_dir / f"{model}.log", os.O_WRONLY | os.O_APPEND | os.O_CREAT, _LOG_MODE ) try: if os.fstat(fd).st_uid == os.geteuid(): os.fchmod(fd, _LOG_MODE) os.write(fd, (line + "\n").encode("utf-8")) finally: os.close(fd) except OSError as exc: logger.warning(f"cannot append to the {model} log in {self._log_dir}: {exc}") @staticmethod def _log_line(entry: Mapping[str, object]) -> str: """One JSON line, its ``error`` cut so the line and its newline stay under the bound.""" line = json.dumps(entry, sort_keys=True, ensure_ascii=False) error = entry.get("error") while ( isinstance(error, str) and error and len(line.encode("utf-8")) >= (_LOG_RECORD_MAX_BYTES - 1) ): excess = len(line.encode("utf-8")) - (_LOG_RECORD_MAX_BYTES - 2) error = _cap_status(error, max(len(error.encode("utf-8")) - excess, 0)) line = json.dumps({**entry, "error": error}, sort_keys=True, ensure_ascii=False) return line