Source code for osprey_connectors.ipc._pool

"""Several connector-host children at once, one per target, behind one object.

A process that talks to more than one control-system target — production, a
read-only view of it, and a simulator IOC serving the *same* channel names, say
— cannot hold them in one address space: a Channel Access client reads its
environment once, process-wide, and pins the process to one gateway for its
lifetime. :class:`ConnectorHostPool` gives each target a
:mod:`connector-host child <osprey_connectors.ipc._host>` of its own and hands
the caller a connector-shaped object per target, so a caller can drive any
number of targets from one process without knowing any of that.

::

    pool = ConnectorHostPool(control_system_section, config_file="config.yml")
    live = await pool.connector("live")
    live_ro = await pool.connector("live", execution_mode="readonly")
    sim = await pool.connector("standin")
    value = await sim.read_channel("SR:DCCT")
    await pool.close()

Keys and children
-----------------
A child is keyed by ``(target, execution_mode)``: ``live`` and ``live`` in
``readonly`` mode are two children, each with its own gateway selection. A
caller may also name the **owner** whose switch the child's writes follow and
a program's **write authorization**
(:mod:`~osprey_connectors.write_authorization`) they are judged by; the key is
then ``(target, execution_mode, owner, authorization)``, and the child's
environment carries ``OSPREY_CONTROL_OWNER`` and
``OSPREY_WRITE_AUTHORIZATION`` for it. A key that names neither stays the
two-part key it always was, and its child is spawned exactly as before but
without any authorization this process's own environment names: a child is
authorized only by its key. A new authorization is therefore a new key and a
new child, which selects its gateway on that authorization.
:meth:`ConnectorHostPool.connector` brings the key's child up if there is none
and returns a :class:`PooledConnector` bound to the *key*, not to the process —
so a handle a caller keeps goes on working across a replacement child. Calls to
different keys never wait on each other; the only lock is per key, around
bringing a child up and around :meth:`PooledConnector.disconnect`, which waits
for a start in progress and stops the child it produced.

Bringing a child up
-------------------
The ``control_system`` section is resolved against the environment
(``${VAR}`` placeholders, via :func:`~osprey_connectors.config.resolve_env_vars`)
**at every spawn**, in this process, so a port chosen after the pool was built
is the one a fresh child dials. A placeholder that stays unresolved in the
target's connector block refuses the spawn: a child sent the literal text
``${EPICS_TESTING_PORT}`` as a port would configure an endpoint nobody meant.

Two more refusals come from the resolved section alone, before anything is
spawned — the pool's share of the eligibility gates the controls MCP server's
target switch runs:

* a Channel Access type (``epics``, ``virtual_accelerator``, ``live_standin``)
  whose block has no gateway for the role this run selects. Its child would set
  no ``EPICS_CA_*`` variable at all and search by broadcast, which the real
  machine can answer — and the post-connect check below would pass it, since a
  gatewayless derivation expects a child that configured nothing;
* a simulated or stand-in target (``standin``, ``va``) whose selected endpoint
  is one the ``live`` target derives — its read-only or write-access gateway:
  the same address string, stripped and case-folded, and the same port as
  verification compares ports — such as a stand-in port variable set to the
  live gateway's port, or a live block copied into ``live_standin``. That is
  all it requires: unlike the switch, the pool asks for no loopback host (a
  stand-in may be a compose service such as ``tuning-epics-ioc:5064``) and
  resolves no name.

The child's post-connect report is then verified against what this process
derives the child should have done
(:func:`~osprey_connectors.ipc.derive_endpoints` and
:func:`~osprey_connectors.ipc.verify_host_report`, the same
check the controls MCP server's target switch runs), and the report must name
the target and connector type this process resolved. A child that fails any of
it is stopped before the caller sees anything.

Write posture is never granted by the pool. The child decides every write by
the write verdict (:func:`osprey_connectors.posture_store.write_verdict`) over
the ``control_system`` section of the project config at ``config_file`` — not
the section it was sent — and the login's switch for its target, read from the
record its inherited environment names; with no config file reachable its
writes stay off. ``execution_mode="readonly"`` can only take writes away.
Verification compares the write posture the child reports with the verdict this
process takes over the section it was given, for every connector type, gateways
or none: a child armed where that verdict refuses — or unarmed where it allows —
is refused, so the section a caller hands the pool and ``config_file`` must
agree. A switch turned between this process's verdict and the child's report
is not such a disagreement: the verdict is taken again, and when it moved the
child is replaced by a fresh one, once, before the start fails.

The gateway a child selects follows the verdict at spawn time and does not
follow a switch turned later; a caller that wants the next write to use the
write gateway after a turn-on asks for a fresh child (the notebook kernel's
runtime does, by disconnecting the handle). Until then the child refuses an
allowed write ``switching`` when a write gateway is configured and was not
selected. A block with a ``write_access`` gateway and no ``read_only`` one is
read through the write gateway while writes are off.

A key that names an authorization selects its gateway on the authorization's
verdict, channels aside (:func:`~osprey_connectors.write_authorization.authorized_write_verdict`),
here and in the child alike. A child of such a key that came up before its
authorization could write — its handle taken before the grant was approved or
the lease started — refuses a write ``switching`` without spending a step, and
that refusal comes back to the caller like any other: take the handle once the
lease is open.

Failure
-------
Every error the pool raises of its own is a :class:`ConnectorHostError`, which
is a :class:`~osprey_connectors.errors.ConnectorUnavailableError`: the
connector is gone. Its message is the operator-facing sentence, and its
``recovery`` names what the pool did about it.

* :class:`ConnectorHostStartError` — the child could not be brought up, and
  nothing was left running: unresolved config, spawn failure, no answer to its
  init frame within ``start_timeout_s``, exit before answering, or a report that
  failed verification. ``stage`` says which; an ``init``-stage failure names the
  last start phase the child reached, with times (``phases``).
* A child that answers its init frame with an **error of its own** — the
  connector's ``connect()`` failing, say — has that error raised as-is, in the
  class the child raised it as, with a note naming the child and target.
* :class:`ConnectorHostLostError` — the child went away with the call in
  flight: it exited (``cause="exited"``), or the pool stopped it because it was
  closed or :meth:`PooledConnector.disconnect` was called
  (``cause="stopped"``). The call fails; the child is dropped; the **next**
  call to that key brings up a fresh one. The same error, with
  ``cause="unavailable"``, answers a call whose connector reported itself
  unavailable (a :class:`~osprey_connectors.errors.ConnectorUnavailableError`,
  or a bare :class:`ConnectionError` from a connector that does not say which)
  in a child that answered: the child is discarded once and the next call
  starts a fresh one, and ``recovery`` says so (``child_replaced``).
* :class:`ConnectorHostUnresponsiveError` — a ``ConnectorHostLostError`` that
  is also a :class:`TimeoutError` (``cause="unresponsive"``): the child let a
  call's own ``timeout`` plus ``timeout_grace_s`` pass — or ``call_deadline_s``
  for a call with no timeout — without replying, **and then failed to answer a
  ping** within ``ping_timeout_s``. It is killed, and every other call in flight
  on it fails the same way. A ``TimeoutError`` because that is what the call
  would have raised in-process: the write may or may not have landed.
* :class:`ConnectorHostWriteTimeoutError` — a ``ConnectorHostLostError`` that is
  also a :class:`TimeoutError` (``cause="write_timeout"``), from a pool built
  with ``kill_on_write_timeout=True`` only: a *write* missed its deadline, and
  its child was killed **without** being pinged. Any other call in flight on
  that child fails as a ``ConnectorHostLostError`` with the same cause.

Which failure restarts what is decided by the one recovery policy,
:mod:`osprey_connectors.recovery`, at lazy timing: the rules this section
states are that policy's table, and every call that fails on a child the
policy replaced carries the policy's record in ``recovery``, whichever error
reached the pool first.
A child the pool stopped itself (``cause="stopped"``) is the pool's own answer
and no recovery.

A failed call is **never retried**: a write may already have reached the IOC,
and only the caller can decide whether to send it again. A refusal is an
answer, not a failure: the recovery policy never hears of one.

A child that misses a call's deadline but **does** answer the ping is alive and
merely slow — a batched or confirmed write can legitimately take several of its
call's timeouts — so it is left running, and the call raises a plain
:class:`TimeoutError`; it may still complete in the child. Its ``recovery``
says the child was kept (``child_kept``); a child that missed the ping too is
replaced (``child_replaced``).

That has an ordering hazard for writes. The child serves each request as a task
of its own, so a write that timed out here may still reach the machine **after
a newer write to the same channel** that the caller sent and saw complete:
``SP=1`` times out, ``SP=2`` completes, then ``SP=1`` lands. A caller for whom
that matters builds the pool with ``kill_on_write_timeout=True``, which kills
the child of any write that misses its deadline, answering or not, so nothing
left in it can send the stale write later. The write itself still may or may
not have landed; reads keep the ping rule either way.

A failure about one channel the *connector* raised in a healthy child — a
:class:`~osprey_connectors.errors.ChannelUnreachableError`, say — is passed
through unchanged and leaves the child in place, as is a connector bug
(:class:`~osprey_connectors.errors.ConnectorInternalError`). A connection
failure the proxy made itself (the pipe is gone) and one the child sent are
told apart by origin (:func:`~osprey_connectors.ipc.raised_by_child`),
never by timing.

Event-loop affinity
-------------------
The pool is bound to the event loop it is first used on. Its children's pipes,
its locks and every in-flight future belong to that loop, and a call from any
other loop raises :class:`RuntimeError`. A synchronous caller drives it from
one dedicated loop thread (``asyncio.run_coroutine_threadsafe`` onto that
loop); nothing here is thread-safe on its own.

Lifetime
--------
:meth:`ConnectorHostPool.close` stops every child, waiting for any that is
still starting and for any teardown already under way. A child also never
outlives this process: it exits when its stdin reaches end-of-file — which the
kernel delivers when this process dies, however it dies, because the pipe
descriptors are non-inheritable (PEP 446) and no sibling child holds another's
pipe open — and its watchdog exits it once its parent PID differs from the one
recorded at start.

Teardown is shielded from the caller's cancellation: a call cancelled while its
child is being killed does not leave the kill half-done.

Nothing in this module imports a control-system client library, and nothing
in it sets an ``EPICS_*`` variable.
"""

from __future__ import annotations

import asyncio
import contextlib
import copy
import logging
import os
import sys
import time
from collections.abc import Awaitable, Callable, Coroutine, Mapping, Sequence
from dataclasses import dataclass
from os import PathLike
from pathlib import Path
from typing import TypeVar

from osprey_connectors import posture_store, write_authorization
from osprey_connectors.config import resolve_env_vars, unresolved_placeholders
from osprey_connectors.control_system._base import ChannelValue, ChannelWriteResult
from osprey_connectors.errors import ConnectorUnavailableError, RecoveryRecord
from osprey_connectors.ipc._host import START_PHASES
from osprey_connectors.ipc._launch import (
    AttributedReader,
    host_env,
    kill_host,
    spawn_host,
    stop_host,
)
from osprey_connectors.ipc._proxy import (
    ChildUnresponsiveError,
    ConnectorHostProxy,
    _expect,
    raised_by_child,
)
from osprey_connectors.ipc._verification import (
    ROLE_WRITE_ACCESS,
    TargetDerivation,
    connector_block,
    derive_endpoints,
    live_collision,
    readonly_gateway_refusal,
    verify_host_report,
)
from osprey_connectors.posture_env import is_readonly_run
from osprey_connectors.process import DEFAULT_TERMINATE_GRACE_S, reap_exit_code, terminate
from osprey_connectors.recovery import RecoveryAction, RecoveryPolicy, Timing
from osprey_connectors.types import TRANSPORT_CA, machine_kind

__all__ = [
    "DEFAULT_CALL_DEADLINE_S",
    "DEFAULT_PING_TIMEOUT_S",
    "READONLY",
    "ConnectorHostError",
    "ConnectorHostLostError",
    "ConnectorHostPool",
    "ConnectorHostSlowCallError",
    "ConnectorHostStartError",
    "ConnectorHostUnresponsiveError",
    "ConnectorHostWriteTimeoutError",
    "PooledConnector",
]

# Named for the public namespace, not this private module, so a logging config
# keyed on the name keeps matching whatever the module is called.
logger = logging.getLogger("osprey_connectors.ipc.pool")

_T = TypeVar("_T")

#: The one execution mode a key may carry besides ``None``.
READONLY = "readonly"

#: Bound on "process started and answered its init frame". Generous, because it
#: covers a cold import of a control-system client library.
DEFAULT_START_TIMEOUT_S = 30.0

#: Local deadline for a call that names no timeout of its own.
DEFAULT_CALL_DEADLINE_S = 60.0

#: How long a child that missed a call's deadline gets to answer a ping before
#: it is judged wedged and killed.
DEFAULT_PING_TIMEOUT_S = 2.0

#: Stages of :class:`ConnectorHostStartError`.
STAGE_CONFIG = "config"
STAGE_SPAWN = "spawn"
STAGE_INIT = "init"
STAGE_VERIFY = "verify"

#: Causes of :class:`ConnectorHostLostError`.
CAUSE_EXITED = "exited"
CAUSE_UNRESPONSIVE = "unresponsive"
CAUSE_STOPPED = "stopped"
CAUSE_WRITE_TIMEOUT = "write_timeout"
CAUSE_UNAVAILABLE = "unavailable"

#: The calls that can move the machine, as :class:`PooledConnector` names them
#: to :meth:`ConnectorHostPool._invoke`. ``write_channel_checked`` is composed
#: in the proxy from ``write_channel``, so its deadline is that write's.
_WRITE_METHODS = frozenset({"write_channel", "write_multiple_channels", "write_channel_checked"})

#: ``(target, execution_mode)``, or ``(target, execution_mode, owner,
#: authorization)`` for a key that names an owner or an authorization.
_Key = tuple[str, str | None] | tuple[str, str | None, str | None, str | None]


def _make_key(target: str, mode: str | None, owner: str | None, authorization: str | None) -> _Key:
    """The pool key: two parts when it names neither owner nor authorization."""
    if owner is None and authorization is None:
        return (target, mode)
    return (target, mode, owner, authorization)


def _key_parts(key: _Key) -> tuple[str, str | None, str | None, str | None]:
    """``(target, execution_mode, owner, authorization)`` of any key."""
    if len(key) == 2:
        return key[0], key[1], None, None
    return key[0], key[1], key[2], key[3]


def _child_env(key: _Key) -> dict[str, str]:
    """The environment a child for *key* is spawned with.

    :func:`~osprey_connectors.ipc._launch.host_env`, with the key's owner
    stamped as ``OSPREY_CONTROL_OWNER`` when it names one, and
    ``OSPREY_WRITE_AUTHORIZATION`` set to the key's authorization or removed:
    a child is authorized by its key alone, never by what this process's own
    environment names.
    """
    _target, _mode, owner, authorization = _key_parts(key)
    env = host_env()
    if owner is not None:
        env[posture_store.CONTROL_OWNER_ENV_VAR] = owner
    env.pop(write_authorization.AUTHORIZATION_ENV_VAR, None)
    if authorization is not None:
        env[write_authorization.AUTHORIZATION_ENV_VAR] = authorization
    return env


# ---------------------------------------------------------------------------
# Errors
# ---------------------------------------------------------------------------


[docs] class ConnectorHostError(ConnectorUnavailableError): """A connector-host child could not serve a call (see the module docstring). Attributes: target: The control target the child serves. execution_mode: ``"readonly"`` or ``None``. pid: The child's process id, or ``None`` when none was started. """
[docs] def __init__( self, message: str, *, target: str, execution_mode: str | None, pid: int | None, recovery: tuple[RecoveryRecord, ...] = (), ) -> None: super().__init__(message, recovery=recovery) self.target = target self.execution_mode = execution_mode self.pid = pid
[docs] class ConnectorHostStartError(ConnectorHostError): """A child could not be brought up. Nothing was left running. Attributes: stage: ``"config"`` (an unresolved placeholder, a Channel Access type with no gateway to pin it to, or a simulated or stand-in target that would dial an endpoint ``live`` derives), ``"spawn"`` (the interpreter could not be started), ``"init"`` (no answer within the start timeout, an exit before answering, or an unusable answer) or ``"verify"`` (the report did not match the derivation). phases: The start marks the child had written when an ``init``-stage failure was raised, as ``(name, seconds after the spawn)`` in the order of :data:`~osprey_connectors.ipc._host.START_PHASES`; empty for every other stage, and for a child that wrote none. """
[docs] def __init__( self, message: str, *, target: str, execution_mode: str | None, pid: int | None, stage: str, phases: tuple[tuple[str, float], ...] = (), ) -> None: super().__init__(message, target=target, execution_mode=execution_mode, pid=pid) self.stage = stage self.phases = phases
class _StartMarkPipe: """The pipe a starting child writes its start marks to. Opened only where a child can inherit a pipe by descriptor; elsewhere there is no descriptor to pass and nothing is ever read. """ def __init__(self) -> None: self._read_fd: int | None = None self.write_fd: int | None = None self._written = b"" if os.name == "posix": self._read_fd, self.write_fd = os.pipe() os.set_blocking(self._read_fd, False) def close_write(self) -> None: """Close this process's copy of the write end.""" if self.write_fd is not None: with contextlib.suppress(OSError): os.close(self.write_fd) self.write_fd = None def close(self) -> None: """Close both ends.""" self.close_write() if self._read_fd is not None: with contextlib.suppress(OSError): os.close(self._read_fd) self._read_fd = None def read(self, spawned_at: float) -> tuple[tuple[str, float], ...]: """Every whole mark written so far, as (name, seconds after *spawned_at*).""" while self._read_fd is not None: try: chunk = os.read(self._read_fd, 4096) except OSError: # Nothing more is written yet (BlockingIOError), or the pipe is unusable. break if not chunk: break self._written += chunk whole = self._written[: self._written.rfind(b"\n") + 1] marks: list[tuple[str, float]] = [] for line in whole.decode("ascii", "replace").splitlines(): name, _, at = line.partition(" ") if name not in START_PHASES: continue try: marks.append((name, float(at) - spawned_at)) except ValueError: continue return tuple(marks) def _describe_start(phases: tuple[tuple[str, float], ...]) -> str: """One sentence on how far a child got in starting, from the marks it wrote.""" if not phases: return "It had not finished starting its interpreter and loading the connector host." last, seconds = phases[-1] reached = { "main": f"It was up after {seconds:.1f}s and had not read its init frame.", "init": ( f"It read its init frame after {seconds:.1f}s and was importing the connector modules." ), "imports": ( f"It had imported the connector modules after {seconds:.1f}s and was building " "its connector (connect())." ), "connected": f"It had built its connector after {seconds:.1f}s and had not answered.", }[last] listed = ", ".join(f"{name} +{at:.1f}s" for name, at in phases) return f"{reached} Start marks: {listed}"
[docs] class ConnectorHostLostError(ConnectorHostError): """The child went away with this call in flight; the call was not retried. Attributes: cause: ``"exited"`` (the child died), ``"stopped"`` (the pool was closed or the key disconnected), ``"unresponsive"`` (see :class:`ConnectorHostUnresponsiveError`), ``"write_timeout"`` (a write on the child missed its deadline and the pool, built with ``kill_on_write_timeout``, killed it; see :class:`ConnectorHostWriteTimeoutError`) or ``"unavailable"`` (the child's connector reported itself unavailable and the pool discarded the child). returncode: The child's exit status once reaped, when known. """
[docs] def __init__( self, message: str, *, target: str, execution_mode: str | None, pid: int | None, cause: str, returncode: int | None, recovery: tuple[RecoveryRecord, ...] = (), ) -> None: super().__init__( message, target=target, execution_mode=execution_mode, pid=pid, recovery=recovery, ) self.cause = cause self.returncode = returncode
[docs] class ConnectorHostUnresponsiveError(ConnectorHostLostError, TimeoutError): """The child missed a call's deadline and then a ping, and was killed."""
class ConnectorHostSlowCallError(TimeoutError): """The child missed a call's deadline but still answered a ping, and was kept. Not a :class:`ConnectorHostLostError`: the connector is not gone, and the call may yet complete in the child. ``recovery`` holds the supervisor's ``child_kept`` record, as on :class:`ConnectorUnavailableError`. """ def __init__(self, message: str, *, recovery: Sequence[RecoveryRecord] = ()) -> None: super().__init__(message) self.recovery: tuple[RecoveryRecord, ...] = tuple(recovery)
[docs] class ConnectorHostWriteTimeoutError(ConnectorHostLostError, TimeoutError): """A write missed its deadline and its child was killed for it. Raised only by a pool built with ``kill_on_write_timeout=True``, only to the write whose deadline passed, and whether or not the child would have answered a ping (``cause="write_timeout"``). A :class:`TimeoutError` like :class:`ConnectorHostUnresponsiveError`, because that is what the write would have raised in-process: it may or may not have landed. What the kill buys is that it cannot be *sent* later — nothing is left in a child that could put it on the wire after a newer write to the same target. """
# --------------------------------------------------------------------------- # One child # --------------------------------------------------------------------------- @dataclass class _PoolChild: key: _Key process: asyncio.subprocess.Process proxy: ConnectorHostProxy reader: AttributedReader report: dict[str, object] #: Set once the pool has decided to drop this child, to the cause it named. retired: str | None = None #: The one teardown of this child, shared by everyone who asks for it. teardown: asyncio.Task[None] | None = None @property def pid(self) -> int: return int(self.process.pid) def usable(self) -> bool: return ( self.retired is None and self.process.returncode is None and self.proxy.dead_reason is None ) class _PoolTarget: """One failed call's child, as the recovery policy acts on it. The incarnation is the child's proxy. Retiring discards the child under the cause the failure names; the pool is lazy, so the next call starts a fresh child and nothing here respawns one. """ in_child = True def __init__( self, pool: ConnectorHostPool, child: _PoolChild, method: str, exc: BaseException ) -> None: self._pool = pool self._child = child self._method = method self._exc = exc self._pinged = False @property def incarnation(self) -> ConnectorHostProxy: return self._child.proxy async def ping(self) -> bool: self._pinged = True return await self._pool._answers_ping(self._child) async def retire(self) -> None: if isinstance(self._exc, ChildUnresponsiveError): cause = CAUSE_UNRESPONSIVE if self._pinged else CAUSE_WRITE_TIMEOUT elif raised_by_child(self._exc): cause = CAUSE_UNAVAILABLE else: cause = CAUSE_EXITED reason = self._pool._replaced_reason(self._child, self._method, self._exc, cause) await self._pool._discard(self._child, reason, cause) class _WritesMoved(Exception): """A starting child's writes disagreed with a verdict that has since moved to agree.""" def _label(key: _Key) -> str: target, mode, owner, authorization = _key_parts(key) label = f"target {target!r}" + (f" ({mode})" if mode else "") if owner is not None: label += f" for {owner!r}" if authorization is not None: label += f" under authorization {authorization!r}" return label def _landing(method: str) -> str: """What a lost call's message says about a write that may have been sent.""" return " The write may or may not have landed." if method in _WRITE_METHODS else "" # --------------------------------------------------------------------------- # The pool # ---------------------------------------------------------------------------
[docs] class ConnectorHostPool: """One connector-host child per key, on demand. A key is ``(target, execution_mode)``, or ``(target, execution_mode, owner, authorization)`` when the caller names either (see the module docstring). Args: control_system: The ``control_system`` config section, as loaded. Copied; ``${VAR}`` placeholders in it are resolved at each spawn. config_file: The project config the children read their write posture and limits from. Without one, a child finds whatever ``CONFIG_FILE`` (or ``./config.yml``) this process's environment leads it to, and with none its writes stay off. python: The interpreter children run under. start_timeout_s: Bound on a child starting and answering its init frame. call_deadline_s: Local deadline for a call that names no ``timeout``; a child silent that long is pinged. timeout_grace_s: How long past a call's own ``timeout`` the child may take to answer before it is pinged. The child applies the timeout to the control-system call and reports its own :class:`TimeoutError` first, which is the better error. ping_timeout_s: How long a child that missed a deadline gets to answer a ping before it is judged wedged and killed. terminate_grace_s: Time between ``SIGTERM`` and ``SIGKILL``. kill_on_write_timeout: Kill a child whose *write* misses its deadline, even one that still answers a ping, and fail the write with :class:`ConnectorHostWriteTimeoutError`. Off, a timed-out write's child is kept when it answers, and the write may still land after a newer one (see the module docstring). Reads are never affected. """
[docs] def __init__( self, control_system: Mapping[str, object], *, config_file: str | PathLike[str] | None = None, python: str = sys.executable, start_timeout_s: float = DEFAULT_START_TIMEOUT_S, call_deadline_s: float = DEFAULT_CALL_DEADLINE_S, timeout_grace_s: float = 1.0, ping_timeout_s: float = DEFAULT_PING_TIMEOUT_S, terminate_grace_s: float = DEFAULT_TERMINATE_GRACE_S, kill_on_write_timeout: bool = False, ) -> None: self._section: dict[str, object] = copy.deepcopy(dict(control_system)) self._config_file = str(Path(config_file).resolve()) if config_file else None self._python = python self._start_timeout_s = start_timeout_s self._call_deadline_s = call_deadline_s self._timeout_grace_s = timeout_grace_s self._ping_timeout_s = ping_timeout_s self._terminate_grace_s = terminate_grace_s self._policy = RecoveryPolicy( timing=Timing.LAZY, replace_on_write_timeout=kill_on_write_timeout ) self._teardowns: set[asyncio.Task[None]] = set() self._children: dict[_Key, _PoolChild] = {} self._locks: dict[_Key, asyncio.Lock] = {} self._loop: asyncio.AbstractEventLoop | None = None self._closed = False
# -- public surface --------------------------------------------------
[docs] async def connector( self, target: str, *, execution_mode: str | None = None, owner: str | None = None, authorization: str | None = None, ) -> PooledConnector: """The connector for *target*, bringing its child up if there is none. Args: target: ``"live"``, ``"va"`` or ``"standin"``, resolved per :func:`~osprey_connectors.types.resolve_target`. execution_mode: ``None``, or ``"readonly"`` for a child that refuses every write whatever the config arms. owner: The login whose switch the child's writes follow, stamped as ``OSPREY_CONTROL_OWNER`` in its environment; ``None`` leaves the owner to the ladder the child inherits. authorization: The id of the program's write authorization the child's writes are judged by, set as ``OSPREY_WRITE_AUTHORIZATION`` in its environment; ``None`` is a child that names none. Raises: ValueError: *target* does not resolve on this deployment, *execution_mode* is not one of the two accepted values, or *owner* or *authorization* is not a non-blank name. ConnectorHostStartError: The child could not be brought up. Exception: The child's own typed error from ``connect()``. """ if execution_mode not in (None, READONLY): raise ValueError(f"execution_mode must be None or {READONLY!r}, got {execution_mode!r}") for name, value in (("owner", owner), ("authorization", authorization)): if value is not None and (not isinstance(value, str) or not value.strip()): raise ValueError(f"{name} must be None or a non-blank name, got {value!r}") key = _make_key( target, execution_mode, owner.strip() if owner is not None else None, authorization.strip() if authorization is not None else None, ) await self._ensure(key) return PooledConnector(self, key)
[docs] def pids(self) -> dict[_Key, int]: """The live children's process ids, by key.""" return {key: child.pid for key, child in self._children.items() if child.usable()}
[docs] async def close(self) -> None: """Stop every child. Idempotent; the pool refuses calls afterwards. Waits for a child still starting (it is stopped as soon as it is up) and for every teardown already under way, so nothing this pool started is running when this returns. """ if self._loop is not None: self._check_loop() self._closed = True # A start in progress holds its key's lock; taking each lock in turn # waits it out, and _ensure stops what it started once it sees _closed. for lock in list(self._locks.values()): async with lock: pass children = list(self._children.values()) self._children.clear() await asyncio.gather(*(self._stop(child) for child in children)) if self._teardowns: await asyncio.gather(*list(self._teardowns), return_exceptions=True)
async def __aenter__(self) -> ConnectorHostPool: return self async def __aexit__(self, *exc_info: object) -> None: await self.close() # -- calls ------------------------------------------------------------- async def _invoke(self, key: _Key, method: str, *args: object, **kwargs: object) -> object: """Run the proxy method named *method* on *key*'s child. Never retried.""" async def call(proxy: ConnectorHostProxy) -> object: bound: Callable[..., Awaitable[object]] = getattr(proxy, method) return await bound(*args, **kwargs) return await self._call(key, method, call) async def _call( self, key: _Key, method: str, call: Callable[[ConnectorHostProxy], Awaitable[_T]] ) -> _T: """Run *call*, the proxy's *method*, on *key*'s child. Never retried.""" child = await self._ensure(key) ticket = self._policy.begin() try: return await call(child.proxy) except ChildUnresponsiveError as exc: if child.retired is not None: # Another call already retired this child; its cause stands. reason = ( f"The connector-host child (pid {child.pid}) serving {_label(key)} was " f"retired ({child.retired}) while {method!r} was in flight; the call was " f"not retried.{_landing(method)}" ) await self._discard(child, reason, child.retired) raise self._lost( child, reason, child.retired, method, recovery=await self._recovered(child, method, exc, ticket), ) from exc await self._recover(child, method, exc, ticket) raise except ConnectionError as exc: if not raised_by_child(exc) and child.retired == CAUSE_STOPPED: # The pool stopped this child itself; that is no failure to recover from. reason = ( f"The connector-host child (pid {child.pid}) serving {_label(key)} was " f"stopped — the pool was closed, or the target disconnected — while " f"{method!r} was in flight; the call was not retried." ) await self._discard(child, reason, CAUSE_EXITED) raise self._lost(child, reason, CAUSE_STOPPED, method) from exc await self._recover(child, method, exc, ticket) raise async def _recover( self, child: _PoolChild, method: str, exc: BaseException, ticket: int ) -> None: """Answer a failed call as the recovery policy decides; return only when it restarts nothing.""" target = _PoolTarget(self, child, method, exc) record = await self._policy.recover( exc, target, ticket=ticket, write=method in _WRITE_METHODS ) if record is None: return if record.action is RecoveryAction.CHILD_KEPT: raise ConnectorHostSlowCallError( f"The connector-host child (pid {child.pid}) serving {_label(child.key)} did not " f"answer {method!r} in time ({exc}), but it still answers a ping, so it was " "left running. The call may yet complete in the child; it was not retried.", recovery=(record.as_dict(),), ) from exc cause = child.retired or CAUSE_EXITED reason = self._replaced_reason(child, method, exc, cause) raise self._lost(child, reason, cause, method, recovery=(record.as_dict(),)) from exc async def _recovered( self, child: _PoolChild, method: str, exc: BaseException, ticket: int ) -> tuple[RecoveryRecord, ...]: """The record of the recovery that replaced a retired child; none when the pool stopped it.""" if child.retired == CAUSE_STOPPED: return () record = await self._policy.recover( exc, _PoolTarget(self, child, method, exc), ticket=ticket, write=method in _WRITE_METHODS, ) if record is None or record.action is RecoveryAction.CHILD_KEPT: return () return (record.as_dict(),) def _replaced_reason( self, child: _PoolChild, method: str, exc: BaseException, cause: str ) -> str: """The sentence a call fails with once the pool has replaced the child it was on.""" label = _label(child.key) if cause == CAUSE_UNAVAILABLE and raised_by_child(exc): return ( f"The connector in the connector-host child (pid {child.pid}) serving " f"{label} is unavailable ({exc}); the child was discarded, the call " "was not retried, and the next call starts a fresh child." ) if cause == CAUSE_WRITE_TIMEOUT: if method in _WRITE_METHODS and isinstance(exc, ChildUnresponsiveError): return ( f"The connector-host child (pid {child.pid}) serving {label} did not " f"answer {method!r} in time ({exc}) and was killed, as this pool kills a " "child on any write timeout. The write may or may not have landed; it was " "not retried, and the next call starts a fresh child." ) return ( f"The connector-host child (pid {child.pid}) serving {label} was " f"killed because another write on it missed its deadline, while " f"{method!r} was in flight; the call was not retried, and the next call " f"starts a fresh child.{_landing(method)}" ) if cause == CAUSE_UNRESPONSIVE and isinstance(exc, ChildUnresponsiveError): return ( f"The connector-host child (pid {child.pid}) serving {label} did not " f"answer {method!r} in time, nor a ping afterwards, and was killed; the call " "was not retried, and the next call starts a fresh child." ) return ( f"The connector-host child (pid {child.pid}) serving {label} was lost " f"while {method!r} was in flight ({exc}); the call was not retried, and " "the next call starts a fresh child." ) async def _answers_ping(self, child: _PoolChild) -> bool: """Whether a child that missed a deadline is still alive enough to answer.""" if not child.usable(): return False try: await child.proxy.supervisor_request("ping", {}, self._ping_timeout_s) except Exception: return False return True def _lost( self, child: _PoolChild, detail: str, cause: str, method: str, *, recovery: tuple[RecoveryRecord, ...] = (), ) -> ConnectorHostLostError: target, mode, _owner, _authorization = _key_parts(child.key) error_class: type[ConnectorHostLostError] = ConnectorHostLostError if cause == CAUSE_UNRESPONSIVE: error_class = ConnectorHostUnresponsiveError elif cause == CAUSE_WRITE_TIMEOUT and method in _WRITE_METHODS: # Only a write is a write timeout: a read that was in flight on the # child a write timeout killed was simply lost with it. error_class = ConnectorHostWriteTimeoutError return error_class( detail, target=target, execution_mode=mode, pid=child.pid, cause=cause, returncode=child.process.returncode, recovery=recovery, ) async def _retire_key(self, key: _Key) -> None: """Stop *key*'s child in an orderly way; the next call starts another. Taken under the key's lock, so a disconnect that lands while the key's child is still starting waits for that start and then stops the child it produced: the in-flight :meth:`connector` call returns its handle, and the child behind it is gone by the time this returns. Without the lock the disconnect would find no child yet, return at once, and the child would come up anyway behind a caller that believes the key is disconnected. Nothing that already holds a key's lock reaches this. """ self._check_loop() async with self._locks.setdefault(key, asyncio.Lock()): child = self._children.pop(key, None) if child is not None: await self._stop(child) # -- children ------------------------------------------------------------ def _check_loop(self) -> None: loop = asyncio.get_running_loop() if self._loop is None: self._loop = loop elif loop is not self._loop: raise RuntimeError( "ConnectorHostPool is bound to the event loop it was first used on; " "drive it from that loop only" ) async def _ensure(self, key: _Key) -> _PoolChild: self._check_loop() if self._closed: raise RuntimeError("ConnectorHostPool is closed") child = self._children.get(key) if child is not None and child.usable(): return child lock = self._locks.setdefault(key, asyncio.Lock()) async with lock: if self._closed: raise RuntimeError("ConnectorHostPool is closed") child = self._children.get(key) if child is not None and child.usable(): return child if child is not None: await self._discard( child, f"The connector-host child (pid {child.pid}) serving {_label(key)} " "had exited; a fresh one is starting.", CAUSE_EXITED, ) child = await self._start(key) if self._closed: # close() ran while this child was starting; it waits on this # lock, so stopping the child here is what it is waiting for. await self._stop(child) raise RuntimeError("ConnectorHostPool was closed while this child was starting") self._children[key] = child return child async def _start(self, key: _Key) -> _PoolChild: try: return await self._start_once(key, may_respawn=True) except _WritesMoved: logger.info( "connector host pool: the write verdict for %s moved while its child " "was starting; starting a fresh one", _label(key), ) return await self._start_once(key, may_respawn=False) async def _start_once(self, key: _Key, *, may_respawn: bool) -> _PoolChild: """Spawn, initialise and verify one child for *key*. With *may_respawn*, a child whose reported writes disagree with the verdict taken before it was spawned — where the verdict taken again now agrees with the child, so the switch moved in between — is discarded and :class:`_WritesMoved` raised for :meth:`_start` to try once more. """ target, mode, _owner, _authorization = _key_parts(key) readonly = mode == READONLY section: dict[str, object] = resolve_env_vars(copy.deepcopy(self._section)) # Derived once, so the config refusal and the post-connect verification # judge the same posture snapshot. derivation, readonly_run, writes = self._derive(key, section) connector_type = derivation.connector_type unresolved = unresolved_placeholders( connector_block({"control_system": section}, connector_type) ) if unresolved: raise ConnectorHostStartError( f"Refusing to start a connector-host child for {_label(key)}: " f"'control_system.connector.{connector_type}' still carries " f"{', '.join(sorted(set(unresolved)))} after environment resolution. " "Set the variable before the first call to this target.", target=target, execution_mode=mode, pid=None, stage=STAGE_CONFIG, ) refusal = self._config_refusal(key, section, derivation) if refusal is not None: raise ConnectorHostStartError( f"Refusing to start a connector-host child for {_label(key)}: {refusal}", target=target, execution_mode=mode, pid=None, stage=STAGE_CONFIG, ) marks = _StartMarkPipe() spawned_at = time.monotonic() process = None try: process = await spawn_host(self._python, _child_env(key), start_marks_fd=marks.write_fd) except OSError as exc: raise ConnectorHostStartError( f"Could not spawn a connector-host child for {_label(key)}: {exc}", target=target, execution_mode=mode, pid=None, stage=STAGE_SPAWN, ) from exc finally: # The child holds the only write end from here on, so the pipe # ends when the child closes it or exits. marks.close_write() if process is None: marks.close() # spawn_host attaches both pipes. assert process.stdout is not None and process.stdin is not None reader = AttributedReader(process.stdout) proxy = ConnectorHostProxy( reader, process.stdin, timeout_grace_s=self._timeout_grace_s, deadline_s=self._call_deadline_s, ) child = _PoolChild(key=key, process=process, proxy=proxy, reader=reader, report={}) pid = child.pid init: dict[str, object] = {"control_system": section, "target": target} if self._config_file: init["config_file"] = self._config_file if readonly: init["execution_mode"] = READONLY def failure(stage: str, detail: str) -> ConnectorHostStartError: phases = marks.read(spawned_at) if stage == STAGE_INIT else () reached = f" {_describe_start(phases)}" if stage == STAGE_INIT else "" return ConnectorHostStartError( f"The connector-host child (pid {pid}) for {_label(key)} {detail}{reached}", target=target, execution_mode=mode, pid=pid, stage=stage, phases=phases, ) try: try: report = await proxy.supervisor_request("init", init, self._start_timeout_s) except ChildUnresponsiveError as exc: raise failure( STAGE_INIT, f"did not answer its init frame within {self._start_timeout_s}s." ) from exc except Exception as exc: if isinstance(exc, ConnectionError) and not raised_by_child(exc): # The pipe closed, so the child is exiting: its status is # reaped before anything signals it. A signal sent to an # exited, unreaped child reaps it out from under the event # loop's watcher, which then reports exit code 255. The # pipe is known closed here, so this wait runs the full # grace on purpose, longer than terminate's own window. await reap_exit_code(process, self._terminate_grace_s) await terminate(process, self._terminate_grace_s) raise failure( STAGE_INIT, f"exited before answering its init frame (exit code " f"{process.returncode}): {exc}", ) from exc exc.add_note(f"raised by the connector-host child (pid {pid}) for {_label(key)}") raise if not isinstance(report, dict): raise failure( STAGE_INIT, f"answered its init frame with {type(report).__name__}, " "not the post-connect report.", ) child.report = report if may_respawn and self._writes_moved(key, section, readonly_run, writes, report): raise _WritesMoved self._verify(derivation, readonly_run, writes, report, failure) except BaseException: await self._discard( child, f"The connector-host child for {_label(key)} failed to start.", CAUSE_EXITED ) raise finally: marks.close() logger.info( "connector host pool: child pid %s serving %s as %r (role=%r %s:%s)", pid, _label(key), connector_type, report.get("selected_role"), report.get("host"), report.get("port"), ) return child def _derive( self, key: _Key, section: Mapping[str, object] ) -> tuple[TargetDerivation, bool, bool]: """The derivation for *key*, and the ``(readonly_run, writes)`` it was taken under. That posture is what the child will select its gateway on, and the derivation's role is :func:`derive_endpoints`'s own: the one gateway rule. """ target, mode, _owner, _authorization = _key_parts(key) readonly_run = mode == READONLY or is_readonly_run() writes = False if readonly_run else self._key_writes_allowed(section, key) derivation = derive_endpoints( {"control_system": section}, target, writes_enabled=writes, readonly_run=readonly_run, # The child fills an unset VA gateway port from config_file, so the # derivation must read the same file or the ports disagree. config_path=self._config_file, ) return derivation, readonly_run, writes def _key_writes_allowed(self, section: Mapping[str, object], key: _Key) -> bool: """:meth:`_writes_allowed` for *key*; a two-part key asks it with two arguments.""" target, _mode, owner, authorization = _key_parts(key) if owner is None and authorization is None: return self._writes_allowed(section, target) return self._writes_allowed(section, target, owner, authorization) @staticmethod def _writes_allowed( section: Mapping[str, object], target: str, owner: str | None = None, authorization: str | None = None, ) -> bool: """Whether the write verdict allows a write to *target*, launch pin honoured. The verdict the child's own write guard takes, over the same record: the child inherits this process's environment, so it names the same owner and the same agent-data root. A key's *owner* is the one stamped in the child's environment; a key's *authorization* is judged as the child judges its gateway (:func:`~osprey_connectors.write_authorization.authorized_write_verdict`). """ return write_authorization.authorized_write_verdict( section, target, authorization=authorization, owner=owner ).allowed def _writes_moved( self, key: _Key, section: Mapping[str, object], readonly_run: bool, writes: bool, report: dict[str, object], ) -> bool: """Whether a writes mismatch in *report* is a switch that moved, not a disagreement. True only when the child reports the other posture *and* the verdict, taken again now, has moved to it. A mismatch the verdict does not explain — ``config_file`` arming what the section does not — is left for :meth:`_verify` to refuse. """ if readonly_run or report.get("writes_enabled") is not (not writes): return False return self._key_writes_allowed(section, key) is not writes def _config_refusal( self, key: _Key, section: Mapping[str, object], derivation: TargetDerivation ) -> str | None: """Why *key* must not be spawned at all, from config alone, or ``None``. The two eligibility gates the controls MCP server's target switch runs before it spawns, restated for what the pool can see — the ``control_system`` section, not the full project config: * A connector whose transport is Channel Access must select a gateway. ``connect()`` configures the process only ``if gateway_config:``, so a block with no gateway for the role this run selects sets no ``EPICS_CA_*`` variable at all — not even ``EPICS_CA_AUTO_ADDR_LIST=NO`` — and libca broadcasts its searches on the subnet, where the real machine may answer. The post-connect check cannot catch it: with no rows derived, a child that configured nothing is exactly what it expects. * A simulated or stand-in type must not select an endpoint the ``live`` target derives from the same section — read-only or write-access alike — or its writes reach the real machine under a soft label. The switch proves a stand-in positively (the co-deployed port on loopback); that needs ``virtual_accelerator.live_standin`` from the full config, which the pool is never handed, and a stand-in the pool serves may legitimately sit on a named, non-loopback host. So here the refusal is the negative half — whatever the stand-in is, it is not the machine — with addresses compared as written, never resolved. A connector whose transport is in process — the simulator served inside the child, and the test suite's dotted doubles, which have no transport row — derives no rows and dials no gateway, so neither gate applies to it. """ target = _key_parts(key)[0] connector_type = derivation.connector_type selected = derivation.selected_endpoint() block_key = f"control_system.connector.{connector_type}" if derivation.transport == TRANSPORT_CA and selected is None: if ROLE_WRITE_ACCESS in derivation.endpoints: # Only a readonly run selects no endpoint while a write gateway # is configured: it never selects write_access, and this block # has no read gateway for it. return readonly_gateway_refusal(machine_kind(section, target)) configured = sorted(derivation.endpoints) or "none" return ( f"it resolves to {connector_type!r}, which speaks Channel Access, and " f"'{block_key}.gateways.{derivation.selected_role}' — the gateway this run " f"selects — is missing or empty (configured roles: {configured}). A child " "with no gateway to pin it to would broadcast its channel searches on the " "local network, and any IOC that answers — the real machine included — " f"would serve it. Configure '{block_key}.gateways.{derivation.selected_role}'." ) collision = live_collision( {"control_system": section}, derivation, config_path=self._config_file ) if collision is None: return None chosen, row = collision.selected, collision.live return ( f"'{block_key}.gateways.{derivation.selected_role}' selects " f"{chosen.host}:{chosen.port}, which is the {collision.role!r} gateway the " f"live machine derives ('control_system.connector.{collision.connector_type}'" f".gateways.{collision.role} at {row.host}:{row.port}). Target {target!r} is " f"{connector_type!r}, never the real machine, so its reads and writes " "must not reach that endpoint. Point its gateways at its own server — " "check any port variable it takes, such as EPICS_TESTING_PORT." ) def _verify( self, derivation: TargetDerivation, readonly_run: bool, writes: bool, report: dict[str, object], failure: Callable[[str, str], ConnectorHostStartError], ) -> None: """Refuse a child whose report is not what this process derives.""" verification = verify_host_report( derivation, report, readonly_run=readonly_run, writes_enabled=writes ) if verification.ok: return if verification.field == "connector_type": raise failure( STAGE_VERIFY, f"reports connector_type {verification.got!r} where " f"{verification.expected!r} was derived.", ) if verification.field == "transport": raise failure( STAGE_VERIFY, f"reports transport {verification.got!r} where " f"{verification.expected!r} was derived.", ) if verification.field == "readonly_run": raise failure( STAGE_VERIFY, "was asked to run readonly but reports it is not in a readonly run.", ) if verification.field == "writes_enabled": child_writes = verification.got is True source = self._config_file or "none given, so CONFIG_FILE or ./config.yml" raise failure( STAGE_VERIFY, f"reports writes {'armed' if child_writes else 'off'} where the section " f"given to the pool has them {'armed' if writes else 'off'}. The child " f"takes its write tier from config_file ({source}), not from the " "section; the two must agree.", ) raise failure( STAGE_VERIFY, f"came up somewhere other than derived: {verification.detail}", ) async def _discard(self, child: _PoolChild, reason: str, cause: str) -> None: """Drop a child that failed: attribute, kill, and let the proxy settle. Only the key's *current* child is removed from the pool — a stale failure processed after a respawn must not drop the replacement. The first caller starts the teardown; every caller waits for it. """ if self._children.get(child.key) is child: del self._children[child.key] if child.teardown is None: child.retired = cause child.reader.retire(reason) child.teardown = self._track(self._kill(child)) await asyncio.shield(child.teardown) async def _stop(self, child: _PoolChild) -> None: """Stop a healthy child in order: acknowledge, then make sure it is gone.""" if child.teardown is None: child.retired = CAUSE_STOPPED child.teardown = self._track(self._orderly_stop(child)) await asyncio.shield(child.teardown) def _track(self, teardown: Coroutine[object, object, None]) -> asyncio.Task[None]: """Run a teardown as its own task, so a cancelled caller cannot cut it short.""" task: asyncio.Task[None] = asyncio.get_running_loop().create_task(teardown) self._teardowns.add(task) task.add_done_callback(self._teardowns.discard) return task async def _kill(self, child: _PoolChild) -> None: await kill_host( child.process, child.proxy, child.reader, reason=None, grace_s=self._terminate_grace_s, ) async def _orderly_stop(self, child: _PoolChild) -> None: await stop_host(child.process, child.proxy, grace_s=self._terminate_grace_s)
# --------------------------------------------------------------------------- # The caller's handle # ---------------------------------------------------------------------------
[docs] class PooledConnector: """A target's connector, as the pool serves it. Bound to a pool key, not to a process: every call goes to the key's current child, starting a fresh one if the last one was lost. The call surface mirrors :class:`~osprey_connectors.control_system.ControlSystemConnector` for the methods it offers; writes are guarded and limit-checked in the child, exactly as they would be in-process. """
[docs] def __init__(self, pool: ConnectorHostPool, key: _Key) -> None: self._pool = pool self._key = key
@property def target(self) -> str: return _key_parts(self._key)[0] @property def execution_mode(self) -> str | None: return _key_parts(self._key)[1] @property def owner(self) -> str | None: """The owner the key stamps on its child, or ``None`` for the inherited ladder.""" return _key_parts(self._key)[2] @property def authorization(self) -> str | None: """The write authorization the key's child is judged by, or ``None``.""" return _key_parts(self._key)[3] @property def pid(self) -> int | None: """The current child's pid, or ``None`` when none is running.""" child = self._pool._children.get(self._key) return child.pid if child is not None and child.usable() else None @property def report(self) -> dict[str, object]: """The current child's post-connect report, empty when none is running.""" child = self._pool._children.get(self._key) return dict(child.report) if child is not None and child.usable() else {}
[docs] async def read_channel( self, channel_address: str, timeout: float | None = None ) -> ChannelValue: return await self._pool._call( self._key, "read_channel", lambda proxy: proxy.read_channel(channel_address, timeout) )
[docs] async def read_multiple_channels( self, channel_addresses: list[str], timeout: float | None = None ) -> dict[str, ChannelValue]: return await self._pool._call( self._key, "read_multiple_channels", lambda proxy: proxy.read_multiple_channels(channel_addresses, timeout), )
[docs] async def write_channel( self, channel_address: str, value: object, timeout: float | None = None, confirm: bool | None = None, ) -> ChannelWriteResult: return await self._pool._call( self._key, "write_channel", lambda proxy: proxy.write_channel(channel_address, value, timeout, confirm), )
[docs] async def write_multiple_channels( self, operations: Sequence[tuple[str, object]], timeout: float | None = None, confirm: bool | None = None, ) -> list[ChannelWriteResult]: return await self._pool._call( self._key, "write_multiple_channels", lambda proxy: proxy.write_multiple_channels(operations, timeout, confirm), )
[docs] async def write_channel_checked( self, channel_address: str, value: object, **kwargs: object ) -> ChannelWriteResult: """Write, and raise unless the write came back confirmed or unrequested. ``ChannelWriteBlockedError`` for a refusal — writes disabled, a readonly child, limits — and ``ChannelWriteFailedError`` for a write that was sent but not confirmed. ``confirm`` and ``timeout`` pass through to :meth:`write_channel`. """ # The proxy's own write_channel_checked is the base class's function, # bound by name: it is typed for a ControlSystemConnector, not the proxy. result = await self._pool._invoke( self._key, "write_channel_checked", channel_address, value, **kwargs ) return _expect(result, ChannelWriteResult, "write_channel_checked")
[docs] async def validate_channel(self, channel_address: str) -> bool: return await self._pool._call( self._key, "validate_channel", lambda proxy: proxy.validate_channel(channel_address) )
[docs] async def disconnect(self) -> None: """Stop this key's child. A later call on this handle starts a fresh one. A child still starting is waited for and then stopped, so no child for this key is running when this returns. """ await self._pool._retire_key(self._key)