Supervising Connector Processes#

What a supervisor needs to run connectors in child processes: the connector host’s launch, proxy and pool, starting and stopping children, and the policy for a call that failed.

Inter-process plumbing for the connector-host child.

The child process owns the control-system client library; the parent talks to it over a pipe pair. This namespace holds what a supervisor needs: launching and stopping a child, the proxy that drives one and the pool that keeps them, the endpoint derivation and verification of what a child reports, and the two errors the wire format raises when a frame cannot be written or read. The wire codec itself and the typed-exception registry that carries a failure across the boundary are internal to this package (_frames, _exceptions).

The frame errors are imported here. Every other name is resolved on access (PEP 562): the child runs as python -m osprey_connectors.ipc._host, and a package that imported _host eagerly would load it a second time under __main__.

osprey_connectors.ipc.scrub_epics_env(environ=None)[source]#

Remove every inherited EPICS_CA_*/EPICS_PVA_* variable.

Parameters:

environ (dict[str, str] | None) – Mapping to scrub, defaulting to os.environ.

Returns:

an endpoint this child was not configured with must not be reachable from it, and a variable that survived the scrub would silently widen what connect() decided.

Return type:

The removed name/value pairs, for logging. Nothing is put back

class osprey_connectors.ipc.AttributedReader(stream)[source]#

Bases: object

A child’s stdout, able to say why reading it stopped.

The proxy fails every outstanding request with whatever ended its read stream. Left to itself that is an anonymous end-of-pipe, which tells an operator nothing about the decision that caused it — so the supervisor names the reason here before it kills the child, and the proxy’s ConnectionError carries it.

__init__(stream)[source]#
retire(reason)[source]#

Name the reason the stream is about to end. Call before the kill.

async read(count)[source]#
Return type:

bytes

osprey_connectors.ipc.host_env()[source]#

The environment a child is launched with: os.environ, minus EPICS.

The child scrubs EPICS_CA_*/EPICS_PVA_* again on its own first line, and that is the scrub the design depends on. This one is the defense-in-depth half: an ambient gateway never reaches the process that could act on it, so no window exists between exec and scrub.

The child is stamped with ENV_CHAIN_APPLIED_ENV. Reading its config file would otherwise load the project .env from its working directory after the scrub, and an EPICS_* line there would put back what both scrubs took out.

START_MARKS_FD_ENV is dropped too: a supervisor that is itself running under one never passes on a descriptor number that names something else in its child.

Return type:

dict[str, str]

async osprey_connectors.ipc.kill_host(process, proxy, reader, *, reason, grace_s)[source]#

Put down a child that failed, and let its proxy settle. Never raises.

Parameters:
  • process (asyncio.subprocess.Process) – The child’s asyncio.subprocess.Process.

  • proxy (ConnectorHostProxy) – The ConnectorHostProxy over its pipes.

  • reader (AttributedReader) – The proxy’s AttributedReader.

  • reason (str | None) – Why the child is put down, named on the reader before the kill so every request in flight fails with it. None when the caller has already retired the reader.

  • grace_s (float) – How long the child gets to exit after SIGTERM before SIGKILL.

async osprey_connectors.ipc.spawn_host(python, env, *, start_marks_fd=None)[source]#

Launch one connector-host child with its frame pipes attached.

Parameters:
  • python (str) – The interpreter to run the child under.

  • env (Mapping[str, str]) – The child’s environment, normally host_env().

  • start_marks_fd (int | None) – The write end of a pipe the child writes its start marks to (see osprey_connectors.ipc._host), or None for a child that writes none. The child inherits the descriptor under the same number; the caller closes its own copy after the spawn.

Returns:

The asyncio.subprocess.Process. stderr is inherited, so the child’s diagnostics land wherever the supervisor’s own do.

Raises:

OSError – The interpreter could not be executed. Callers wrap this in whatever their own failure type is.

Return type:

Process

async osprey_connectors.ipc.stop_host(process, proxy, *, grace_s)[source]#

Stop a healthy child in order: acknowledge, then make sure it is gone. Never raises.

Parameters:
  • process (asyncio.subprocess.Process) – The child’s asyncio.subprocess.Process.

  • proxy (ConnectorHostProxy) – The ConnectorHostProxy over its pipes.

  • grace_s (float) – How long the child gets to exit after SIGTERM before SIGKILL.

exception osprey_connectors.ipc.ConnectorHostError(message, *, target, execution_mode, pid, recovery=())[source]#

Bases: ConnectorUnavailableError

A connector-host child could not serve a call (see the module docstring).

Variables:
  • target – The control target the child serves.

  • execution_mode – "readonly" or None.

  • pid – The child’s process id, or None when none was started.

__init__(message, *, target, execution_mode, pid, recovery=())[source]#
exception osprey_connectors.ipc.ConnectorHostLostError(message, *, target, execution_mode, pid, cause, returncode, recovery=())[source]#

Bases: ConnectorHostError

The child went away with this call in flight; the call was not retried.

Variables:
  • cause – "exited" (the child died), "stopped" (the pool was closed or the key disconnected), "unresponsive" (see ConnectorHostUnresponsiveError), "write_timeout" (a write on the child missed its deadline and the pool, built with kill_on_write_timeout, killed it; see ConnectorHostWriteTimeoutError) or "unavailable" (the child’s connector reported itself unavailable and the pool discarded the child).

  • returncode – The child’s exit status once reaped, when known.

__init__(message, *, target, execution_mode, pid, cause, returncode, recovery=())[source]#
class osprey_connectors.ipc.ConnectorHostPool(control_system, *, config_file=None, python=sys.executable, start_timeout_s=DEFAULT_START_TIMEOUT_S, call_deadline_s=DEFAULT_CALL_DEADLINE_S, timeout_grace_s=1.0, ping_timeout_s=DEFAULT_PING_TIMEOUT_S, terminate_grace_s=DEFAULT_TERMINATE_GRACE_S, kill_on_write_timeout=False)[source]#

Bases: object

One connector-host child per key, on demand.

A key is (target, execution_mode), or (target, execution_mode, owner, authorization) when the caller names either (see the module docstring).

Parameters:
  • control_system (Mapping[str, object]) – The control_system config section, as loaded. Copied; ${VAR} placeholders in it are resolved at each spawn.

  • config_file (str | PathLike[str] | None) – The project config the children read their write posture and limits from. Without one, a child finds whatever CONFIG_FILE (or ./config.yml) this process’s environment leads it to, and with none its writes stay off.

  • python (str) – The interpreter children run under.

  • start_timeout_s (float) – Bound on a child starting and answering its init frame.

  • call_deadline_s (float) – Local deadline for a call that names no timeout; a child silent that long is pinged.

  • timeout_grace_s (float) – How long past a call’s own timeout the child may take to answer before it is pinged. The child applies the timeout to the control-system call and reports its own TimeoutError first, which is the better error.

  • ping_timeout_s (float) – How long a child that missed a deadline gets to answer a ping before it is judged wedged and killed.

  • terminate_grace_s (float) – Time between SIGTERM and SIGKILL.

  • kill_on_write_timeout (bool) – Kill a child whose write misses its deadline, even one that still answers a ping, and fail the write with ConnectorHostWriteTimeoutError. Off, a timed-out write’s child is kept when it answers, and the write may still land after a newer one (see the module docstring). Reads are never affected.

__init__(control_system, *, config_file=None, python=sys.executable, start_timeout_s=DEFAULT_START_TIMEOUT_S, call_deadline_s=DEFAULT_CALL_DEADLINE_S, timeout_grace_s=1.0, ping_timeout_s=DEFAULT_PING_TIMEOUT_S, terminate_grace_s=DEFAULT_TERMINATE_GRACE_S, kill_on_write_timeout=False)[source]#
async connector(target, *, execution_mode=None, owner=None, authorization=None)[source]#

The connector for target, bringing its child up if there is none.

Parameters:
  • target (str) – "live", "va" or "standin", resolved per resolve_target().

  • execution_mode (str | None) – None, or "readonly" for a child that refuses every write whatever the config arms.

  • owner (str | None) – The login whose switch the child’s writes follow, stamped as OSPREY_CONTROL_OWNER in its environment; None leaves the owner to the ladder the child inherits.

  • authorization (str | None) – The id of the program’s write authorization the child’s writes are judged by, set as OSPREY_WRITE_AUTHORIZATION in its environment; None is a child that names none.

Raises:
  • ValueError – target does not resolve on this deployment, execution_mode is not one of the two accepted values, or owner or authorization is not a non-blank name.

  • ConnectorHostStartError – The child could not be brought up.

  • Exception – The child’s own typed error from connect().

Return type:

PooledConnector

pids()[source]#

The live children’s process ids, by key.

Return type:

dict[tuple[str, str | None] | tuple[str, str | None, str | None, str | None], int]

async close()[source]#

Stop every child. Idempotent; the pool refuses calls afterwards.

Waits for a child still starting (it is stopped as soon as it is up) and for every teardown already under way, so nothing this pool started is running when this returns.

exception osprey_connectors.ipc.ConnectorHostStartError(message, *, target, execution_mode, pid, stage, phases=())[source]#

Bases: ConnectorHostError

A child could not be brought up. Nothing was left running.

Variables:
  • stage – "config" (an unresolved placeholder, a Channel Access type with no gateway to pin it to, or a simulated or stand-in target that would dial an endpoint live derives), "spawn" (the interpreter could not be started), "init" (no answer within the start timeout, an exit before answering, or an unusable answer) or "verify" (the report did not match the derivation).

  • phases – The start marks the child had written when an init-stage failure was raised, as (name, seconds after the spawn) in the order of START_PHASES; empty for every other stage, and for a child that wrote none.

__init__(message, *, target, execution_mode, pid, stage, phases=())[source]#
exception osprey_connectors.ipc.ConnectorHostUnresponsiveError(message, *, target, execution_mode, pid, cause, returncode, recovery=())[source]#

Bases: ConnectorHostLostError, TimeoutError

The child missed a call’s deadline and then a ping, and was killed.

exception osprey_connectors.ipc.ConnectorHostWriteTimeoutError(message, *, target, execution_mode, pid, cause, returncode, recovery=())[source]#

Bases: ConnectorHostLostError, TimeoutError

A write missed its deadline and its child was killed for it.

Raised only by a pool built with kill_on_write_timeout=True, only to the write whose deadline passed, and whether or not the child would have answered a ping (cause="write_timeout"). A TimeoutError like ConnectorHostUnresponsiveError, because that is what the write would have raised in-process: it may or may not have landed. What the kill buys is that it cannot be sent later — nothing is left in a child that could put it on the wire after a newer write to the same target.

class osprey_connectors.ipc.PooledConnector(pool, key)[source]#

Bases: object

A target’s connector, as the pool serves it.

Bound to a pool key, not to a process: every call goes to the key’s current child, starting a fresh one if the last one was lost. The call surface mirrors ControlSystemConnector for the methods it offers; writes are guarded and limit-checked in the child, exactly as they would be in-process.

__init__(pool, key)[source]#
property target: str#
property execution_mode: str | None#
property owner: str | None#

The owner the key stamps on its child, or None for the inherited ladder.

property authorization: str | None#

The write authorization the key’s child is judged by, or None.

property pid: int | None#

The current child’s pid, or None when none is running.

property report: dict[str, object]#

The current child’s post-connect report, empty when none is running.

async read_channel(channel_address, timeout=None)[source]#
Return type:

ChannelValue

async read_multiple_channels(channel_addresses, timeout=None)[source]#
Return type:

dict[str, ChannelValue]

async write_channel(channel_address, value, timeout=None, confirm=None)[source]#
Return type:

ChannelWriteResult

async write_multiple_channels(operations, timeout=None, confirm=None)[source]#
Return type:

list[ChannelWriteResult]

async write_channel_checked(channel_address, value, **kwargs)[source]#

Write, and raise unless the write came back confirmed or unrequested.

ChannelWriteBlockedError for a refusal — writes disabled, a readonly child, limits — and ChannelWriteFailedError for a write that was sent but not confirmed. confirm and timeout pass through to write_channel().

Return type:

ChannelWriteResult

async validate_channel(channel_address)[source]#
Return type:

bool

async disconnect()[source]#

Stop this key’s child. A later call on this handle starts a fresh one.

A child still starting is waited for and then stopped, so no child for this key is running when this returns.

exception osprey_connectors.ipc.ChildUnresponsiveError[source]#

Bases: TimeoutError

The child did not answer a request before the proxy’s own deadline.

Distinct from a TimeoutError the child reported — that one means the control-system call timed out in a child that is still answering. This one means the child itself said nothing in time, so it may be wedged.

class osprey_connectors.ipc.ConnectorHostProxy(reader, writer, *, timeout_grace_s=1.0, deadline_s=None)[source]#

Bases: object

A connector-shaped handle on the connector running in the child process.

Parameters:
  • reader (Any) – Anything with await read(n) -> bytes returning b"" at EOF — an asyncio.StreamReader, and therefore an asyncio.subprocess.Process.stdout.

  • writer (Any) – Anything with write(bytes), await drain() and close() — an asyncio.StreamWriter, and therefore an asyncio.subprocess.Process.stdin. wait_closed() is used when present.

  • timeout_grace_s (float) – Extra time the proxy waits beyond a call’s own timeout before giving up locally. The child applies the timeout to the control-system call itself and reports its own TimeoutError, which is the more informative one; the local deadline exists only so a wedged child cannot hang a caller forever. Set it to 0 to make the local deadline the only one.

  • deadline_s (float | None) – Local deadline for a call whose timeout is None. Never sent to the child — the connector keeps its own default — and None (the default) means such a call waits as long as the child takes.

The transport is duck-typed rather than a subprocess handle so tests can drive the proxy over a socketpair, and so the supervisor can hand it a process’s pipes without this class knowing anything about process lifetime. from_process() is the convenience for the latter.

__init__(reader, writer, *, timeout_grace_s=1.0, deadline_s=None)[source]#
classmethod from_process(process, *, timeout_grace_s=1.0, deadline_s=None)[source]#

Wrap an asyncio.subprocess.Process’s stdin/stdout pipes.

The proxy does not own the process: it neither starts it nor reaps it. Killing and restarting the child belongs to the supervisor.

Return type:

ConnectorHostProxy

property dead_reason: str | None#

Why the proxy is dead, or None while the child is still answering.

Set once the pipe to the child is gone for good — end of stream, an unreadable stream, a failed write, or disconnect() — and never cleared. A refusal from refuse_new_requests() is not death.

async read_channel(channel_address, timeout=None)[source]#

Read one channel. Returns the ChannelValue the child produced.

Return type:

ChannelValue

async read_multiple_channels(channel_addresses, timeout=None)[source]#

Read many channels in one request frame.

The batch crosses the boundary whole rather than as N round trips: the child’s connector decides how to fan it out, exactly as it would in-process, and one pipe round trip covers the lot.

Return type:

dict[str, ChannelValue]

async write_channel(channel_address, value, timeout=None, confirm=None)[source]#

Write one channel. Returns the ChannelWriteResult from the child.

A refusal (limits, writes-disabled) arrives as a typed exception frame and is re-raised here as the real exception class with its fields intact, so a caller cannot tell the refusal came from another process.

confirm=None leaves the keyword off the request frame entirely, so the child’s connector resolves the policy for this channel; an explicit confirm=False crosses the wire as the answer it is.

Return type:

ChannelWriteResult

async write_multiple_channels(operations, timeout=None, confirm=None)[source]#

Write many channels in one request frame.

The whole batch goes over as a single request so the child’s connector keeps whatever transactional behaviour it implements — a simulator that suppresses recalculation between writes, for instance, cannot do that if the parent has already split the batch into separate calls.

Return type:

list[ChannelWriteResult]

async write_channel_checked(channel_address, value, **kwargs)#

The base class’s own function, not a copy: see the module docstring.

Return type:

ChannelWriteResult

async validate_channel(channel_address)[source]#

Whether the channel exists and is reachable, as the child’s connector sees it.

Return type:

bool

async disconnect(*, ack_timeout=2.0)[source]#

Ask the child to release its connector, then close the pipe.

Idempotent and never raises: it is called from teardown paths and from the switch’s failure branch, where a second exception would only bury the first. The acknowledgement is best-effort — a child that has already died, or that never answers, still leaves this method returning with the pipe closed and every outstanding call failed.

async supervisor_request(method, kwargs, timeout)[source]#

Send one supervisor request (init, ping) and await its reply.

Not part of the connector surface: a supervisor that brings a child up over this proxy’s pipes, or asks whether it is still alive, uses this, so the reply is demultiplexed and its error decoded by the same plumbing every other call uses. A typed error frame is raised as its real class (and raised_by_child() says so); a child that dies before answering fails the call with a ConnectionError this proxy made; no answer within timeout plus the grace raises ChildUnresponsiveError.

Return type:

object

refuse_new_requests(reason)[source]#

Stop accepting new calls; let the in-flight ones finish.

reason is what a caller sees in the ConnectionError raised by any later call, so it should say what is happening (“target switch in progress”), not merely that something is closed. Calling this twice keeps the first reason — the first one explains the decision.

async drain(timeout)[source]#

Wait for outstanding requests to finish.

Returns True when nothing is left in flight, False when the deadline expired with calls still pending. It never raises and never cancels anything: a caller that gets False decides for itself whether to wait longer or to abandon the child.

Return type:

bool

osprey_connectors.ipc.raised_by_child(exc)[source]#

Whether exc is an error the child sent, rather than one this proxy made.

A ConnectionError can come from two places: the connector in a child that answered says it lost its own transport, or the child itself is gone. The first arrives in an error frame and is marked here as it is decoded; the second is manufactured by this proxy when the pipe ends. A supervisor classifies by this mark, never by whatever state the proxy is in when the caller happens to resume. A failure about one channel is no ConnectionError at all.

Return type:

bool

class osprey_connectors.ipc.Endpoint(host, port, mode)[source]#

Bases: object

Where one gateway role points, and how the child will say so to EPICS.

host: str#
port: object#

The port the gateway block carries, as written (or the default the derivation filled in): usually an int, but whatever the config holds.

mode: str#
as_dict()[source]#
Return type:

dict[str, object]

__init__(host, port, mode)#
class osprey_connectors.ipc.LiveCollision(selected, live, role, connector_type)[source]#

Bases: object

A live gateway that a never-live target’s selected endpoint lands on.

selected and live name the same endpoint, each as its own block spells it; role and connector_type locate live in the config.

selected: Endpoint#
live: Endpoint#
role: str#
connector_type: str#
__init__(selected, live, role, connector_type)#
class osprey_connectors.ipc.TargetDerivation(target, connector_type, endpoints, selected_role, transport, gateways_declared='missing')[source]#

Bases: object

What a connector-host child will do for a target, derived from config.

endpoints carries one row per configured role: read_only and write_access when the gateways table has them, plus pva when the block configures PVA routing and a PVA gateway — the same conjunction connect() requires before it touches a PVA environment variable.

selected_role is the row the child will actually configure the process with: EPICS keeps one process-wide context, so exactly one gateway is used.

transport is the wire the child’s connector will speak (connector_transport()), so a child that built the other venue of the simulator is refused by what it echoes.

gateways_declared is declared_gateways() of the block: only an explicit empty gateways: [] is GATEWAYS_NONE_DECLARED, the facility saying this target has nothing to probe. A missing table is not that statement — a forgotten table must never read as a direct connection — so it is GATEWAYS_MISSING. The simulator served in process reads no block, and keeps the default.

target: str#
connector_type: str#
endpoints: dict[str, Endpoint]#
selected_role: str#
transport: str | None#
gateways_declared: str = 'missing'#
property declares_no_endpoints: bool#

Whether the block spells gateways: [], and so has nothing to probe.

selected_endpoint()[source]#

The row the child will configure, or None when config has none.

Return type:

Endpoint | None

__init__(target, connector_type, endpoints, selected_role, transport, gateways_declared='missing')#
class osprey_connectors.ipc.Verification(ok, field=None, expected=None, got=None, detail='')[source]#

Bases: object

Whether the child came up where the derivation said it would.

ok: bool#
field: str | None = None#
expected: object = None#
got: object = None#
detail: str = ''#
as_dict()[source]#
Return type:

dict[str, object]

__init__(ok, field=None, expected=None, got=None, detail='')#
osprey_connectors.ipc.config_section(config, name)[source]#

One top-level section of a rendered config, or an empty mapping.

Nested sections only, the way MCPServerConfig and ConfigBuilder read this file: a top-level dotted line in config.yml configures nothing.

Return type:

dict[str, object]

osprey_connectors.ipc.connector_block(config, connector_type)[source]#

The control_system.connector.<type> block, as written.

Return type:

object

osprey_connectors.ipc.declared_gateways(block)[source]#

How a connector block spells its gateways table: the one reader of that fact.

Only the explicit empty list is GATEWAYS_NONE_DECLARED. A block that is not a mapping, or whose table is missing, null, {} or anything but a non-empty mapping, is GATEWAYS_MISSING — a table nobody filled in, never “nothing to probe”. A non-empty mapping is GATEWAYS_ROWS.

Return type:

str

osprey_connectors.ipc.derive_endpoints(config, target, *, writes_enabled=None, readonly_run=None, config_path=None)[source]#

Derive the per-role endpoints and selected role for target.

Parameters:
  • config (object) – The full rendered config mapping (config.yml as loaded).

  • target (str) – The control target, 'live' or 'va'.

  • writes_enabled (bool | None) – Whether writes are armed for target. Defaults to the deployment’s write tier — tier_allows() over the control_system section, which is what the connector’s write verdict reads — never the per-type writes_enabled key on its own; injectable so a caller (or a test) can derive the endpoints of a posture other than the configured one.

  • readonly_run (bool | None) – Whether this is a readonly executor run. Defaults to is_readonly_run().

  • config_path (str | None) – The project config an unset virtual-accelerator gateway port is filled from — the file the child reads, so the port derived here is the port it connects to. None reads CONFIG_FILE, else ./config.yml.

Returns:

The derivation, whose endpoints may be empty when the deployment has no gateways for this target — an eligibility question, not an error, so the derivation still answers “which role would be selected”.

Raises:

ValueError – Propagated from resolve_target() when the target is unknown, or is live on a deployment that has never named its real machine, and from config_flag() when a use_name_server spells neither true nor false. evaluate_eligibility() is where that becomes a reason rather than an exception.

Return type:

TargetDerivation

osprey_connectors.ipc.live_collision(config, derivation, *, config_path=None)[source]#

The live gateway derivation’s selected endpoint lands on, or None.

Only a target whose type is in NEVER_LIVE_TYPES is asked: its reads and writes must not reach an endpoint live derives from the same config, read-only or write-access alike, or they reach the real machine under a soft label. Addresses are compared as written (same_endpoint()), never resolved, so this is the negative half only — whatever the target is, it is not the machine. A deployment with no live machine has nothing to collide with.

Parameters:
Return type:

LiveCollision | None

osprey_connectors.ipc.subsection(section, name)[source]#

One nested section of section, or an empty mapping when it is absent or not one.

Return type:

dict[str, object]

osprey_connectors.ipc.verify_host_report(derivation, report, *, readonly_run=None, writes_enabled=None)[source]#

Assert the child came up where the derivation said it would.

Given the posture the derivation was taken under (readonly_run, writes_enabled), the report is first held to it: its connector_type and its transport must be the derived ones, and then a readonly run requires the report’s readonly_run to be True while any other run requires its writes_enabled to equal writes_enabled. With neither given, only the endpoint check below runs.

verify_child_report() answers this for every target whose config names a gateway. A deployment can also select a connector that talks to no gateway at all — the simulator served in process is one, and it is the fail-safe default — and for that one the derivation has no endpoint and the child reports none. Nothing is verified there because there is no endpoint to get wrong, but the symmetry is: a child that configured Channel Access where the config derived nothing has inherited an environment from somewhere, and that is a mismatch as serious as any other.

The unverified branch is entered only when the derivation has no endpoint rows at all. A target with rows whose selected role is missing — a write-only gateway table on a deployment that selects read_only, say — is a gateway deployment with a hole in it, not a gatewayless one: its child connects to a real control system over whatever default broadcast address it finds, and that is precisely the unpinned-CA case verification exists to catch. It goes to verify_child_report(), which refuses it either for reporting no gateway or for having no derived endpoint to compare against.

Parameters:
  • derivation (TargetDerivation) – The derivation for the target, from derive_endpoints().

  • report (Mapping[str, object]) – The child’s post-connect report.

  • readonly_run (bool | None) – Whether the derivation was taken for a readonly run.

  • writes_enabled (bool | None) – Whether the derivation was taken with writes armed.

Returns:

A passing Verification, or a failing one naming the field (connector_type, transport, readonly_run, writes_enabled or an endpoint field), the expected value and the value the child reported.

Return type:

Verification

exception osprey_connectors.ipc.FrameDecodeError[source]#

Bases: ValueError

A frame could not be parsed: bad magic, bad version, or unknown tag.

exception osprey_connectors.ipc.FrameEncodeError[source]#

Bases: TypeError

A value was handed to the codec that the wire format cannot carry.

Starting and putting down the child processes a supervisor owns.

Every child a supervisor starts is put down the same way, and this module is where that way is spelled once: terminate() puts down one child. Two invariants hold throughout:

  1. Nothing signals a child before the event loop has had the chance to reap its exit.

  2. Every exit code reported comes from a reap.

It imports only the standard library and knows nothing about the control system its callers talk to.

osprey_connectors.process.DEFAULT_TERMINATE_GRACE_S = 2.0#

How long a child gets between SIGTERM and SIGKILL.

async osprey_connectors.process.reap_exit_code(process, window_s)[source]#

Wait up to window_s for the event loop to collect process’s exit.

Returns the exit code, or None if the child is still running when the window closes. Never signals the child and never raises.

Return type:

int | None

async osprey_connectors.process.terminate(process, grace_s)[source]#

SIGTERM, then SIGKILL after the grace period. Never raises.

A child that already exited is collected by the event loop before anything signals it: a signal sent to an exited, uncollected child reaps it out from under the loop’s watcher, which then loses the child’s own exit code and reports 255. The wait for that collection is bounded by min(grace_s, _REAP_WINDOW_S), so it only delays putting down a child that is in fact still running.

Returns:

The reaped exit code, or None if the child outlived SIGKILL’s grace period.

Return type:

int | None

What a supervisor does about a connector call that failed: one policy, one table.

A supervisor owns a connector’s lifetime: the connector-host pool owns its children, a server owns its connectors. A supervisor that calls the policy — the connector-host pool does — hands the error and the supervised connector to RecoveryPolicy.recover() when a call fails, and the policy answers with what it did, as a Recovery record, or None when the failure restarts nothing.

The decision is one table, failure class and origin in, one step out:

A failure under one of the three contract roots of osprey_connectors.errors (a channel failure, a lost connector, a connector bug) is decided by its root alone; the builtin rows decide only a failure under no root. So no failure reaches two steps, and no decision depends on the order of the rows.

Probe pings the incarnation: one that answers is kept (child_kept), one that does not is replaced. Replace retires the incarnation under Timing.LAZY (the next call starts a fresh one: child_replaced for a child, connector_dropped for an in-process connector) and respawns it under Timing.EAGER (child_replaced, or respawn_refused when no new child could be brought up).

Invariants:

  • Callers hand over the error; the policy decides where it came from. An error the connector-host child sent is _Origin.CHILD, one the link to the child made (a missed deadline, a closed pipe) is _Origin.LINK, and anything from a connector in this process is _Origin.IN_PROCESS.

  • Recovery happens at most once per incarnation. Every failure of one incarnation awaits the one act the first started, and an incarnation that has been replaced or dropped is answered with that record and never acted on again.

  • One attempt per failure episode. A refused respawn answers every call that began before the refusal; a call that begins later may try once more. A kept child is no episode’s end: its next missed deadline pings it again.

  • The policy never retries a call, owns no deadline, and holds no incarnation alive.

class osprey_connectors.recovery.Recovery(action, text)[source]#

Bases: object

One thing a supervisor did about a failure, in one sentence.

action: RecoveryAction#
text: str#
as_dict()[source]#

The record as a failure’s recovery carries it.

Return type:

RecoveryRecord

__init__(action, text)#
class osprey_connectors.recovery.RecoveryAction(value)[source]#

Bases: StrEnum

What a supervisor did about a failure.

CONNECTOR_DROPPED = 'connector_dropped'#
CHILD_REPLACED = 'child_replaced'#
CHILD_KEPT = 'child_kept'#
RESPAWN_REFUSED = 'respawn_refused'#
class osprey_connectors.recovery.RecoveryPolicy(*, timing, replace_on_write_timeout=False)[source]#

Bases: object

One supervisor’s recovery: the table above, at one timing, with one memo.

Parameters:
  • timing (Timing) – When a replaced incarnation’s successor is brought up. An in-process connector always recovers lazily.

  • replace_on_write_timeout (bool) – Replace the child of a write that missed its deadline without pinging it, so nothing left in it can send that write after a newer one.

__init__(*, timing, replace_on_write_timeout=False)[source]#
begin()[source]#

A ticket for a call that is starting; later calls get larger tickets.

Return type:

int

async recover(exc, target, *, ticket, write=False)[source]#

Act on target for exc, at most once per incarnation; None when nothing is done.

ticket is the one begin() gave the failed call when it started; write says the call could move the machine.

Raises:

ValueError – An eager policy was handed a target that is not in a child, or that cannot respawn.

Return type:

Recovery | None

class osprey_connectors.recovery.Supervised(*args, **kwargs)[source]#

Bases: Protocol

A connector as the policy acts on it.

incarnation is the one running instance the failure belongs to: the connector-host proxy of a child, or the in-process connector itself. It is hashable and weakly referable, and a successor is a different object. respawn is the eager-only member, required only under Timing.EAGER and never called under Timing.LAZY, so a lazy supervisor’s target leaves it out: async respawn() -> bool brings up a successor in the incarnation’s place, retires the incarnation once the successor serves, and answers whether it did.

property incarnation: object#
property in_child: bool#
async ping()[source]#
Return type:

bool

async retire()[source]#
__init__(*args, **kwargs)#
class osprey_connectors.recovery.Timing(value)[source]#

Bases: StrEnum

When a replaced incarnation’s successor is brought up.

LAZY = 'lazy'#

By the next call.

EAGER = 'eager'#

At once, by the recovery itself.