Supervising Connector Processes#
What a supervisor needs to run connectors in child processes: the connector host’s launch, proxy and pool, starting and stopping children, and the policy for a call that failed.
Inter-process plumbing for the connector-host child.
The child process owns the control-system client library; the parent talks to
it over a pipe pair. This namespace holds what a supervisor needs: launching
and stopping a child, the proxy that drives one and the pool that keeps them,
the endpoint derivation and verification of what a child reports, and the two
errors the wire format raises when a frame cannot be written or read. The wire
codec itself and the typed-exception registry that carries a failure across
the boundary are internal to this package (_frames, _exceptions).
The frame errors are imported here. Every other name is resolved on access
(PEP 562): the child runs as python -m osprey_connectors.ipc._host, and a
package that imported _host eagerly would load it a second time under
__main__.
- osprey_connectors.ipc.scrub_epics_env(environ=None)[source]#
Remove every inherited
EPICS_CA_*/EPICS_PVA_*variable.- Parameters:
environ (dict[str, str] | None) – Mapping to scrub, defaulting to
os.environ.- Returns:
an endpoint this child was not configured with must not be reachable from it, and a variable that survived the scrub would silently widen what
connect()decided.- Return type:
The removed name/value pairs, for logging. Nothing is put back
- class osprey_connectors.ipc.AttributedReader(stream)[source]#
Bases:
objectA child’s stdout, able to say why reading it stopped.
The proxy fails every outstanding request with whatever ended its read stream. Left to itself that is an anonymous end-of-pipe, which tells an operator nothing about the decision that caused it — so the supervisor names the reason here before it kills the child, and the proxy’s
ConnectionErrorcarries it.
- osprey_connectors.ipc.host_env()[source]#
The environment a child is launched with:
os.environ, minus EPICS.The child scrubs
EPICS_CA_*/EPICS_PVA_*again on its own first line, and that is the scrub the design depends on. This one is the defense-in-depth half: an ambient gateway never reaches the process that could act on it, so no window exists between exec and scrub.The child is stamped with
ENV_CHAIN_APPLIED_ENV. Reading its config file would otherwise load the project.envfrom its working directory after the scrub, and anEPICS_*line there would put back what both scrubs took out.START_MARKS_FD_ENVis dropped too: a supervisor that is itself running under one never passes on a descriptor number that names something else in its child.- Return type:
dict[str, str]
- async osprey_connectors.ipc.kill_host(process, proxy, reader, *, reason, grace_s)[source]#
Put down a child that failed, and let its proxy settle. Never raises.
- Parameters:
process (asyncio.subprocess.Process) – The child’s
asyncio.subprocess.Process.proxy (ConnectorHostProxy) – The
ConnectorHostProxyover its pipes.reader (AttributedReader) – The proxy’s
AttributedReader.reason (str | None) – Why the child is put down, named on the reader before the kill so every request in flight fails with it.
Nonewhen the caller has already retired the reader.grace_s (float) – How long the child gets to exit after SIGTERM before SIGKILL.
- async osprey_connectors.ipc.spawn_host(python, env, *, start_marks_fd=None)[source]#
Launch one connector-host child with its frame pipes attached.
- Parameters:
python (str) – The interpreter to run the child under.
env (Mapping[str, str]) – The child’s environment, normally
host_env().start_marks_fd (int | None) – The write end of a pipe the child writes its start marks to (see
osprey_connectors.ipc._host), orNonefor a child that writes none. The child inherits the descriptor under the same number; the caller closes its own copy after the spawn.
- Returns:
The
asyncio.subprocess.Process. stderr is inherited, so the child’s diagnostics land wherever the supervisor’s own do.- Raises:
OSError – The interpreter could not be executed. Callers wrap this in whatever their own failure type is.
- Return type:
Process
- async osprey_connectors.ipc.stop_host(process, proxy, *, grace_s)[source]#
Stop a healthy child in order: acknowledge, then make sure it is gone. Never raises.
- Parameters:
process (asyncio.subprocess.Process) – The child’s
asyncio.subprocess.Process.proxy (ConnectorHostProxy) – The
ConnectorHostProxyover its pipes.grace_s (float) – How long the child gets to exit after SIGTERM before SIGKILL.
- exception osprey_connectors.ipc.ConnectorHostError(message, *, target, execution_mode, pid, recovery=())[source]#
Bases:
ConnectorUnavailableErrorA connector-host child could not serve a call (see the module docstring).
- Variables:
target – The control target the child serves.
execution_mode –
"readonly"orNone.pid – The child’s process id, or
Nonewhen none was started.
- exception osprey_connectors.ipc.ConnectorHostLostError(message, *, target, execution_mode, pid, cause, returncode, recovery=())[source]#
Bases:
ConnectorHostErrorThe child went away with this call in flight; the call was not retried.
- Variables:
cause –
"exited"(the child died),"stopped"(the pool was closed or the key disconnected),"unresponsive"(seeConnectorHostUnresponsiveError),"write_timeout"(a write on the child missed its deadline and the pool, built withkill_on_write_timeout, killed it; seeConnectorHostWriteTimeoutError) or"unavailable"(the child’s connector reported itself unavailable and the pool discarded the child).returncode – The child’s exit status once reaped, when known.
- class osprey_connectors.ipc.ConnectorHostPool(control_system, *, config_file=None, python=sys.executable, start_timeout_s=DEFAULT_START_TIMEOUT_S, call_deadline_s=DEFAULT_CALL_DEADLINE_S, timeout_grace_s=1.0, ping_timeout_s=DEFAULT_PING_TIMEOUT_S, terminate_grace_s=DEFAULT_TERMINATE_GRACE_S, kill_on_write_timeout=False)[source]#
Bases:
objectOne connector-host child per key, on demand.
A key is
(target, execution_mode), or(target, execution_mode, owner, authorization)when the caller names either (see the module docstring).- Parameters:
control_system (Mapping[str, object]) – The
control_systemconfig section, as loaded. Copied;${VAR}placeholders in it are resolved at each spawn.config_file (str | PathLike[str] | None) – The project config the children read their write posture and limits from. Without one, a child finds whatever
CONFIG_FILE(or./config.yml) this process’s environment leads it to, and with none its writes stay off.python (str) – The interpreter children run under.
start_timeout_s (float) – Bound on a child starting and answering its init frame.
call_deadline_s (float) – Local deadline for a call that names no
timeout; a child silent that long is pinged.timeout_grace_s (float) – How long past a call’s own
timeoutthe child may take to answer before it is pinged. The child applies the timeout to the control-system call and reports its ownTimeoutErrorfirst, which is the better error.ping_timeout_s (float) – How long a child that missed a deadline gets to answer a ping before it is judged wedged and killed.
terminate_grace_s (float) – Time between
SIGTERMandSIGKILL.kill_on_write_timeout (bool) – Kill a child whose write misses its deadline, even one that still answers a ping, and fail the write with
ConnectorHostWriteTimeoutError. Off, a timed-out write’s child is kept when it answers, and the write may still land after a newer one (see the module docstring). Reads are never affected.
- __init__(control_system, *, config_file=None, python=sys.executable, start_timeout_s=DEFAULT_START_TIMEOUT_S, call_deadline_s=DEFAULT_CALL_DEADLINE_S, timeout_grace_s=1.0, ping_timeout_s=DEFAULT_PING_TIMEOUT_S, terminate_grace_s=DEFAULT_TERMINATE_GRACE_S, kill_on_write_timeout=False)[source]#
- async connector(target, *, execution_mode=None, owner=None, authorization=None)[source]#
The connector for target, bringing its child up if there is none.
- Parameters:
target (str) –
"live","va"or"standin", resolved perresolve_target().execution_mode (str | None) –
None, or"readonly"for a child that refuses every write whatever the config arms.owner (str | None) – The login whose switch the child’s writes follow, stamped as
OSPREY_CONTROL_OWNERin its environment;Noneleaves the owner to the ladder the child inherits.authorization (str | None) – The id of the program’s write authorization the child’s writes are judged by, set as
OSPREY_WRITE_AUTHORIZATIONin its environment;Noneis a child that names none.
- Raises:
ValueError – target does not resolve on this deployment, execution_mode is not one of the two accepted values, or owner or authorization is not a non-blank name.
ConnectorHostStartError – The child could not be brought up.
Exception – The child’s own typed error from
connect().
- Return type:
- exception osprey_connectors.ipc.ConnectorHostStartError(message, *, target, execution_mode, pid, stage, phases=())[source]#
Bases:
ConnectorHostErrorA child could not be brought up. Nothing was left running.
- Variables:
stage –
"config"(an unresolved placeholder, a Channel Access type with no gateway to pin it to, or a simulated or stand-in target that would dial an endpointlivederives),"spawn"(the interpreter could not be started),"init"(no answer within the start timeout, an exit before answering, or an unusable answer) or"verify"(the report did not match the derivation).phases – The start marks the child had written when an
init-stage failure was raised, as(name, seconds after the spawn)in the order ofSTART_PHASES; empty for every other stage, and for a child that wrote none.
- exception osprey_connectors.ipc.ConnectorHostUnresponsiveError(message, *, target, execution_mode, pid, cause, returncode, recovery=())[source]#
Bases:
ConnectorHostLostError,TimeoutErrorThe child missed a call’s deadline and then a ping, and was killed.
- exception osprey_connectors.ipc.ConnectorHostWriteTimeoutError(message, *, target, execution_mode, pid, cause, returncode, recovery=())[source]#
Bases:
ConnectorHostLostError,TimeoutErrorA write missed its deadline and its child was killed for it.
Raised only by a pool built with
kill_on_write_timeout=True, only to the write whose deadline passed, and whether or not the child would have answered a ping (cause="write_timeout"). ATimeoutErrorlikeConnectorHostUnresponsiveError, because that is what the write would have raised in-process: it may or may not have landed. What the kill buys is that it cannot be sent later — nothing is left in a child that could put it on the wire after a newer write to the same target.
- class osprey_connectors.ipc.PooledConnector(pool, key)[source]#
Bases:
objectA target’s connector, as the pool serves it.
Bound to a pool key, not to a process: every call goes to the key’s current child, starting a fresh one if the last one was lost. The call surface mirrors
ControlSystemConnectorfor the methods it offers; writes are guarded and limit-checked in the child, exactly as they would be in-process.- property target: str#
- property execution_mode: str | None#
- property owner: str | None#
The owner the key stamps on its child, or
Nonefor the inherited ladder.
- property authorization: str | None#
The write authorization the key’s child is judged by, or
None.
- property pid: int | None#
The current child’s pid, or
Nonewhen none is running.
- property report: dict[str, object]#
The current child’s post-connect report, empty when none is running.
- async read_multiple_channels(channel_addresses, timeout=None)[source]#
- Return type:
dict[str, ChannelValue]
- async write_multiple_channels(operations, timeout=None, confirm=None)[source]#
- Return type:
list[ChannelWriteResult]
- async write_channel_checked(channel_address, value, **kwargs)[source]#
Write, and raise unless the write came back confirmed or unrequested.
ChannelWriteBlockedErrorfor a refusal — writes disabled, a readonly child, limits — andChannelWriteFailedErrorfor a write that was sent but not confirmed.confirmandtimeoutpass through towrite_channel().- Return type:
- exception osprey_connectors.ipc.ChildUnresponsiveError[source]#
Bases:
TimeoutErrorThe child did not answer a request before the proxy’s own deadline.
Distinct from a
TimeoutErrorthe child reported — that one means the control-system call timed out in a child that is still answering. This one means the child itself said nothing in time, so it may be wedged.
- class osprey_connectors.ipc.ConnectorHostProxy(reader, writer, *, timeout_grace_s=1.0, deadline_s=None)[source]#
Bases:
objectA connector-shaped handle on the connector running in the child process.
- Parameters:
reader (Any) – Anything with
await read(n) -> bytesreturningb""at EOF — anasyncio.StreamReader, and therefore anasyncio.subprocess.Process.stdout.writer (Any) – Anything with
write(bytes),await drain()andclose()— anasyncio.StreamWriter, and therefore anasyncio.subprocess.Process.stdin.wait_closed()is used when present.timeout_grace_s (float) – Extra time the proxy waits beyond a call’s own
timeoutbefore giving up locally. The child applies the timeout to the control-system call itself and reports its ownTimeoutError, which is the more informative one; the local deadline exists only so a wedged child cannot hang a caller forever. Set it to0to make the local deadline the only one.deadline_s (float | None) – Local deadline for a call whose
timeoutisNone. Never sent to the child — the connector keeps its own default — andNone(the default) means such a call waits as long as the child takes.
The transport is duck-typed rather than a subprocess handle so tests can drive the proxy over a socketpair, and so the supervisor can hand it a process’s pipes without this class knowing anything about process lifetime.
from_process()is the convenience for the latter.- classmethod from_process(process, *, timeout_grace_s=1.0, deadline_s=None)[source]#
Wrap an
asyncio.subprocess.Process’s stdin/stdout pipes.The proxy does not own the process: it neither starts it nor reaps it. Killing and restarting the child belongs to the supervisor.
- Return type:
- property dead_reason: str | None#
Why the proxy is dead, or
Nonewhile the child is still answering.Set once the pipe to the child is gone for good — end of stream, an unreadable stream, a failed write, or
disconnect()— and never cleared. A refusal fromrefuse_new_requests()is not death.
- async read_channel(channel_address, timeout=None)[source]#
Read one channel. Returns the
ChannelValuethe child produced.- Return type:
- async read_multiple_channels(channel_addresses, timeout=None)[source]#
Read many channels in one request frame.
The batch crosses the boundary whole rather than as N round trips: the child’s connector decides how to fan it out, exactly as it would in-process, and one pipe round trip covers the lot.
- Return type:
dict[str, ChannelValue]
- async write_channel(channel_address, value, timeout=None, confirm=None)[source]#
Write one channel. Returns the
ChannelWriteResultfrom the child.A refusal (limits, writes-disabled) arrives as a typed exception frame and is re-raised here as the real exception class with its fields intact, so a caller cannot tell the refusal came from another process.
confirm=Noneleaves the keyword off the request frame entirely, so the child’s connector resolves the policy for this channel; an explicitconfirm=Falsecrosses the wire as the answer it is.- Return type:
- async write_multiple_channels(operations, timeout=None, confirm=None)[source]#
Write many channels in one request frame.
The whole batch goes over as a single request so the child’s connector keeps whatever transactional behaviour it implements — a simulator that suppresses recalculation between writes, for instance, cannot do that if the parent has already split the batch into separate calls.
- Return type:
list[ChannelWriteResult]
- async write_channel_checked(channel_address, value, **kwargs)#
The base class’s own function, not a copy: see the module docstring.
- Return type:
- async validate_channel(channel_address)[source]#
Whether the channel exists and is reachable, as the child’s connector sees it.
- Return type:
bool
- async disconnect(*, ack_timeout=2.0)[source]#
Ask the child to release its connector, then close the pipe.
Idempotent and never raises: it is called from teardown paths and from the switch’s failure branch, where a second exception would only bury the first. The acknowledgement is best-effort — a child that has already died, or that never answers, still leaves this method returning with the pipe closed and every outstanding call failed.
- async supervisor_request(method, kwargs, timeout)[source]#
Send one supervisor request (
init,ping) and await its reply.Not part of the connector surface: a supervisor that brings a child up over this proxy’s pipes, or asks whether it is still alive, uses this, so the reply is demultiplexed and its error decoded by the same plumbing every other call uses. A typed error frame is raised as its real class (and
raised_by_child()says so); a child that dies before answering fails the call with aConnectionErrorthis proxy made; no answer withintimeoutplus the grace raisesChildUnresponsiveError.- Return type:
object
- refuse_new_requests(reason)[source]#
Stop accepting new calls; let the in-flight ones finish.
reasonis what a caller sees in theConnectionErrorraised by any later call, so it should say what is happening (“target switch in progress”), not merely that something is closed. Calling this twice keeps the first reason — the first one explains the decision.
- async drain(timeout)[source]#
Wait for outstanding requests to finish.
Returns
Truewhen nothing is left in flight,Falsewhen the deadline expired with calls still pending. It never raises and never cancels anything: a caller that getsFalsedecides for itself whether to wait longer or to abandon the child.- Return type:
bool
- osprey_connectors.ipc.raised_by_child(exc)[source]#
Whether exc is an error the child sent, rather than one this proxy made.
A
ConnectionErrorcan come from two places: the connector in a child that answered says it lost its own transport, or the child itself is gone. The first arrives in an error frame and is marked here as it is decoded; the second is manufactured by this proxy when the pipe ends. A supervisor classifies by this mark, never by whatever state the proxy is in when the caller happens to resume. A failure about one channel is noConnectionErrorat all.- Return type:
bool
- class osprey_connectors.ipc.Endpoint(host, port, mode)[source]#
Bases:
objectWhere one gateway role points, and how the child will say so to EPICS.
- host: str#
- port: object#
The
portthe gateway block carries, as written (or the default the derivation filled in): usually anint, but whatever the config holds.
- mode: str#
- __init__(host, port, mode)#
- class osprey_connectors.ipc.LiveCollision(selected, live, role, connector_type)[source]#
Bases:
objectA live gateway that a never-live target’s selected endpoint lands on.
selectedandlivename the same endpoint, each as its own block spells it;roleandconnector_typelocatelivein the config.- role: str#
- connector_type: str#
- __init__(selected, live, role, connector_type)#
- class osprey_connectors.ipc.TargetDerivation(target, connector_type, endpoints, selected_role, transport, gateways_declared='missing')[source]#
Bases:
objectWhat a connector-host child will do for a target, derived from config.
endpointscarries one row per configured role:read_onlyandwrite_accesswhen the gateways table has them, pluspvawhen the block configures PVA routing and a PVA gateway — the same conjunctionconnect()requires before it touches a PVA environment variable.selected_roleis the row the child will actually configure the process with: EPICS keeps one process-wide context, so exactly one gateway is used.transportis the wire the child’s connector will speak (connector_transport()), so a child that built the other venue of the simulator is refused by what it echoes.gateways_declaredisdeclared_gateways()of the block: only an explicit emptygateways: []isGATEWAYS_NONE_DECLARED, the facility saying this target has nothing to probe. A missing table is not that statement — a forgotten table must never read as a direct connection — so it isGATEWAYS_MISSING. The simulator served in process reads no block, and keeps the default.- target: str#
- connector_type: str#
- selected_role: str#
- transport: str | None#
- gateways_declared: str = 'missing'#
- property declares_no_endpoints: bool#
Whether the block spells
gateways: [], and so has nothing to probe.
- selected_endpoint()[source]#
The row the child will configure, or
Nonewhen config has none.- Return type:
Endpoint | None
- __init__(target, connector_type, endpoints, selected_role, transport, gateways_declared='missing')#
- class osprey_connectors.ipc.Verification(ok, field=None, expected=None, got=None, detail='')[source]#
Bases:
objectWhether the child came up where the derivation said it would.
- ok: bool#
- field: str | None = None#
- expected: object = None#
- got: object = None#
- detail: str = ''#
- __init__(ok, field=None, expected=None, got=None, detail='')#
- osprey_connectors.ipc.config_section(config, name)[source]#
One top-level section of a rendered config, or an empty mapping.
Nested sections only, the way
MCPServerConfigandConfigBuilderread this file: a top-level dotted line inconfig.ymlconfigures nothing.- Return type:
dict[str, object]
- osprey_connectors.ipc.connector_block(config, connector_type)[source]#
The
control_system.connector.<type>block, as written.- Return type:
object
- osprey_connectors.ipc.declared_gateways(block)[source]#
How a connector block spells its
gatewaystable: the one reader of that fact.Only the explicit empty list is
GATEWAYS_NONE_DECLARED. A block that is not a mapping, or whose table is missing,null,{}or anything but a non-empty mapping, isGATEWAYS_MISSING— a table nobody filled in, never “nothing to probe”. A non-empty mapping isGATEWAYS_ROWS.- Return type:
str
- osprey_connectors.ipc.derive_endpoints(config, target, *, writes_enabled=None, readonly_run=None, config_path=None)[source]#
Derive the per-role endpoints and selected role for target.
- Parameters:
config (object) – The full rendered config mapping (
config.ymlas loaded).target (str) – The control target,
'live'or'va'.writes_enabled (bool | None) – Whether writes are armed for target. Defaults to the deployment’s write tier —
tier_allows()over thecontrol_systemsection, which is what the connector’s write verdict reads — never the per-typewrites_enabledkey on its own; injectable so a caller (or a test) can derive the endpoints of a posture other than the configured one.readonly_run (bool | None) – Whether this is a readonly executor run. Defaults to
is_readonly_run().config_path (str | None) – The project config an unset virtual-accelerator gateway port is filled from — the file the child reads, so the port derived here is the port it connects to.
NonereadsCONFIG_FILE, else./config.yml.
- Returns:
The derivation, whose
endpointsmay be empty when the deployment has no gateways for this target — an eligibility question, not an error, so the derivation still answers “which role would be selected”.- Raises:
ValueError – Propagated from
resolve_target()when the target is unknown, or isliveon a deployment that has never named its real machine, and fromconfig_flag()when ause_name_serverspells neither true nor false.evaluate_eligibility()is where that becomes a reason rather than an exception.- Return type:
- osprey_connectors.ipc.live_collision(config, derivation, *, config_path=None)[source]#
The live gateway derivation’s selected endpoint lands on, or
None.Only a target whose type is in
NEVER_LIVE_TYPESis asked: its reads and writes must not reach an endpointlivederives from the same config, read-only or write-access alike, or they reach the real machine under a soft label. Addresses are compared as written (same_endpoint()), never resolved, so this is the negative half only — whatever the target is, it is not the machine. A deployment with no live machine has nothing to collide with.- Parameters:
config (object) – The rendered config mapping derivation was derived from.
derivation (TargetDerivation) – The target being judged, from
derive_endpoints().config_path (str | None) – See
derive_endpoints(); pass the one derivation used.
- Return type:
LiveCollision | None
- osprey_connectors.ipc.subsection(section, name)[source]#
One nested section of section, or an empty mapping when it is absent or not one.
- Return type:
dict[str, object]
- osprey_connectors.ipc.verify_host_report(derivation, report, *, readonly_run=None, writes_enabled=None)[source]#
Assert the child came up where the derivation said it would.
Given the posture the derivation was taken under (readonly_run, writes_enabled), the report is first held to it: its
connector_typeand itstransportmust be the derived ones, and then a readonly run requires the report’sreadonly_runto beTruewhile any other run requires itswrites_enabledto equal writes_enabled. With neither given, only the endpoint check below runs.verify_child_report()answers this for every target whose config names a gateway. A deployment can also select a connector that talks to no gateway at all — the simulator served in process is one, and it is the fail-safe default — and for that one the derivation has no endpoint and the child reports none. Nothing is verified there because there is no endpoint to get wrong, but the symmetry is: a child that configured Channel Access where the config derived nothing has inherited an environment from somewhere, and that is a mismatch as serious as any other.The unverified branch is entered only when the derivation has no endpoint rows at all. A target with rows whose selected role is missing — a write-only gateway table on a deployment that selects
read_only, say — is a gateway deployment with a hole in it, not a gatewayless one: its child connects to a real control system over whatever default broadcast address it finds, and that is precisely the unpinned-CA case verification exists to catch. It goes toverify_child_report(), which refuses it either for reporting no gateway or for having no derived endpoint to compare against.- Parameters:
derivation (TargetDerivation) – The derivation for the target, from
derive_endpoints().report (Mapping[str, object]) – The child’s post-connect report.
readonly_run (bool | None) – Whether the derivation was taken for a readonly run.
writes_enabled (bool | None) – Whether the derivation was taken with writes armed.
- Returns:
A passing
Verification, or a failing one naming the field (connector_type,transport,readonly_run,writes_enabledor an endpoint field), the expected value and the value the child reported.- Return type:
- exception osprey_connectors.ipc.FrameDecodeError[source]#
Bases:
ValueErrorA frame could not be parsed: bad magic, bad version, or unknown tag.
- exception osprey_connectors.ipc.FrameEncodeError[source]#
Bases:
TypeErrorA value was handed to the codec that the wire format cannot carry.
Starting and putting down the child processes a supervisor owns.
Every child a supervisor starts is put down the same way, and this module is
where that way is spelled once: terminate() puts down one child. Two
invariants hold throughout:
Nothing signals a child before the event loop has had the chance to reap its exit.
Every exit code reported comes from a reap.
It imports only the standard library and knows nothing about the control system its callers talk to.
- osprey_connectors.process.DEFAULT_TERMINATE_GRACE_S = 2.0#
How long a child gets between
SIGTERMandSIGKILL.
- async osprey_connectors.process.reap_exit_code(process, window_s)[source]#
Wait up to window_s for the event loop to collect process’s exit.
Returns the exit code, or
Noneif the child is still running when the window closes. Never signals the child and never raises.- Return type:
int | None
- async osprey_connectors.process.terminate(process, grace_s)[source]#
SIGTERM, thenSIGKILLafter the grace period. Never raises.A child that already exited is collected by the event loop before anything signals it: a signal sent to an exited, uncollected child reaps it out from under the loop’s watcher, which then loses the child’s own exit code and reports 255. The wait for that collection is bounded by
min(grace_s, _REAP_WINDOW_S), so it only delays putting down a child that is in fact still running.- Returns:
The reaped exit code, or
Noneif the child outlivedSIGKILL’s grace period.- Return type:
int | None
What a supervisor does about a connector call that failed: one policy, one table.
A supervisor owns a connector’s lifetime: the connector-host pool owns its
children, a server owns its connectors. A supervisor that calls the policy —
the connector-host pool does — hands the error and the supervised connector to
RecoveryPolicy.recover() when a call fails, and the policy answers with
what it did, as a Recovery record, or None when the failure
restarts nothing.
The decision is one table, failure class and origin in, one step out:
a connector bug (
ConnectorInternalError), a failure about one channel (ChannelError), a batch read that failed (ChannelReadFailedError) and a refused client library (ClientLibraryRefusedError), from anywhere: none;a
TimeoutErroron the link: replace for a write when writes replace, probe otherwise;a lost connector (
ConnectorUnavailableError) and a bareConnectionError, from anywhere: replace;anything else, a
TimeoutErrorfrom the child or from this process included: none.
A failure under one of the three contract roots of
osprey_connectors.errors (a channel failure, a lost connector, a
connector bug) is decided by its root alone; the builtin rows decide only a
failure under no root. So no failure reaches two steps, and no decision depends
on the order of the rows.
Probe pings the incarnation: one that answers is kept (child_kept), one
that does not is replaced. Replace retires the incarnation under
Timing.LAZY (the next call starts a fresh one: child_replaced for a
child, connector_dropped for an in-process connector) and respawns it under
Timing.EAGER (child_replaced, or respawn_refused when no new
child could be brought up).
Invariants:
Callers hand over the error; the policy decides where it came from. An error the connector-host child sent is
_Origin.CHILD, one the link to the child made (a missed deadline, a closed pipe) is_Origin.LINK, and anything from a connector in this process is_Origin.IN_PROCESS.Recovery happens at most once per incarnation. Every failure of one incarnation awaits the one act the first started, and an incarnation that has been replaced or dropped is answered with that record and never acted on again.
One attempt per failure episode. A refused respawn answers every call that began before the refusal; a call that begins later may try once more. A kept child is no episode’s end: its next missed deadline pings it again.
The policy never retries a call, owns no deadline, and holds no incarnation alive.
- class osprey_connectors.recovery.Recovery(action, text)[source]#
Bases:
objectOne thing a supervisor did about a failure, in one sentence.
- action: RecoveryAction#
- text: str#
- __init__(action, text)#
- class osprey_connectors.recovery.RecoveryAction(value)[source]#
Bases:
StrEnumWhat a supervisor did about a failure.
- CONNECTOR_DROPPED = 'connector_dropped'#
- CHILD_REPLACED = 'child_replaced'#
- CHILD_KEPT = 'child_kept'#
- RESPAWN_REFUSED = 'respawn_refused'#
- class osprey_connectors.recovery.RecoveryPolicy(*, timing, replace_on_write_timeout=False)[source]#
Bases:
objectOne supervisor’s recovery: the table above, at one timing, with one memo.
- Parameters:
timing (Timing) – When a replaced incarnation’s successor is brought up. An in-process connector always recovers lazily.
replace_on_write_timeout (bool) – Replace the child of a write that missed its deadline without pinging it, so nothing left in it can send that write after a newer one.
- begin()[source]#
A ticket for a call that is starting; later calls get larger tickets.
- Return type:
int
- async recover(exc, target, *, ticket, write=False)[source]#
Act on target for exc, at most once per incarnation;
Nonewhen nothing is done.ticket is the one
begin()gave the failed call when it started; write says the call could move the machine.- Raises:
ValueError – An eager policy was handed a target that is not in a child, or that cannot respawn.
- Return type:
Recovery | None
- class osprey_connectors.recovery.Supervised(*args, **kwargs)[source]#
Bases:
ProtocolA connector as the policy acts on it.
incarnationis the one running instance the failure belongs to: the connector-host proxy of a child, or the in-process connector itself. It is hashable and weakly referable, and a successor is a different object.respawnis the eager-only member, required only underTiming.EAGERand never called underTiming.LAZY, so a lazy supervisor’s target leaves it out:async respawn() -> boolbrings up a successor in the incarnation’s place, retires the incarnation once the successor serves, and answers whether it did.- property incarnation: object#
- property in_child: bool#
- __init__(*args, **kwargs)#