Source code for osprey_connectors.control_context

"""The control-context record — one file per identity.

Which machine a deployment is pointed at, how many times that has moved, and
what one operator has narrowed are properties of the deployment as that
operator holds it, not of any process tree inside it. They live in a single
JSON file, :data:`RECORD_FILENAME` inside the acting identity's directory
under :data:`STATE_DIR_NAME` (the state directory,
:func:`~osprey_connectors.posture_store.state_dir`), and this module is where
it is written and read.

The file has exactly one writer at a time — its **owner**: the web terminal
when one is running, otherwise a controls MCP server. Everything else is a
reader: every chat session, every notebook kernel, every
executor sandbox, every sibling MCP server, and the stdlib-only hooks, which
cannot import this module and therefore restate the rules below. Treat
them as the contract; a change here is a change there.

**1. Where the file is.** :func:`record_path` — the record sits in the same
directory as the posture store, and the root resolves by that store's rule 1
(:func:`osprey_connectors.posture_store.state_dir`) rather than by a
second derivation here: a record one process writes and another looks for
somewhere else is worse than no record at all. An unresolvable root makes
:func:`record_path` ``None``, which reads as "no record" and makes
:func:`write_record` raise — a route surfaces that as ``store_unavailable``
rather than writing where no reader will look.

**2. What a payload must carry.** ``schema`` (exactly
:data:`_SCHEMA_VERSION`), ``target`` (a member of
:data:`~osprey_connectors.types.CONTROL_TARGETS`) and ``generation`` (a
non-negative integer) are the record's identity. If any of the three is
absent, of the wrong type, or unrecognised, there is no record: this is a
file that decides whether a real machine is written to, so a hand-edited or
future-version payload must not be honoured in part.

The other fields are annotations on that identity and degrade on their
own, because each has a meaningful "not stated" value that a reader can act
on safely:

* ``owner`` → ``None``, which is *ownerless* — the state a claim is for. An
  owner that does not name one of :data:`_OWNER_KINDS` and a live PID is not
  an owner anybody should defer to, so reading it as ownerless is what lets
  the next process take the file over instead of stalling behind a ghost.
* ``posture`` → ``{}``. A mirror of ``switch`` kept for old readers, never
  an input: :meth:`ControlContext.to_payload` writes ``sandbox`` for every
  machine in :data:`~osprey_connectors.types.CONTROL_TARGETS` whose position
  is off (:func:`_posture_mirror`), whatever the record it was built
  from carried, and no write decision reads it back. A file written before
  the mirror may still hold an old narrowing; it is parsed by the posture
  store's grammar (rule 2 there) and ignored by the verdict.
* ``last_switch`` → ``None``. It is the terminus of the last switch request
  — applied or refused — and it is carried verbatim. This module owns the
  block's shape, because a requester polls it for its own ``request_id``
  (:func:`terminus` builds it, :func:`write_terminus` stores it and reads it
  back); the switch machinery owns what one says. **A refusal is a record
  write that moves nothing else**: the same target, the same generation, the
  same posture, a new ``last_switch``. That is the whole of how a refusal is
  reported.
* ``switch`` → ``{}``, every machine's writes off. A per-target map of which
  only ``"asked"`` and ``"allowed"`` leaves survive, and a target absent from
  it is off. The legacy ``"on"`` reads as ``"asked"``, and any write stores the
  whole map in the new spelling; any other value is dropped on parse, so a
  degraded switch can only refuse. :func:`switch_position` is the one reader of
  the map.
* ``switch_epoch`` → ``None``, a record written before epochs existed. An
  epoch is a ``time.time_ns()`` stamp that only moves forward; the person who
  widens a switch names the epoch they saw, so a stop or a lost record in
  between makes their gesture stale. ``None`` is never "older" than anything.

The schema stays :data:`_SCHEMA_VERSION` with the two switch fields added:
every pre-existing reader, the stdlib hooks included, ignores keys it does
not know, so a record written by this version reads unchanged through them.

**3. How it is written.** :func:`write_json_atomic` replaces a file rather
than rewriting it, whole through ``safety_file.atomic_replace``, so no reader ever sees half of one and two writers never interleave into one.
Every file in this directory is written through it, the record
(:func:`write_record`) and the per-server reports and requests alike.

The record is written only under its lock: an exclusive ``fcntl.flock`` on
the sibling file ``<record path>.lock``, derived from the path being written
(so ``path=`` moves the lock with the file), waited on for a bounded time and
then refused with ``RuntimeError``. Inside the lock :func:`write_record`
re-reads the file on disk and writes the caller's record with the on-disk
``switch`` and ``switch_epoch`` substituted: whatever a caller read earlier,
no writer but the person's own switch gesture (:func:`set_switch`,
which reads, compares and writes in one hold) moves either field, so a
target switch or a refusal terminus racing a stop cannot turn a machine
back on. A write that finds the record missing or unreadable writes the
switch empty and stamps a fresh epoch, never 0. What a caller changes in
the other fields, and whether it is still entitled to, remains the owner's
business.

The exclusion holds among processes sharing one kernel — one host, or
containers on a Linux host over a bind mount. Across a macOS host and a VM
file share ``flock`` is not guaranteed to exclude anything; that residual is
closed by the sandboxed connector host, not here.

**4. When a change is seen.** Every read re-stats the file and re-parses on
``(st_mtime_ns, st_size, st_ino)``, the same signature the posture store uses
and for the same reason: writes arrive as temp+rename, so two of them inside
one filesystem clock tick differ by inode when mtime and size do not.

**5. Who else lives in that directory.** The record says what the deployment
is pointed at; it does not say what any one controls server has managed to do
about it. That is the per-server **report**, ``server_<pid>.json``, one file
per controls server, written by that server through its publishers and read
here (:func:`read_report`, :func:`live_reports`). A report is an observation,
so it degrades field by field around its one identity, ``server_pid``, and a
report nobody can parse is skipped rather than fatal — the readers of these
files describe a fleet, and one bad file must not blank the others.

**6. Whether anything may act.** The record says where the deployment is
pointed; the reports say who has got there. :func:`converged` is the one place
those two are compared, and it is what the executor's stamp, a notebook cell
and a controls server's write tools ask before they touch a machine: a swap in
flight stops everyone, a server acquiring its first connector stops only its
own session, a server that could not follow stops only its own session, and
nothing else stops anybody.

Every file in that directory is named for the process that owns it, which is
what lets a reader judge one without opening it: :func:`is_process_alive` and
:func:`sweep_dead` are that rule and its consequence. Liveness lives here, at
the bottom, because the record's owner, the reports' authors and the request
files' addressees are all judged by it and there must be one answer.

This module imports nothing from ``osprey`` and holds no config: it runs
inside connector-host children, executor sandboxes and notebook kernels,
where the dependency budget is the lean connector chain. The import direction
is one-way — the posture store below it, never the reverse at module scope.
"""

from __future__ import annotations

import contextlib
import fcntl
import json
import logging
import os
import threading
import time
from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass, field, replace
from datetime import UTC, datetime
from pathlib import Path
from typing import Generic, Literal, TypeVar

from osprey_connectors import posture_store
from osprey_connectors.identity import acting_identity
from osprey_connectors.refusals import refusal_phrase
from osprey_connectors.safety_file import (
    SHARED_FILE_MODE,
    SHARED_LOCK_MODE,
    atomic_replace,
    ensure_shared_dir,
)
from osprey_connectors.types import CONTROL_TARGETS

logger = logging.getLogger("osprey_connectors.control_context")

__all__ = [
    "claimed",
    "REPORT_APPLIED",
    "REPORT_APPLYING",
    "REPORT_FAILED",
    "OWNER_CONTROLS_SERVER",
    "OWNER_WEB_TERMINAL",
    "RECORD_FILENAME",
    "REPORT_FILE_GLOB",
    "REPORT_FILE_PREFIX",
    "REPORT_FILE_SUFFIX",
    "STATE_DIR_NAME",
    "SWITCH_APPLIED",
    "SWITCH_POSITIONS",
    "SWITCH_REFUSED",
    "ControlContext",
    "Owner",
    "ServerReport",
    "StoppedSince",
    "applied_detail",
    "blocking_pids",
    "invalidate_cache",
    "is_process_alive",
    "live_owner",
    "live_reports",
    "write_statement",
    "owned_here",
    "converged",
    "file_signature",
    "parse_record",
    "read_record",
    "read_record_status",
    "read_report",
    "record_path",
    "report_path",
    "report_paths",
    "set_switch",
    "sweep_dead",
    "switch_position",
    "terminus",
    "write_json_atomic",
    "unchanged_detail",
    "write_record",
    "write_terminus",
    "held_epoch",
    "RecordUnavailable",
]

#: The payload version. There is exactly one, and a payload that does not
#: spell it is not this record: hooks and record change together, and a
#: deployment is regenerated rather than migrated.
_SCHEMA_VERSION = 1

#: The record's filename inside the state directory. The stdlib hooks restate
#: this literal.
RECORD_FILENAME = "control_context.json"

#: The tree of per-identity directories the record shares with the posture
#: store, the per-server reports and the request files — one directory per
#: identity, each holding every one of those families. Taken from the store
#: rather than re-spelled so one directory cannot be two directories.
STATE_DIR_NAME = posture_store.STATE_DIR_NAME

#: The two kinds of process that can own the record. A web terminal outranks a
#: controls server — it is the one an operator is looking at.
OWNER_WEB_TERMINAL = "web_terminal"
OWNER_CONTROLS_SERVER = "controls_server"
_OWNER_KINDS = frozenset({OWNER_WEB_TERMINAL, OWNER_CONTROLS_SERVER})

#: The two termini a switch request can reach in the record. ``applied`` moved
#: the target and minted a generation; ``refused`` moved nothing.
SWITCH_APPLIED = "applied"
SWITCH_REFUSED = "refused"

#: How far one controls server has got through the switch the record names.
#: ``applying`` is a swap in flight and stops every session while its bound
#: holds; ``applied`` is arrival, which the binding states as well; ``failed``
#: is a swap that will not land without an operator, and stops only the session
#: whose server it is. Distinct from the record's :data:`SWITCH_APPLIED` /
#: :data:`SWITCH_REFUSED`, which are a request's terminus rather than a
#: server's progress.
REPORT_APPLYING = "applying"
REPORT_APPLIED = "applied"
REPORT_FAILED = "failed"

#: One report file per controls server, in the state directory beside the record.
#: The PID in the name is the address and is what makes a report left behind by
#: a killed server sweepable without opening it.
REPORT_FILE_PREFIX = "server_"
REPORT_FILE_SUFFIX = ".json"
REPORT_FILE_GLOB = f"{REPORT_FILE_PREFIX}*{REPORT_FILE_SUFFIX}"

#: Guards every :class:`_ParsedFiles` cache below.
_CACHE_LOCK = threading.Lock()

#: How many parsed files a cache holds before it is dropped whole. Reports
#: are named for PIDs, so a long-lived reader would otherwise keep an entry per
#: server that ever ran; dropping the lot costs one re-read each and cannot go
#: stale, because every read re-stats before it trusts what it holds.
_CACHE_MAX_ENTRIES = 128

_Parsed = TypeVar("_Parsed")

#: A machine's three positions. ``off`` is never stored: it is a target absent
#: from the map.
_SWITCH_OFF: Literal["off"] = "off"
_SWITCH_ASKED: Literal["asked"] = "asked"
_SWITCH_ALLOWED: Literal["allowed"] = "allowed"

#: The word a record written before the three positions arms a machine with.
#: Read, and taken by :func:`set_switch`, as :data:`_SWITCH_ASKED`; never written. Every other value a payload
#: carries under ``switch`` is dropped on parse (rule 2).
_SWITCH_LEGACY_ON = "on"

#: The record lock's file, beside the record: ``<record path>`` plus this.
#: Rule 3. Not one of the PID-named families, so :func:`sweep_dead` never
#: touches it.
_LOCK_SUFFIX = ".lock"

#: How long a record write waits for the lock before it refuses. A holder
#: does one read and one atomic replace; seconds of contention is a wedged
#: process, and a write path must answer rather than hang behind it.
_LOCK_TIMEOUT_S = 5.0

#: How often a waiting writer retries the lock.
_LOCK_POLL_S = 0.01


# -- the record -------------------------------------------------------------


[docs] @dataclass(frozen=True) class Owner: """The process currently allowed to write the record. Attributes: kind: One of :data:`_OWNER_KINDS`. pid: The owning process's PID — how every other process decides whether the owner is still alive. port: The web terminal's port, so a follower can name where the owner is in a refusal an operator has to act on. ``None`` for a controls server, which serves nothing an operator can open. """ kind: str pid: int port: int | None = None
[docs] def to_payload(self) -> dict[str, object]: """This owner as the record's ``owner`` object.""" return {"kind": self.kind, "pid": self.pid, "port": self.port}
[docs] @dataclass(frozen=True) class ControlContext: """One deployment's control context. Treat an instance as immutable. :func:`read_record` hands back the cached instance, so a caller that needs a changed record builds one with :func:`dataclasses.replace` rather than editing this one — an in-place edit would reach every other reader in the process. Attributes: target: The control target the deployment is pointed at. generation: How many times that has moved. Minted by the owner and by nobody else; a reader pins its work to it. owner: The process allowed to write this file, or ``None`` when the record is ownerless and free to be claimed. posture: The ``posture`` field as read off disk — the mirror of ``switch`` (:func:`_posture_mirror`), or an old narrowing in a file written before it. A reading only: :meth:`to_payload` ignores it and derives the field from ``switch``, so nothing set here is ever stored, and no write decision reads it. last_switch: The terminus of the last switch request, verbatim, or ``None`` when no request has reached one. switch: The position the person has put each machine at, ``{target: "asked" | "allowed"}``; an absent target is off, and :func:`switch_position` is how it is read. :func:`write_record` never changes it — it keeps whatever the file on disk says. switch_epoch: The ``time.time_ns()`` stamp of the last stop or loss of the record, or ``None`` for a record written before epochs existed. Kept by :func:`write_record` like ``switch``. """ target: str generation: int owner: Owner | None = None posture: dict[str, str] = field(default_factory=dict) last_switch: dict[str, object] | None = None switch: Mapping[str, str] = field(default_factory=dict) switch_epoch: int | None = None
[docs] def to_payload(self) -> dict[str, object]: """This record as the JSON object :func:`write_record` stores. The seven contract fields and the schema, and nothing else: a payload round-trips through :func:`parse_record` unchanged, and a key some future version added does not survive a rewrite by this one. The ``posture`` field is the mirror of ``switch`` (:func:`_posture_mirror`), never :attr:`posture`: a narrowing is not something a writer can store. """ return { "schema": _SCHEMA_VERSION, "owner": None if self.owner is None else self.owner.to_payload(), "target": self.target, "generation": self.generation, "posture": _posture_mirror(self.switch), "last_switch": None if self.last_switch is None else dict(self.last_switch), "switch": dict(self.switch), "switch_epoch": self.switch_epoch, }
def _posture_mirror(switch: Mapping[str, str]) -> dict[str, str]: """The ``posture`` field a record with *switch* stores — ``{target: "sandbox"}``. ``sandbox`` for every machine in :data:`~osprey_connectors.types.CONTROL_TARGETS` at off, so an old reader of the field refuses every machine the person has not turned on; ``asked`` and ``allowed`` both store no entry, which is how the old grammar spells "writes". Derived on every write and never read by a write decision: the switch is the state, this is its shadow in the old grammar. Args: switch: The record's switch, ``{target: "asked" | "allowed"}``. """ return { target: posture_store.POSTURE_SANDBOX for target in CONTROL_TARGETS if _switch_word(switch.get(target)) is None }
[docs] def switch_position( record: ControlContext | None, target: str ) -> Literal["off", "asked", "allowed"]: """Where the person has put *target* — the one reader of the ``switch`` map. Never raises. No record, a target absent from the map and an unknown value all read as off; a hand-built record still spelling ``"on"`` reads as asked, as a parsed one does. Args: record: The record, or ``None`` when there is none. target: The machine. """ if record is None: return _SWITCH_OFF return _switch_word(record.switch.get(target)) or _SWITCH_OFF
[docs] def claimed(record: ControlContext, baseline: str) -> ControlContext: """*record* as a process claiming it holds it, given the deployment's *baseline*. A record nobody has switched follows the deployment's baseline; every applied switch mints ``generation + 1``, so generation 0 is a claim and never a switch. An unswitched record on another target moves to *baseline* and keeps its generation, owner and last switch terminus. A switched record, or one already on *baseline*, is returned as it is. """ if record.generation != 0 or record.target == baseline: return record logger.warning( "control context: unswitched record on %r follows the deployment's baseline %r", record.target, baseline, ) return replace(record, target=baseline)
# -- path resolution -------------------------------------------------------- def _record_path_under(root: Path) -> Path: """The record's path under an explicit agent-data *root*. The one place the three path hops are spelled, for a caller that holds a root rather than an environment and means the file of the identity it is itself acting as — a test fixture, and any same-process writer. The identity hop is :func:`~osprey_connectors.identity.acting_identity`, the same rung the state directory ends on, so a caller that holds the root this process is stamped with names the file this process reads. A writer preparing ANOTHER identity's directory is not that caller and gets the wrong path here; it names the identity instead, through :func:`osprey.deployment.compose_generator.control_target_identity_dir`. The container bind deliberately does not apply: a caller passing a root has said which tree it means, and that is a statement about the host layout. """ return root / STATE_DIR_NAME / acting_identity() / RECORD_FILENAME
[docs] def record_path() -> Path | None: """The record's path, or ``None`` when the agent-data root is unresolvable. ``None`` is not an error: for a reader it is indistinguishable from an empty deployment, and for a writer it is what a route reports as ``store_unavailable``. """ directory = posture_store.state_dir() return None if directory is None else directory / RECORD_FILENAME
# -- parsing ---------------------------------------------------------------- def _positive_int(value: object) -> int | None: """*value* as a positive integer, or ``None``. ``bool`` is excluded on purpose: it is an ``int`` in Python, and a payload carrying ``true`` where a PID belongs states nothing about a process. """ if isinstance(value, bool) or not isinstance(value, int) or value <= 0: return None return int(value) def _parse_owner(value: object) -> Owner | None: """One ``owner`` object as an :class:`Owner`, or ``None`` when ownerless. Rule 2 of the module contract for this field: an owner has to name a kind this version knows and a plausible PID, because the answer decides whether another process defers to it or takes the record over. Anything less is ownerless — the claimable state — rather than an error. """ if not isinstance(value, dict): return None kind = value.get("kind") pid = _positive_int(value.get("pid")) if kind not in _OWNER_KINDS or pid is None: return None return Owner(kind=kind, pid=pid, port=_positive_int(value.get("port"))) def _switch_word(value: object) -> Literal["asked", "allowed"] | None: """One stored ``switch`` value as the word it arms with, or ``None`` for off. Rule 2: an exact match, no folding and no trimming. The legacy ``"on"`` is ``"asked"``. """ if not isinstance(value, str): return None if value == _SWITCH_LEGACY_ON: return _SWITCH_ASKED if value == _SWITCH_ASKED: return _SWITCH_ASKED if value == _SWITCH_ALLOWED: return _SWITCH_ALLOWED return None def _parse_switch(value: object) -> dict[str, str]: """One ``switch`` value as ``{target: "asked" | "allowed"}`` (rule 2). Keys are kept whether or not they name a known target, so a turn-off of every machine can see and clear them; a reader only ever asks about the target it is writing to. Because every write stores what this returned, a map spelling ``"on"`` is rewritten in the new words by the next write. """ if not isinstance(value, dict): return {} return { key: word for key, state in value.items() if isinstance(key, str) and (word := _switch_word(state)) is not None } def _parse_epoch(value: object) -> int | None: """One ``switch_epoch`` as a non-negative integer, or ``None`` (epoch-less).""" if isinstance(value, bool) or not isinstance(value, int) or value < 0: return None return int(value)
[docs] def parse_record(raw: object) -> ControlContext | None: """Decode a record, or ``None`` when it states no usable context. Accepts the decoded JSON object or the raw text/bytes of one, so a reader that has already read the file and one that has not share this filter. Never raises: rule 2 of the module contract says every degraded outcome — missing, unparseable, wrong schema, unusable identity — arrives as the one value, because readers act on "no record" and must not have to tell the failures apart. """ if isinstance(raw, str | bytes | bytearray): try: raw = json.loads(raw) except Exception: # an unreadable record is "no record" logger.warning("Control-context record is not valid JSON; ignoring") return None if not isinstance(raw, dict): return None if raw.get("schema") is not _SCHEMA_VERSION: # Identity comparison, so a JSON ``true`` cannot pass for schema 1. return None target = raw.get("target") if target not in CONTROL_TARGETS: return None generation = raw.get("generation") if isinstance(generation, bool) or not isinstance(generation, int) or generation < 0: return None last_switch = raw.get("last_switch") return ControlContext( target=target, generation=generation, owner=_parse_owner(raw.get("owner")), posture=posture_store.parse_posture_value(raw.get("posture")), last_switch=dict(last_switch) if isinstance(last_switch, dict) else None, switch=_parse_switch(raw.get("switch")), switch_epoch=_parse_epoch(raw.get("switch_epoch")), )
# -- reading ----------------------------------------------------------------
[docs] def file_signature(path: Path | None) -> tuple[int, int, int] | None: """``(mtime_ns, size, inode)`` for *path*, or ``None`` when it is absent. Rule 4 of the module contract, and the same answer for everything that watches one of these files: the cache below re-parses on it, and the reconciler wakes on it. The inode is the third element because :func:`write_json_atomic` replaces the file rather than rewriting it, so two switches inside one filesystem clock tick differ there and nowhere else. An unresolvable path has no signature, which reads as "no file". """ if path is None: return None try: st = path.stat() except OSError: return None return (st.st_mtime_ns, st.st_size, st.st_ino)
class _ParsedFiles(Generic[_Parsed]): """The files one parser has read, each against the signature that produced it. One cache per parser (the record, the server reports), so each holds the one type its parser returns. Keyed by path, because a reader holds several of these files at once and a change to one of them is not a reason to re-parse the others. """ def __init__(self, parse: Callable[[Path], _Parsed]) -> None: self._parse = parse self._entries: dict[str, tuple[tuple[int, int, int], _Parsed]] = {} def read(self, path: Path) -> _Parsed | None: """*path* through the parser, re-read only when its signature moved. Rule 4. An absent file drops whatever the cache held for that path: a report whose server exited and whose file has been swept must not go on being answered from memory. """ key = str(path) signature = file_signature(path) if signature is None: with _CACHE_LOCK: self._entries.pop(key, None) return None with _CACHE_LOCK: cached = self._entries.get(key) if cached is not None and cached[0] == signature: return cached[1] value = self._parse(path) with _CACHE_LOCK: if len(self._entries) >= _CACHE_MAX_ENTRIES: self._entries.clear() self._entries[key] = (signature, value) return value def clear(self) -> None: with _CACHE_LOCK: self._entries.clear() #: How a record read ended: a usable record, no file, or a file (or root) #: that states nothing usable. _RecordStatus = Literal["ok", "missing", "unreadable"] def _read(path: Path) -> tuple[ControlContext | None, _RecordStatus]: """*path* parsed, with how the read ended. Uncached; never raises.""" try: raw = path.read_text(encoding="utf-8") except FileNotFoundError: return None, "missing" except (OSError, UnicodeDecodeError): # ``UnicodeDecodeError`` is a ``ValueError`` rather than an ``OSError`` # and so needs naming: a record that is not UTF-8 is unreadable for the # same reason a corrupt one is, and every reader here answers that with # "no record" instead of an exception raised inside a write path. logger.warning("Could not read the control-context record at %s", path, exc_info=True) return None, "unreadable" record = parse_record(raw) return (None, "unreadable") if record is None else (record, "ok") _RECORDS = _ParsedFiles(_read)
[docs] def read_record(*, path: Path | None = None) -> ControlContext | None: """The deployment's control context, or ``None`` when there is none. ``None`` for every degraded outcome (rule 2): no agent-data root, no file, an unreadable or corrupt file, a payload from another schema, or one whose target or generation cannot be used. Callers fall back to the deployment baseline or refuse, and none of them has to tell those cases apart. Re-stats on every call and re-parses only when the signature moved (rule 4). The returned record is the cached instance — treat it as immutable. Args: path: Read this file instead of :func:`record_path`. For a caller that has already resolved a root, and for tests. """ record, _status = read_record_status(path=path) return record
[docs] def read_record_status( *, path: Path | None = None ) -> tuple[ControlContext | None, Literal["ok", "missing", "unreadable"]]: """The record and how reading it ended — for a caller that must tell why. :func:`read_record` answers ``None`` for "nobody has written one yet" and for "something broke it" alike; a write verdict refuses both but names them differently. ``"missing"`` is no file at the path; ``"unreadable"`` is every other way of having no usable record — an unresolvable agent-data root, an OS or decode failure, invalid JSON, a foreign schema, an unusable identity. ``"ok"`` comes with the record. Cached like :func:`read_record` (rule 4); never raises. Args: path: Read this file instead of :func:`record_path`. """ target_path = record_path() if path is None else path if target_path is None: return None, "unreadable" cached = _RECORDS.read(target_path) if cached is not None: return cached # No signature: absent, or a stat that failed for another reason. try: target_path.stat() except FileNotFoundError: return None, "missing" except OSError: return None, "unreadable" # It appeared between the two stats; this read answers for that instant. return _read(target_path)
[docs] def invalidate_cache() -> None: """Forget every parsed file. For tests and for a process that moved roots.""" _RECORDS.clear() _REPORTS.clear()
# -- writing ----------------------------------------------------------------
[docs] def write_json_atomic(path: Path, payload: Mapping[str, object]) -> None: """Write *payload* to *path* as JSON, replacing any existing file. Rule 3. The file is replaced whole through ``safety_file.atomic_replace``, so a failed write leaves neither half a file nor litter beside it. Every file in the state directory is written this way, the record and the per-server reports and requests alike: they are read by processes that never coordinate with the writer, so a reader must never be able to see half of one. The mode is load-bearing, not a preference: those readers run under other uids — a dispatch worker at uid 1000 reading what the host account wrote — and reach the file through the setgid group of the state directory, so the file is given 0640 whatever the umask: under umask 077 a record only the owner can read makes every owned lane write refuse ``control_context_unavailable``, while CI at umask 022 stays green and never sees it. Every directory this writer creates — the levels above the leaf as much as the leaf — is set to exactly 0o2770 rather than merely widened, because a directory left at the umask 0755 would let ``other`` list the identity names (``safety_file.ensure_shared_dir``). Raises: OSError: The write itself failed. """ ensure_shared_dir(path.parent) atomic_replace( path, json.dumps(payload, indent=2).encode("utf-8"), mode=SHARED_FILE_MODE, sync="best_effort", )
def _lock_path(path: Path) -> Path: """The lock file for the record at *path*: ``<path>.lock``. Rule 3.""" return path.with_name(path.name + _LOCK_SUFFIX) def _open_lock_file(lock: Path) -> int: """Open *lock* for :func:`_record_lock`, creating it at 0o660 if absent. Neither open follows a symbolic link at the last component: the create uses ``O_EXCL | O_NOFOLLOW``, and the reopen of an existing lock uses ``O_NOFOLLOW``. A lock removed between the two is created once more. Raises: OSError: A symbolic link sits at *lock* (``errno.ELOOP``), or *lock* could not be created or opened. """ for _ in range(2): try: fd = os.open(lock, os.O_RDWR | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, SHARED_LOCK_MODE) except FileExistsError: pass else: try: # The create honoured the umask; this file is this writer's own. os.fchmod(fd, SHARED_LOCK_MODE) except BaseException: os.close(fd) raise return fd try: return os.open(lock, os.O_RDWR | os.O_NOFOLLOW) except FileNotFoundError: continue return os.open(lock, os.O_RDWR | os.O_NOFOLLOW) @contextlib.contextmanager def _record_lock(path: Path) -> Iterator[None]: """Hold the exclusive lock on the record at *path*. Rule 3. A bounded wait: a holder that does not let go inside :data:`_LOCK_TIMEOUT_S` is a wedged process, and the write refuses rather than hanging a route behind it. The lock file is given 0o660 like the record is given 0o640: writers under other uids reach it through the state directory's setgid group. Only a lock file this call created is given that mode. One that already exists is reused as it stands: it may be another uid's, and a mode set on it here would be a mode set on whatever the path resolves to. Neither open follows a symbolic link at the last component (``O_NOFOLLOW``), because the directory is group-writable by parties other than its owner: a link planted there would otherwise have this writer chmod and lock a file of the planter's choosing. A link refuses the write. Raises: RuntimeError: The lock was not acquired in time. OSError: A symbolic link sits where the lock file belongs (``errno.ELOOP``), or the lock file could not be created or opened. """ ensure_shared_dir(path.parent) fd = _open_lock_file(_lock_path(path)) try: deadline = time.monotonic() + _LOCK_TIMEOUT_S while True: try: fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) break except BlockingIOError: if time.monotonic() >= deadline: raise RuntimeError( f"The control-context record at {path} stayed locked for " f"{_LOCK_TIMEOUT_S:g} s; nothing was written" ) from None time.sleep(_LOCK_POLL_S) try: yield finally: fcntl.flock(fd, fcntl.LOCK_UN) finally: os.close(fd) def _write_record_unlocked(record: ControlContext, path: Path) -> None: """Store *record* exactly as given. The caller holds :func:`_record_lock`. The one place the record's bytes are written. Private, and called only from inside this module with the lock held: a write that skipped the lock, or that wrote a ``switch`` it did not read, is the race rule 3 closes. """ write_json_atomic(path, record.to_payload())
[docs] def write_record(record: ControlContext, *, path: Path | None = None) -> Path: """Store *record*, replacing any existing one atomically. Rule 3. Under the record lock, this re-reads the file on disk and stores *record* with the on-disk ``switch`` and ``switch_epoch`` in place of its own: no caller of this function changes either, whatever it read before. A file that is missing or unreadable is replaced with the switch empty and a fresh ``time.time_ns()`` epoch, never 0, so a widening begun before the record was lost names an older epoch and is refused. It does not check who owns the record: the owner decides what it is changing and whether it is still entitled to, around this call. Args: record: The context to store. path: Write this file instead of :func:`record_path`. Returns: The path written. Raises: RuntimeError: The agent-data root does not resolve, so there is nowhere a reader would look for what this call would write; or the record stayed locked past the bounded wait. OSError: The write itself failed. """ target_path = record_path() if path is None else path if target_path is None: raise RuntimeError( "No agent-data root: the control-context record has nowhere to be written" ) with _record_lock(target_path): on_disk, _status = _read(target_path) if on_disk is None: stored = replace(record, switch={}, switch_epoch=time.time_ns()) else: stored = replace(record, switch=dict(on_disk.switch), switch_epoch=on_disk.switch_epoch) _write_record_unlocked(stored, target_path) return target_path
#: The keyword a person's gesture uses for "every machine": only ever off. _ALL_TARGETS = "all" #: A machine's three positions, narrowest first: a later one widens an earlier #: one. The gesture's words and every reader's order. ``"off"`` is not stored #: (rule 2). SWITCH_POSITIONS: tuple[Literal["off"], Literal["asked"], Literal["allowed"]] = ( _SWITCH_OFF, _SWITCH_ASKED, _SWITCH_ALLOWED, )
[docs] class StoppedSince(Exception): """A widening refused because a stop, or a lost record, came after it began. Rule 3: a person widening a machine names the epoch they saw. A record whose epoch differs from it — newer or older —, a gesture that names none, a record that has no epoch, and no record at all each mean that what they saw is no longer what is on disk, so nothing is armed. The chip retries on its own, with :attr:`epoch`, only when :attr:`stamped`; otherwise it re-renders and needs a new click, so a widening that raced a stop stays off. Not a ``RuntimeError``: a caller that answers ``RuntimeError`` with "the store is unavailable" must not answer a stale gesture the same way. Attributes: epoch: The record's epoch now — the one a retried gesture names — or ``None`` when there is no usable record to retry against. stamped: ``True`` only when this refusal just stamped an epoch on an epoch-less record — nothing was stopped, so a retry is safe. reason: Always ``"stopped_since"``, a member of :data:`osprey_connectors.refusals.REASONS`. """
[docs] def __init__(self, *, epoch: int | None, stamped: bool = False) -> None: self.epoch = epoch self.stamped = stamped self.reason = "stopped_since" super().__init__(refusal_phrase(self.reason))
def _gesture_targets(targets: Sequence[str] | Literal["all"], state: str) -> list[str] | None: """*targets* checked as named machines, or ``None`` for ``"all"``. Raises: ValueError: Anything that is not ``"all"`` or a non-empty sequence of names; ``"all"`` with anything but ``"off"``; ``"asked"`` or ``"allowed"`` for a name outside :data:`~osprey_connectors.types.CONTROL_TARGETS`. """ if targets == _ALL_TARGETS: if state != _SWITCH_OFF: raise ValueError(f'set_switch("all", {state!r}): machines are turned on one at a time') return None if isinstance(targets, str) or not isinstance(targets, Sequence): raise ValueError( f'set_switch targets must be "all" or a sequence of names, not {targets!r}' ) names = list(targets) if not names or not all(isinstance(name, str) for name in names): raise ValueError( f"set_switch targets must be a non-empty sequence of names, not {targets!r}" ) if state != _SWITCH_OFF: unknown = [name for name in names if name not in CONTROL_TARGETS] if unknown: raise ValueError(f"set_switch cannot turn on unknown targets {unknown!r}") return names def _widens(state: str, position: str) -> bool: """Whether moving a machine at *position* to *state* widens it.""" return SWITCH_POSITIONS.index(state) > SWITCH_POSITIONS.index(position)
[docs] def set_switch( targets: Sequence[str] | Literal["all"], state: Literal["off", "on", "asked", "allowed"], *, epoch: int | None = None, path: Path | None = None, by: str | None = None, ) -> ControlContext | None: """The person's switch gesture — the one writer of ``switch``. Rule 3. Reads, compares and writes in one hold of the record lock, through the same private writer :func:`write_record` uses, so no other write lands between what this compared and what it stored. Each named target's move is classified inside the lock, against the map on disk, so a move believed to be a narrowing that raced a stop is judged a widening and refused. * A **widening** (one name moving up off → asked → allowed gates the whole call) requires *epoch*, the ``switch_epoch`` the person saw, and succeeds only when it equals the record's epoch — a compare-and-swap. It raises :class:`StoppedSince` carrying the on-disk epoch when the two differ (newer or older) or *epoch* is ``None``, when the record has no epoch (one is stamped and ``stamped`` is set, so the retried gesture succeeds), or when there is no usable record. * A **narrowing or a no-op** applies unconditionally and never refuses. ``"off"`` clears each named key, known target or not. * ``"all"`` (with ``"off"``) clears every key and moves the epoch to ``max(old + 1, time.time_ns())``, which makes every widening begun before it stale. Only a stop moves an existing epoch; a gesture on an epoch-less record stamps a fresh ``time.time_ns()``. Every other field is kept as on disk. A stop also ends every program authorization (a grant of :mod:`~osprey_connectors.write_authorization`) on the machines it names, or on every machine for ``"all"``: inside the same hold of the record lock, after the record is stored, each grant in the ``authorizations`` directory beside the record is marked revoked, so the holder's next write refuses. ``"all"`` also moves the epoch, which ends every grant issued under the old one whatever its file says. The marks are written even when no record is there. A stop fails closed on its marks: when they cannot be written (the grants directory locked past the wait, a lock or a grant file this process cannot open), a stop of named machines moves the epoch as well, in the same hold, so every grant on every machine of this record ends — wider than asked, never narrower — and the stop is taken. Raising instead would report the stop not taken with the switch already stored and the grant still in force; revoking before storing would leave the switch on as well. A stop with no record needs no fallback: no grant stands without an epoch. Args: targets: The machines, or ``"all"``. state: ``"off"``, ``"asked"`` or ``"allowed"``; the legacy ``"on"`` is taken as ``"asked"`` and never stored. epoch: The epoch the gesture was begun against. Ignored when nothing widens. path: Write this file instead of :func:`record_path`. by: Whose stop a revocation names; ``None`` is this process's :func:`~osprey_connectors.identity.acting_identity`. Returns: The record as stored, or ``None`` when a non-widening gesture found no usable record — nothing is on, so no record was written. Raises: ValueError: ``"all"`` with anything but ``"off"``, an unknown *state*, malformed *targets*, or an *epoch* that is neither ``None`` nor an integer. Nothing is written. StoppedSince: A widening that is stale, as above. RuntimeError: The agent-data root does not resolve, or the record stayed locked past the bounded wait. OSError: The write itself failed. """ if state == _SWITCH_LEGACY_ON: # A framework released before the three positions sends "on". state = _SWITCH_ASKED if state not in SWITCH_POSITIONS: raise ValueError(f"set_switch state must be one of {SWITCH_POSITIONS}, not {state!r}") names = _gesture_targets(targets, state) if state != _SWITCH_OFF and ( epoch is not None and (isinstance(epoch, bool) or not isinstance(epoch, int)) ): raise ValueError(f"set_switch epoch must be an integer or None, not {epoch!r}") target_path = record_path() if path is None else path if target_path is None: raise RuntimeError( "No agent-data root: the control-context record has nowhere to be written" ) with _record_lock(target_path): current, _status = _read(target_path) if current is None: if state != _SWITCH_OFF: raise StoppedSince(epoch=None) # No record, no epoch: no grant stands without one, marked or not. _try_revoke_on_stop(target_path, names, by) return None on_disk_epoch = current.switch_epoch if names is None: stored = replace( current, switch={}, switch_epoch=max((on_disk_epoch or 0) + 1, time.time_ns()), ) else: if any(_widens(state, switch_position(current, name)) for name in names): if on_disk_epoch is None: stamped = replace(current, switch_epoch=time.time_ns()) _write_record_unlocked(stamped, target_path) raise StoppedSince(epoch=stamped.switch_epoch, stamped=True) if epoch != on_disk_epoch: raise StoppedSince(epoch=on_disk_epoch) if state == _SWITCH_OFF: switch = {key: value for key, value in current.switch.items() if key not in names} else: switch = {**current.switch, **dict.fromkeys(names, state)} stored = replace( current, switch=switch, switch_epoch=time.time_ns() if on_disk_epoch is None else on_disk_epoch, ) _write_record_unlocked(stored, target_path) marked = state != _SWITCH_OFF or _try_revoke_on_stop(target_path, names, by) if not marked and names is not None: # The marks could not be written, so the stop falls back on the # epoch, which ends every grant issued under the old one on every # machine of this record: wider than asked, never narrower. A stop # of every machine has moved it already. stored = replace( stored, switch_epoch=max((stored.switch_epoch or 0) + 1, time.time_ns()) ) _write_record_unlocked(stored, target_path) return stored
[docs] class RecordUnavailable(Exception): """The record is there and cannot be read, so nothing can be stood on it."""
[docs] @contextlib.contextmanager def held_epoch(path: Path, target: str) -> Iterator[int]: """Hold the record lock at *path* and yield the record's ``switch_epoch``. Rule 3. For a writer that issues something standing on the epoch — an approval of a program's authorization — and must not have a stop land between reading it and writing what it stamps: the block runs inside the lock, and any lock the block takes comes after this one. A record that is not there is created on *target* with every switch off and a fresh epoch, and an epoch-less record is given one, so that a later stop of every machine has an epoch to move past. Raises: RecordUnavailable: The record is there and cannot be read. RuntimeError: The record stayed locked past the bounded wait. OSError: The stamp could not be written. """ with _record_lock(path): current, status = _read(path) if status == "unreadable": raise RecordUnavailable(f"The control-context record at {path} cannot be read") epoch = None if current is None else current.switch_epoch if epoch is None: epoch = time.time_ns() base = ControlContext(target=target, generation=0) if current is None else current _write_record_unlocked(replace(base, switch_epoch=epoch), path) yield epoch
def _revoke_on_stop(record_file: Path, names: list[str] | None, by: str | None) -> None: """Mark every program authorization on the stopped machines revoked. Rule 3. The stop's second half: the grants beside the record, in its ``authorizations`` directory, on *names* (``None`` is every machine), get ``revoked: {at_ns, by}``, so the holder's next write refuses ``auth_revoked``. The caller holds the record lock; the grants directory's own lock is taken inside it — the one lock order every writer of both keeps. A self-issued grant sits in its holder's directory, out of reach here; it lapses on its own once the switch has left ``allowed``. Imported here, upward across the layers by one named edge, because the grant format belongs to :mod:`~osprey_connectors.write_authorization`, which reads this module. Raises: RuntimeError: The grants directory stayed locked. OSError: It is not a plain directory, or a mark could not be written. """ from osprey_connectors import write_authorization write_authorization.revoke_grants( by=by or acting_identity(), targets=names, directory=record_file.parent / write_authorization.AUTHORIZATIONS_DIR_NAME, ) def _try_revoke_on_stop(record_file: Path, names: list[str] | None, by: str | None) -> bool: """:func:`_revoke_on_stop`, ``False`` (logged) when the marks could not be written. A stop never reports itself not taken over its marks: the caller ends the grants through the epoch instead (fail-closed, see :func:`set_switch`). """ try: _revoke_on_stop(record_file, names, by) except (OSError, RuntimeError): logger.warning( "A stop could not mark the grants beside %s revoked; it ends them by the epoch", record_file, exc_info=True, ) return False return True # -- process liveness ------------------------------------------------------- _PROC_ROOT = "/proc" def _is_thread_id(pid: int) -> bool: """Whether the kernel knows *pid* as a thread of some other process. Linux draws PIDs and thread ids from one number space, and ``/proc/<tid>`` opens by number though no listing shows it. A process is its own thread group leader, so a ``Tgid`` that differs from *pid* means a thread. ``False`` when ``/proc`` says nothing, which is every platform without one. """ try: with open(f"{_PROC_ROOT}/{pid}/status", "rb") as status: for line in status: if line.startswith(b"Tgid:"): return int(line.split()[1]) != pid except (OSError, ValueError, IndexError): return False # no /proc, or one that does not read like Linux's: no claim either way return False
[docs] def is_process_alive(pid: object) -> bool: """Whether *pid* names a running process. ``os.kill(pid, 0)`` sends no signal and only asks the kernel whether the process exists — the ``osprey web`` PID-file check does the same. A ``PermissionError`` means the process exists but belongs to another user, so it counts as ALIVE: sweeping a file whose owner is merely unreachable would delete live state. Non-positive PIDs are rejected without calling ``os.kill`` at all, because 0 and negatives address process *groups*. ``os.kill`` also answers for a thread id, which is no process: a PID kept in a record that outlived its container can land on a thread of the process reading it, which would then follow itself for ever (:func:`_is_thread_id`). A real process on that PID is not told apart. Anything that is not an ``int`` names no process and is ``False`` without a call either, so a field read off a file can be handed over unconverted. That includes ``bool``: ``True`` is ``1`` to ``os.kill``, and PID 1 is always alive, so a report carrying ``"server_pid": true`` would otherwise read as a running server forever. """ if isinstance(pid, bool) or not isinstance(pid, int) or pid <= 0: return False try: os.kill(pid, 0) except ProcessLookupError: return False except PermissionError: pass # another user's process still exists; the thread check below applies to it too except OSError: # pragma: no cover - platform oddity; assume alive return True return not _is_thread_id(pid)
def _is_foreign_process(pid: object) -> bool: """Whether *pid* names a running process that belongs to another user. The other half of what ``os.kill(pid, 0)`` says. :func:`is_process_alive` reads a ``PermissionError`` as ALIVE, which is right for a sweeper — a file whose author is merely unreachable must not be deleted. It is the wrong answer to "does this process own the record": the record lives in the acting identity's directory (rule 1), so its owner runs as that identity and can always be signalled by a reader that does. A process this reader may not signal is somebody else's process wearing a PID the record's owner once had — PIDs are per container and start again at 1 with every one. ``False`` for anything that is not a positive ``int``, for a PID nothing runs under, and on any other error: only the kernel's explicit refusal says "another user's". """ if isinstance(pid, bool) or not isinstance(pid, int) or pid <= 0: return False try: os.kill(pid, 0) except PermissionError: return True except OSError: return False return False def _pid_from_filename(name: str, prefix: str, suffix: str) -> int | None: """The PID *name* encodes, or ``None`` when it encodes none. Shared by every family of file in the state directory: each is named for the process that owns it precisely so a sweeper can judge it without opening it, and a name that does not follow the rule is judged the same way in all of them — no PID, therefore no live owner. """ if not name.startswith(prefix) or not name.endswith(suffix): return None stem = name[len(prefix) : -len(suffix)] try: return int(stem) except ValueError: return None # -- ownership --------------------------------------------------------------
[docs] def owned_here(record: ControlContext | None) -> bool: """Whether THIS process owns *record*, and may therefore write it. PID equality and nothing else: the record names one owning process, and a second rule about the owner's *kind* would let two processes disagree about who writes one file. An ownerless record is owned by nobody, so it is not owned here either — it is claimed first, and claiming is a write. """ return record is not None and record.owner is not None and record.owner.pid == os.getpid()
[docs] def live_owner(record: ControlContext | None) -> Owner | None: """*record*'s owner while its process still runs, else ``None``. The one question every would-be claimant asks first: an owner whose process is gone holds nothing, and its record is free. ``None`` covers every way of having no live owner — no record, no owner, a dead one — so a caller that only wants to know whether to claim need not tell them apart. A PID that is running under another user is not the owner either (:func:`_is_foreign_process`). The record outlives the container that wrote it, and the PID it names is then handed out again: a record written by a terminal that ran as PID 1 names, in the next container, whatever runs as PID 1 there — an init process owned by root, alive for as long as the container is. Read as a live owner, that process would be followed by every terminal for ever, and nobody could switch the target again. """ if record is None or record.owner is None: return None pid = record.owner.pid if not is_process_alive(pid) or _is_foreign_process(pid): return None return record.owner
# -- the terminus of a switch ----------------------------------------------
[docs] def unchanged_detail(target: str, generation: int) -> str: """The ``detail`` for a request that asked for the target of record. One sentence with one spelling, because four surfaces write it into the same field: the terminal owner's request half, the controls server's reconciler, the HTTP switch route and the agent's own ``control_target_set``. A record whose terminus read differently depending on which surface answered would be two vocabularies in one place, and the chip resolves a pending switch against what it finds there. It is an APPLIED terminus and mints no generation: the deployment is where the request asked for it to be, and a generation bumped for a switch that did not happen would refuse every write bound to the old one, for nothing. """ return f"The control target is already {target!r} (generation {generation})."
[docs] def applied_detail(target: str, generation: int) -> str: """The ``detail`` for a switch that happened. *generation* is the new one. The companion to :func:`unchanged_detail`, written by the same four surfaces and for the same reason. """ return f"The control target is now {target!r} (generation {generation})."
[docs] def terminus( record: ControlContext, *, request_id: str, target: str, requested_at: object, requested_by: str, status: str, reason: str | None, detail: str, generation: int | None, ) -> ControlContext: """*record* as it will be once a switch request has been answered. Every answer to a switch gesture ends here, whoever writes it: the owning server answering its own agent, and the reconciler answering a request another process filed. The block is what a requester polls for — it looks for its own ``request_id`` and reads the outcome beside it — so the shape is fixed here rather than at each terminus. *generation* is what the record carries AFTER the answer, and ``None`` is what makes the answer a refusal: a refusal moves neither target nor generation, and its block names no binding because there is none. Args: record: The record being answered against. Not modified. request_id: The id the requester is waiting on. target: The target that was asked for — recorded whether or not the deployment went there, because a refusal has to say what it refused. requested_at: The request's own stamp, copied verbatim. requested_by: Who to name in the block: an audit session where the requester has one, ``pid:<n>`` where it does not. status: :data:`SWITCH_APPLIED` or :data:`SWITCH_REFUSED`. reason: The refusal's machine-readable reason, ``None`` when applied. detail: The sentence an operator reads. generation: The record's generation after the answer, or ``None`` for a refusal. """ moved = generation is not None return replace( record, target=target if moved else record.target, generation=generation if generation is not None else record.generation, last_switch={ "request_id": request_id, "target": target, "requested_at": requested_at, "requested_by": requested_by, "status": status, "reason": reason, "detail": detail, "generation": generation, }, )
[docs] def write_terminus(updated: ControlContext, request_id: str) -> bool: """Write *updated*, read it back, and say whether the answer stuck. Read-verified because the record has one owner but two processes reach for it: a server answers its own agent in one place and consumes other processes' requests in another, and this call can also lose the record outright to a new owner between the write and the read. A lost update has to be discovered here rather than reported to an agent as a switch that happened. The write goes through :func:`write_record`, so it holds the record lock and keeps the on-disk switch: a terminus never turns a machine on or off. Three things have to hold for the answer to be believed: the record is still readable, this process still owns it, and the block in it is the one just written. The caller decides what a ``False`` means for it — the owner refuses its agent, the reconciler leaves the request file for the next pass — and neither may treat the switch as done. """ try: write_record(updated) except Exception: logger.warning("Could not write the control-context record", exc_info=True) return False confirmed = read_record() return ( confirmed is not None and owned_here(confirmed) and (confirmed.last_switch or {}).get("request_id") == request_id )
# -- the per-server reports -------------------------------------------------
[docs] @dataclass(frozen=True) class ServerReport: """What one controls server says about itself. Treat an instance as immutable. Rule 5 of the module contract. Every field but ``server_pid`` is an observation that can be absent — a server that has just started has answered nothing yet — so each degrades on its own to a value a reader can act on rather than invalidating the report. Attributes: server_pid: The reporting server. Its identity, and how a reader knows whether the report still describes anything. session: The server's ``OSPREY_POSTURE_SESSION``, or ``None`` for a bare ``claude`` that was never stamped with one. applied_target: The target its connector host is actually on, or ``None`` until the first child has answered its init frame. Null is not "baseline": it is "this server has not got there yet". applied_generation: The generation that target was reached at, on the same terms. children: The connector-host PIDs it spawned, so a later server can reap them if this one is killed. reachability: Its last probe per target, ``{}`` when it has not probed. last_switch: Its progress through the current switch — ``applying``, ``applied`` or ``failed``, with the ``expires_at`` bound only this process could compute — or ``None``. last_posture_realign: Its last posture realignment, or ``None``. targets: Per-target display metadata, ``{}`` when it has none. updated_at: When it last wrote, ISO-8601, or ``None``. statement: The machine statement a process that displays the machine (a panel backend) writes in place of controlling it, ``None`` on a controls server's report. ``{"reachable": ..., "writes": ...}``, each axis ``None`` until stated: ``reachable`` is ``{"state", "since", "reason"}`` with ``state`` one of :data:`_REACHABLE_STATES`, and ``writes`` is :func:`write_statement` of the session's execution verdict. A statement that does not have this shape states nothing, and the report is then an ordinary one. """ server_pid: int session: str | None = None applied_target: str | None = None applied_generation: int | None = None children: tuple[int, ...] = () reachability: dict[str, object] = field(default_factory=dict) last_switch: dict[str, object] | None = None last_posture_realign: dict[str, object] | None = None targets: dict[str, object] = field(default_factory=dict) updated_at: str | None = None statement: dict[str, dict[str, object] | None] | None = None @property def display_only(self) -> bool: """Whether this report is a display process's: it carries a statement. Such a process shows the machine rather than controlling it, so the fleet readers leave its report out (:func:`live_reports`); a view that shows the machine reads it with :func:`read_report`. """ return self.statement is not None
[docs] def to_payload(self) -> dict[str, object]: """This report as the JSON object its file holds. The ten contract fields, plus ``statement`` as the eleventh for a display-only report and only then, so a payload round-trips through :func:`_parse_report` unchanged and a controls server's report never reads back as display-only. """ payload: dict[str, object] = { "server_pid": self.server_pid, "session": self.session, "applied_target": self.applied_target, "applied_generation": self.applied_generation, "children": list(self.children), "reachability": dict(self.reachability), "last_switch": None if self.last_switch is None else dict(self.last_switch), "last_posture_realign": ( None if self.last_posture_realign is None else dict(self.last_posture_realign) ), "targets": dict(self.targets), "updated_at": self.updated_at, } if self.statement is not None: payload["statement"] = { axis: None if value is None else dict(value) for axis, value in self.statement.items() } return payload
def _non_negative_int(value: object) -> int | None: """*value* as a non-negative integer, or ``None``. ``bool`` excluded.""" if isinstance(value, bool) or not isinstance(value, int) or value < 0: return None return int(value) def _text(value: object) -> str | None: """*value* as a non-empty string, or ``None``. An empty string states nothing, and a reader that compared one to a session id would be comparing to a value no writer here ever means. """ return value if isinstance(value, str) and value else None def _child_pids(value: object) -> tuple[int, ...]: """*value* as the positive child PIDs it names, in order, without repeats.""" if not isinstance(value, list | tuple): return () pids: list[int] = [] for item in value: pid = _positive_int(item) if pid is not None and pid not in pids: pids.append(pid) return tuple(pids) def _mapping(value: object) -> dict[str, object]: """*value* as a mapping, or an empty one. Anything else states nothing.""" return dict(value) if isinstance(value, dict) else {} #: What a statement says about reaching the machine. ``yes`` until the #: connector is lost, ``no`` once it is — the lost-connector root, #: :class:`~osprey_connectors.errors.ConnectorUnavailableError`. A channel that #: does not answer (:class:`~osprey_connectors.errors.ChannelUnreachableError`) #: is that channel's failure and never makes the machine unreachable. _REACHABLE_STATES: tuple[str, ...] = ("yes", "no") #: The keys of a statement's ``writes``: the execution verdict's own terms. _WRITES_STATEMENT_KEYS: tuple[str, ...] = ("allowed", "position", "reason", "kind") _WRITE_POSITIONS: tuple[str | None, ...] = (None, "asked", "allowed") _MACHINE_KINDS: tuple[str | None, ...] = (None, "real", "rehearsal", "simulator")
[docs] def write_statement(verdict: posture_store.WriteVerdict) -> dict[str, object]: """The ``writes`` half of a machine statement, from one execution verdict. The verdict's own terms and nothing else: ``allowed``, ``position`` (whether a person is asked, never a boolean "on"), ``reason`` (a :data:`~osprey_connectors.refusals.REASONS` member when refused) and ``kind``. A reader renders the refusal with :func:`~osprey_connectors.refusals.refusal_phrase`; no surface words it again. Args: verdict: :func:`~osprey_connectors.posture_store.execution_verdict` for the session's target, so a held session states ``session_held``. """ return { "allowed": verdict.allowed, "position": verdict.position, "reason": verdict.reason, "kind": verdict.kind, }
def _reachable_statement(value: object) -> dict[str, object] | None: """*value* as the ``reachable`` axis, or ``None`` when it is not one.""" if not isinstance(value, dict) or value.get("state") not in _REACHABLE_STATES: return None since, reason = value.get("since"), value.get("reason") if not (since is None or isinstance(since, str)): return None if not (reason is None or isinstance(reason, str)): return None return {"state": value["state"], "since": since, "reason": reason} def _writes_statement(value: object) -> dict[str, object] | None: """*value* as the ``writes`` axis, or ``None`` when it is not a verdict's terms.""" if not isinstance(value, dict) or not isinstance(value.get("allowed"), bool): return None if value.get("position") not in _WRITE_POSITIONS or value.get("kind") not in _MACHINE_KINDS: return None reason = value.get("reason") if not (reason is None or isinstance(reason, str)): return None return {key: value.get(key) for key in _WRITES_STATEMENT_KEYS} def _statement(value: object) -> dict[str, dict[str, object] | None] | None: """*value* as a machine statement, or ``None`` when it is not one. A statement is a mapping holding both ``reachable`` and ``writes``, each ``null`` (not stated yet) or of its axis's shape. Anything else states nothing: the report is then not a display process's, and the fleet reads it as it reads any other. """ if not isinstance(value, dict) or "reachable" not in value or "writes" not in value: return None reachable, writes = value["reachable"], value["writes"] if reachable is not None: reachable = _reachable_statement(reachable) if reachable is None: return None if writes is not None: writes = _writes_statement(writes) if writes is None: return None return {"reachable": reachable, "writes": writes} def _parse_report(raw: object) -> ServerReport | None: """Decode one server report, or ``None`` when it describes no server. Rule 5. ``server_pid`` is the identity and the only field that can make the whole report worthless: without a plausible PID there is nothing to ask the kernel about, so the file says nothing a reader can use. Everything else degrades to its own "not stated" value. A ``statement`` of the statement's shape makes the report a display process's; one of any other shape states nothing, so the writer of a display report keeps to the shape or is read as a controls server. """ if not isinstance(raw, dict): return None server_pid = _positive_int(raw.get("server_pid")) if server_pid is None: return None applied_target = raw.get("applied_target") last_switch = raw.get("last_switch") last_realign = raw.get("last_posture_realign") return ServerReport( server_pid=server_pid, session=_text(raw.get("session")), applied_target=applied_target if applied_target in CONTROL_TARGETS else None, applied_generation=_non_negative_int(raw.get("applied_generation")), children=_child_pids(raw.get("children")), reachability=_mapping(raw.get("reachability")), last_switch=dict(last_switch) if isinstance(last_switch, dict) else None, last_posture_realign=dict(last_realign) if isinstance(last_realign, dict) else None, targets=_mapping(raw.get("targets")), updated_at=_text(raw.get("updated_at")), statement=_statement(raw.get("statement")), ) def _report_path_under(root: Path, server_pid: object) -> Path: """The report path of *server_pid* under an explicit agent-data *root*. Through the same three hops as :func:`_record_path_under`, identity included: every family of file in this directory moves together, so a caller that finds the record under a root finds the reports beside it. """ return ( root / STATE_DIR_NAME / acting_identity() / f"{REPORT_FILE_PREFIX}{server_pid}{REPORT_FILE_SUFFIX}" )
[docs] def report_path(server_pid: object) -> Path | None: """The report path of *server_pid*, or ``None`` without an agent-data root.""" directory = posture_store.state_dir() if directory is None: return None return directory / f"{REPORT_FILE_PREFIX}{server_pid}{REPORT_FILE_SUFFIX}"
[docs] def report_paths() -> list[Path]: """Every report file in the state directory, sorted. Empty when there is none. Never raises: no agent-data root, no directory and an unreadable one all answer "no reports", the same as an empty deployment, because a caller describing a fleet has the same nothing to say in each case. """ directory = posture_store.state_dir() if directory is None: return [] try: return sorted(directory.glob(REPORT_FILE_GLOB)) except OSError: # pragma: no cover - unreadable state dir return []
def _read_payload(path: Path) -> dict[str, object] | None: """One report file as its JSON object, or ``None``. Never raises.""" try: raw = path.read_text(encoding="utf-8") except FileNotFoundError: return None except OSError: logger.debug("Could not read the server report at %s; skipping it", path) return None try: loaded: object = json.loads(raw) except Exception: # a half-written report is skipped, not fatal logger.debug("Server report %s is not valid JSON; skipping it", path.name) return None return loaded if isinstance(loaded, dict) else None def _read_entry(path: Path) -> ServerReport | None: """One report file read and parsed once, for the cache to hold. Cached so that the parsed report a reader holds is the same object until the file actually moves. """ payload = _read_payload(path) return None if payload is None else _parse_report(payload) _REPORTS = _ParsedFiles(_read_entry)
[docs] def read_report(path: Path) -> ServerReport | None: """One server's report, or ``None`` when the file states no usable one.""" return _REPORTS.read(path)
[docs] def live_reports( entries: Iterable[Path] | None = None, *, is_alive: Callable[[object], bool] = is_process_alive, ) -> list[ServerReport]: """Every report of a running controls server, parsed. The fleet as a reader sees it: what each live server is on, at which generation, and how far through a switch it has got. Display-only reports are left out here, at the source, so no fleet reader ever counts a process that only shows the machine (:func:`read_report` reads one, and its :attr:`ServerReport.display_only` says which it is). Args: entries: Report paths; :func:`report_paths` when omitted. is_alive: The liveness predicate, so one pass over a fleet asks the same question the same way throughout. """ return [report for report in _live_parsed(entries, is_alive) if not report.display_only]
def _live_parsed( entries: Iterable[Path] | None, is_alive: Callable[[object], bool], ) -> list[ServerReport]: """Every parsed report among *entries* whose process still runs.""" paths = report_paths() if entries is None else entries reports: list[ServerReport] = [] for entry in paths: report = read_report(entry) if report is None or not is_alive(report.server_pid): continue reports.append(report) return reports # -- sweeping ---------------------------------------------------------------
[docs] def sweep_dead( directory: Path, *, prefix: str, suffix: str, keep: str | None = None, is_alive: Callable[[object], bool] = is_process_alive, ) -> list[Path]: """Unlink every ``prefix<pid>suffix`` file in *directory* whose PID is gone. A process that is killed rather than asked to stop leaves its file behind, and a later process that happens to reuse the PID would otherwise inherit it. The name is the whole judgement — a file whose name encodes no PID encodes no owner either, and goes the same way. Args: directory: Where to look. A missing or unreadable one sweeps nothing. prefix: Filename prefix of the family to sweep. suffix: Filename suffix of the family to sweep. keep: A filename to leave alone without probing it. This is how a process spares its own file: asking whether oneself is alive is a way to get it wrong. is_alive: The liveness predicate. Returns: The paths removed, in discovery order. """ try: entries = sorted(directory.glob(f"{prefix}*{suffix}")) except OSError: return [] removed: list[Path] = [] for entry in entries: if keep is not None and entry.name == keep: continue pid = _pid_from_filename(entry.name, prefix, suffix) if pid is not None and is_alive(pid): continue try: entry.unlink(missing_ok=True) except OSError as exc: # pragma: no cover - unwritable state dir logger.warning("Could not remove stale file %s: %s", entry, exc) continue logger.info("Swept stale file %s (owner pid %s gone)", entry.name, pid) removed.append(entry) return removed
# -- convergence ------------------------------------------------------------ def _switch_at(report: ServerReport, generation: int) -> Mapping[str, object] | None: """*report*'s switch block when it is about *generation*, else ``None``. A block that names a different generation is about a swap this record has already moved past — the server re-adopts on its next tick — and a block that names none names no swap the fleet is coordinating on. Neither is this record's business; a server left behind by either is still judged by what it is bound to. """ block = report.last_switch if not isinstance(block, dict): return None if _non_negative_int(block.get("generation")) != generation: return None return block def _bound_epoch(block: Mapping[str, object]) -> float | None: """The ``expires_at`` of *block* as epoch seconds, or ``None`` for no bound. The bound is a wall clock the publishing server computed from its own spawn, probe and drain timeouts — the only process that knows them — so this reader parses it and does not derive one. A naive stamp is read as UTC, the same rule the in-flight markers use, because the writer and the reader share a host even when they do not share a spelling. ``None`` is "no bound", which keeps blocking rather than expiring immediately: an unreadable deadline is not a reason to decide a swap finished. """ stamp = block.get("expires_at") if not isinstance(stamp, str) or not stamp: return None try: moment = datetime.fromisoformat(stamp) if moment.tzinfo is None: moment = moment.replace(tzinfo=UTC) return moment.timestamp() except (ValueError, OSError, OverflowError): return None def _bound_off_record(report: ServerReport, record: ControlContext) -> bool: """Whether *report* says it arrived somewhere other than the record says. Both halves of the binding have to be stated and both have to match: ``applied_target`` and ``applied_generation`` are published together but parse apart, and null in either is "this server has not got there yet", never "baseline". A server that has not got there yet is behind, not wrong, and holds nothing an operator has to clear. """ if report.applied_target is None or report.applied_generation is None: return False return (report.applied_target, report.applied_generation) != (record.target, record.generation) def _holds_no_connector(report: ServerReport) -> bool: """Whether *report* states no connector on any of the three axes that could. A server between two targets is holding one of them, which is why a swap in flight stops the whole deployment. A server that has never got anywhere is holding nothing, so nothing it is doing can reach a target another session's launch would not. All three axes have to say so, and they say so together: the binding pair and the child PIDs are published in one write once a child has answered its init frame, so a report stating a connector on any one of them is not a server acquiring its first. """ return ( not report.children and report.applied_target is None and report.applied_generation is None )
[docs] def blocking_pids( record: ControlContext, reports: Iterable[ServerReport], session: str | None, *, now: float | None = None, ) -> tuple[int, ...]: """The controls servers that stop *session* acting on *record*, PID order. Empty when there is nothing in the way — which is what :func:`converged` reports. The PIDs are the refusal text: ``switch_in_progress`` names them so an operator can find the process that has to finish or be killed, and the two callers that refuse (the executor's stamp and the kernel's cell gate) both say which. Args: record: The deployment's control context. The generation in it is what the fleet coordinates on. reports: The reports of the servers that are still running, as :func:`live_reports` returns them. A report of a dead server describes nobody and is filtered out there, not here: this function opens no file and asks the kernel nothing. session: The reader's ``OSPREY_POSTURE_SESSION``. ``None`` (and the empty string) for a reader that owns no report — a bare ``claude``, a notebook kernel's ``kernel:<id>``, and the owner asking whether the fleet as a whole has settled. now: Epoch seconds to age the ``applying`` bounds against; the wall clock when omitted. """ fleet = tuple(reports) reference = datetime.now(UTC).timestamp() if now is None else float(now) blocked: set[int] = set() for report in fleet: block = _switch_at(report, record.generation) or {} status = block.get("status") if status == REPORT_APPLYING and not _holds_no_connector(report): bound = _bound_epoch(block) if bound is None or reference <= bound: # A swap in flight: nobody launches anywhere until it lands, # because the server holding the connector is between two # targets. Past its bound it has stopped being in flight, and # only the session whose server it is is still held up by it. # A server holding no connector is not between two targets and # falls out above, on the same terms. blocked.add(report.server_pid) continue if not session or report.session != session: continue if status in (REPORT_APPLYING, REPORT_FAILED) or _bound_off_record(report, record): blocked.add(report.server_pid) return tuple(sorted(blocked))
[docs] def converged( record: ControlContext, reports: Iterable[ServerReport], session: str | None, *, now: float | None = None, ) -> bool: """Whether *session* may act on *record* — rule 6 of the module contract. False in exactly two situations, and the arguments are the same in both: :func:`blocking_pids` names the servers behind them. **A swap is in flight.** Any live server reporting ``applying`` for the record's generation, within the ``expires_at`` bound it wrote, stops every session: that server is between two targets, and a launch admitted now would reach whichever one it happens to be holding. Past the bound the server is stuck rather than working, and it stops only its own session — a process killed mid-swap must not refuse a deployment for ever. A server that reports ``applying`` while it holds no connector at all — no children, nothing bound — is acquiring its first rather than swapping one, and stops only its own session. **The reader's own server is not there.** A live report carrying this *session* that is bound somewhere other than the record's ``(target, generation)``, or that reports ``failed`` at that generation, stops that session and nothing else. That is the whole of what a failed swap costs: the session whose server could not follow is refused, its own MCP writes with it, while every other session, every notebook kernel and every bare ``claude`` carries on. A session that owns no report — ``None``, the empty string, a kernel's ``kernel:<id>`` — is therefore blocked only by the first situation. So is the owner's own check before it consumes a switch request, which passes ``None`` deliberately: it is asking whether the fleet has settled, not whether some session has. This is a judgement over parsed data and nothing else. It opens no file, signals no process and imports nothing from ``osprey``, because the executor stamps every sandbox with it and a notebook kernel gates every cell on it. """ return not blocking_pids(record, reports, session, now=now)