Search Modes#

ARIEL’s search system is built around search modules — leaf-level functions that each implement a single retrieval strategy over the logbook. The framework ships three: keyword full-text search, embedding-based semantic similarity, and hybrid, a merge of the two answered by a separate search sidecar container (qmd). All three produce a common ARIELSearchResult. Higher-level reasoning over results — multi-step retrieval, answer synthesis, custom prompting — lives in the OSPREY agent layer, which calls these search modules through ARIEL’s MCP tools.

Dispatch is registry-driven. A search request names a mode as a plain string — "keyword", "semantic", "hybrid". The ARIELSearchService looks that name up in Osprey’s central registry and calls the module’s own execute; it carries no per-mode branch of its own. The registry is the only source of routable modes. The service, the web interface’s capabilities API and the agent’s MCP tools therefore cannot disagree about which modes exist. Adding a module needs no change to the service.

Note

sql_query is a tool, not a search mode. It runs read-only SQL against the same database. That is precision filtering — exact matches, exhaustive date and author ranges, counts — not ranked retrieval. It has no relevance score to merge with the others, so it is exposed only as its own MCP tool and never appears as a --mode value.

Search Architecture#

Search-mode dispatch A query enters the ARIEL search service with a mode name; the registry resolves the mode to its module — refusing unknown or disabled modes rather than falling back — and that module's execute produces the search result. MODE user query cli · mcp · web search service search(mode=…) registry no silent fallback module execute() result entries · modes used

The service refuses a mode that is not registered, or is registered but disabled in configuration, rather than quietly falling back to another one.

CLI usage:

osprey ariel search "RF cavity fault"                  # ariel.default_search_mode
osprey ariel search "RF cavity fault" --mode keyword
osprey ariel search "RF cavity fault" --mode semantic
osprey ariel search "RF cavity fault" --mode hybrid

The --mode choices are read from the registry when the command runs. A facility that registers its own search module therefore gets it as a choice — and in --help — without any code change.

Search Modules#

Search modules are leaf-level functions that execute a single search strategy against the database. Each module exports a get_tool_descriptor() function that describes its capabilities, input schema, and execution function. The web interface discovers modules through this descriptor via ARIEL’s capabilities API; each built-in module is exposed to the OSPREY agent through its own ARIEL MCP tool (keyword_search, semantic_search, hybrid_search). The framework ships with the following built-in search modules:

Module: search/keyword.py

PostgreSQL full-text search with optional fuzzy matching fallback. Best for specific terms, equipment names, PV names, and exact phrases.

Query syntax:

# Simple terms (implicit AND)
RF cavity fault

# Boolean operators
RF AND cavity
vacuum OR pressure
beam NOT injection

# Quoted phrases
"RF cavity trip"

# Field prefixes
author:smith
date:2024-06

# Combined
author:jones "beam loss" date:2024-01

# Pattern tokens
SR01C___BPM*
/SR0[1-4]C___BPM\d+/

Pattern tokens and query expansion are covered in Vocabulary Expansion and Pattern Search.

How it works:

  1. Validates and preprocesses the query — empty queries return immediately, queries longer than 1,000 characters are truncated, and unbalanced quotes are auto-balanced by removing the last unmatched quote

  2. Parses the query to extract field filters (author:, date:), quoted phrases, and remaining search terms

  3. Builds a PostgreSQL tsquery using the function appropriate for the query shape:

    • plainto_tsquery — for simple terms (implicit AND)

    • websearch_to_tsquery — for queries with Boolean operators (AND, OR, NOT)

    • phraseto_tsquery — for quoted phrases

    When multiple components are present (terms and phrases, for example), they are combined with && (tsquery AND).

  4. Executes full-text search against the entry text (and, on a migrated store, its attachment captions) with ts_rank scoring, applying any field filters (author ILIKE, date range) and time range constraints

  5. If no entry holds every plain word, returns the entries holding most of them instead (see Which entries match below)

  6. If there are still no results and fuzzy fallback is enabled, falls back to pg_trgm trigram similarity at or above fuzzy_threshold (default: 0.3)

  7. Returns results as (entry, score, highlights) tuples — highlights are generated via ts_headline

Which entries match. A query of plain words is answered in up to three steps, all inside the same date, author and source filters:

  1. Every word. Entries containing all the plain words. If there are any, these are the answer.

  2. Most of the words. Only when no entry has them all, and the query has at least two plain words. The answer is the entries containing at least half of them, ranked by how many words they hold, then by ts_rank. Half is the same share hybrid search uses to decide that a picture caption matched. Each of these hits carries matched_terms and missing_terms. The response carries an INFO diagnostic of category partial, for example no entry contains every term; showing entries with at least 4 of 8; terms found in no hit: oct, 6. With vocabulary expansion on, a word counts as present when it or any of its alternatives is.

  3. Similar spelling. Only when both steps above found nothing: the trigram fuzzy fallback.

Quoted phrases stay required in every step. A query with AND, OR or NOT means exactly what its operators say and is never relaxed, and neither is a query carrying a * glob or a /regex/ token. To widen a search on purpose, join the alternatives with OR.

Dates and authors belong in the start_date, end_date and author filters, not in the query text: Oct 6 in the query is two more words an entry has to contain. A bare end_date such as 2025-10-06 includes that whole day (facility time); an end with a time of day is taken exactly as written.

Configuration:

ariel:
  search_modules:
    keyword:
      enabled: true
      settings:
        patterns_enabled: true          # default: true
        pattern_timeout_seconds: 10.0   # default: 10.0
        fuzzy_threshold: 0.3            # default: 0.3

Module: search/semantic.py

Embedding-based similarity search using pgvector. Best for conceptual queries where exact keywords may not appear in the text.

How it works:

  1. Resolves the similarity threshold using a 3-tier priority:

    1. Per-query similarity_threshold parameter (highest)

    2. Config value (search_modules.semantic.settings.similarity_threshold)

    3. Hardcoded default: 0.5 (lowest)

  2. Determines the embedding model from config (search_modules.semantic.model) and resolves provider credentials via Osprey’s centralized api.providers configuration

  3. Generates a query embedding using the configured provider, with a dimension-mismatch warning if the returned embedding size does not match the configured embedding_dimension

  4. Searches the per-model embedding table using cosine distance (<=> operator)

  5. Filters results by similarity threshold and optional time range

  6. Returns results as (entry, similarity_score) tuples

Configuration:

ariel:
  search_modules:
    semantic:
      enabled: true
      provider: ollama
      model: nomic-embed-text
      settings:
        similarity_threshold: 0.5
        embedding_dimension: 768

Requirements: Ollama (or another embedding provider) running with the configured model, embedding table populated via the text_embedding enhancement module, and the pgvector extension installed in PostgreSQL.

A configuration that enables semantic search on a database with no embedding table for its model is refused on every search, and the refusal names the command that creates the table; the MCP server, the web interface and osprey ariel status report it at start.

Module: search/qmd.py

Hybrid keyword-plus-semantic search. The default qmd strategy leaves lexical and vector retrieval inside qmd, preserving the behavior of existing deployments. The optional lightweight rrf strategy has qmd supply a BM25 ranking from its markdown mirror, ARIEL’s configured semantic search supply the dense ranking, and ARIEL combine their positions with equal-weight reciprocal-rank fusion (RRF, k=60). Both are useful when a question mixes specific terms with a described situation, or when keyword search returned too little.

The rrf path searches both qmd and PostgreSQL. It needs three things running together:

  1. the services.qmd sidecar (see The Search Sidecar (qmd)), and

  2. the qmd_export enhancement module, which writes the markdown mirror the sidecar indexes; and

  3. the same configured embedding model and populated pgvector table used by semantic search.

The sidecar and the export need each other: an export with no sidecar indexes nothing, and a sidecar with no export searches an empty corpus. The shipped demo template enables both, together with the sidecar itself. A web-terminal login deploys no sidecar of its own. It reaches the hosting deployment’s sidecar through services.qmd.port instead. The build copies that key from the hosting deployment’s render into every login, so nothing pins it by hand. osprey build refuses a render that keeps hybrid on with no sidecar to dial (Build Profiles).

hybrid also does not degrade the way semantic search does. A query against a sidecar that is not there is reported as search is down, deliberately, so that the agent cannot read an outage as “nothing matched”.

Configuration:

ariel:
  search_modules:
    hybrid:
      enabled: true
      settings:
        strategy: qmd        # default; or rrf
        rerank: true          # default
        candidate_limit: 40   # default

Warning

The knobs must stay under settings:. ARIEL’s search-config loader keeps only enabled, provider, model and settings, and drops every other key without a word. A knob written as a sibling of enabled is inert forever — no error, no warning, just the defaults.

Choose the strategy first. strategy: qmd is the compatible default and uses qmd’s lexical/vector retrieval; it may optionally use qmd’s LLM reranker. strategy: rrf is the lightweight option: it combines qmd BM25 with ARIEL’s configured semantic provider, has no second embedding model in the logbook retrieval path, and never calls the qmd reranker. It therefore requires semantic search and its embeddings to be available, and lets the ariel sidecar skip qmd’s own embedding pass (The Search Sidecar (qmd)).

The rerank decision. For strategy: qmd, rerank turns on qmd’s LLM reranker, which reorders candidates for quality. It is the single most important latency knob. For strategy: rrf it is ignored because RRF returns directly. Measured against a 134,996-entry logbook:

Corpus

p95, rerank: false

p95, rerank: true

135,000 entries

811 ms

3927 ms

2,000 entries

–

1587 ms

An LLM reviews every candidate before the results come back, which is what makes reranking the dominant term. Its cost is dominated by the candidate pool and grows far more slowly than the corpus does, so no logbook is small enough to outrun it. Both surfaces ship with the quality path on.

One key, both surfaces. For the qmd strategy, search_modules.hybrid.settings.rerank governs the agent-facing hybrid_search tool and the web interface’s search panel — and candidate_limit likewise. There is no separate panel key. When the deployment enables reranking, the panel toggle and tool argument may turn it off for an individual search. Setting the deployment key to false is a hard disable: the panel offers no toggle and a request cannot turn the reranker back on. The rrf strategy also omits that toggle because reranking does not participate. (The OKF bundle is a different key over a different corpus and defaults the other way; see The OKF Bundle.)

What the panel does with a reranked search. Rather than hold an empty screen for the length of a reranked query, the panel runs it in two phases: it fetches and paints the fast ranking first, under a status line saying the results are being reranked, then replaces them with the reranked ranking under a line saying they were updated. The second response is drawn from a larger candidate pool, so which entries come back can change, not only the order they come back in.

Reranking is never load-bearing. A reranked query that fails, for any reason, is retried once without the reranker; the results come back normally, carrying a WARNING diagnostic that says the ranking was not improved. The panel keeps the fast results on screen and says so in its status line. Only a failure of that retry is a failed search.

candidate_limit is how many BM25 and dense candidates RRF considers per lane, or qmd’s candidate pool under the qmd strategy. Lowering it trades recall for latency.

Filtering is best-effort. hybrid_search ranks the corpus first and applies the date, author and source filters afterwards, to the top of that ranking — not inside the database. A selective filter can therefore return fewer entries than you asked for even when more matching entries exist. Read a short result set as “the ranked window ran out”, not as “there is nothing else”. When a filter has to be exhaustive, use keyword_search or sql_query, which filter in the database.

The first reranked query after a sidecar start is slow

The first query with rerank: true loads a 610 MB model on CPU, and that load commonly runs past the client’s 30-second request timeout. That first query is not lost: it falls back to the non-reranked ordering with a warning, and every query after the model is resident is reranked normally. This is the ordinary first-run experience after starting or restarting the sidecar, not a fault to chase. Run one throwaway query when the sidecar comes up if you want the first real one to be fast.

Known limitation — entry IDs that are not numeric

qmd normalises the document paths it reports: _ and % both become -, runs collapse, and a leading one is dropped. ARIEL rehydrates each hit from the document’s title rather than the reported path, so the entries you get back are correct. But two entry IDs that differ only in characters qmd collapses — beam_current_setpoint and beam-current-setpoint, say — index as one document, and one of them becomes unreachable through this mode.

Over a real 134,996-entry logbook the measured collision rate is 0.0000%: every entry ID there is a 4-6 digit decimal string, so no real pair can collide. This matters only for a facility whose entry IDs are not numeric.

Registering a custom search module:

To add your own search module, create a Python module that exports get_tool_descriptor() (and optionally get_parameter_descriptors()), then register it through your application’s registry configuration:

from osprey.registry.helpers import extend_framework_registry
from osprey.registry.base import ArielSearchModuleRegistration

app_config = extend_framework_registry(
    ariel_search_modules=[
        ArielSearchModuleRegistration(
            name="my_search",
            module_path="my_app.search.my_module",
            description="Custom search module for my facility",
        ),
    ],
)

Once the module is registered, enable it under config: in profile.yml (ariel.search_modules.my_search.enabled: true). It is then routable by name, with no change to ARIELSearchService: through osprey ariel search --mode my_search and through the web interface’s capabilities API. Making it callable by the OSPREY agent additionally requires a matching ARIEL MCP tool (contributions welcome). The get_tool_descriptor() function must return a SearchToolDescriptor, whose search_mode field is simply the registered module name:

SearchToolDescriptor — a frozen dataclass whose key fields are execute (the async search function), format_result (formats results for agent consumption), and args_schema (a Pydantic model for input validation). See the class definition in the source for the full field list.

Modules may also export get_parameter_descriptors() to declare tunable parameters for the frontend capabilities API. Each ParameterDescriptor describes a single knob — its name, type, default, range, and UI grouping — so the web interface can render controls dynamically.

What a module returns. execute may return a plain list of entries, as it always could. When it has something to report alongside them, it may return a ModuleOutput instead:

  • diagnostics holds the notices the caller should see (a truncated query, a skipped step).

  • expansion holds the vocabulary groups that actually reached the query it ran.

  • total, total_exact and has_more report counts and paging; tier names the step that answered; filter_applied says the module applied the entry filter itself. All are optional.

A bare list stays valid and needs no change.

Opting into the shared query work. Descriptor fields, all off by default, let a module receive work the service has already done. Declare none and the module is called as before, except for paging (see accepts_offset).

  • accepts_expansion — set it to True and the service passes query_expansion= when, and only when, expansion was resolved for that request. The module never sees the expand_query request argument itself: whether to expand is settled once, in the service.

  • accepts_entry_filter — set it to True and the service passes entry_filter= (an EntryFilter, possibly empty) for the module to apply. The service post-filters every result with EntryFilter.matches either way.

  • accepts_offset — set it to True and the service passes offset=, with max_results as the page size. Otherwise the module gets no offset and is asked for min(offset + max_results * 2, 200) results. The service then cuts the page out of them.

  • needs_search_calls — set it to True and the service passes its search-call pool as search_calls=. A module that calls a provider at search time runs that call on the pool it receives, with await search_calls.run(fn, *args, timeout_s=...), and never builds a pool of its own. The service closes the pool when it stops.

  • query_parser — an optional callable that takes the raw query string and returns a ParsedKeywordQuery (search text, field filters, phrases, pattern spans, diagnostics). The service calls it once and hands the result to the module as parsed=. Keyword search declares its own parse_keyword_query here; a module that wants a different query language supplies its own parser, and one that wants none is never given ARIEL’s.

Collaboration Welcome

If you implement a search module that could benefit other facilities — for example, a structured-metadata search, a time-series correlation search, or a cross-entry linking search — we encourage you to open a pull request so it becomes natively available in Osprey.

Choosing Between Semantic and Hybrid#

Both modes retrieve by meaning rather than by matching words, and a deployment can run either, both, or neither. They differ in what they depend on, what they cost per query, and how much of their ranking you can inspect.

hybrid defaults to qmd’s all-in-one retrieval, which combines BM25, vector search and an optional LLM reranker without needing an embedding provider on the host or pgvector in the logbook database. Deployments that already run semantic search can instead select the lightweight RRF strategy, which reuses that same embedding model and combines it with qmd BM25. The shipped templates set hybrid as default_search_mode.

semantic is worth keeping — or choosing — when any of the following applies:

  • You want a stronger embedding model than the sidecar bakes in. semantic takes its embeddings from a configured provider. A facility with its own inference endpoint can therefore point it at a far larger model than the 300M embedder qmd ships.

  • You need the ranking to be inspectable. semantic is cosine distance over a pgvector column and nothing else — no reranker. It does apply deterministic vocabulary expansion when one is configured, and reports every term it added; drop semantic from ariel.vocabulary.expand_modes for the pure path. Any result can then be explained with a single SQL query, which matters where retrieval has to be auditable.

  • You want it composable with structured filters. The embeddings live in text_embeddings_* tables in the logbook database. They therefore join against ordinary columns and are reachable from the sql_query tool. hybrid ranks a markdown mirror and post-filters instead.

  • Query latency matters more than reranking quality. Reranking dominates a qmd hybrid query’s cost; a pure vector lookup is the fast path. The RRF strategy is a lightweight hybrid alternative when both lexical and dense retrieval are wanted without a reranker.

Running both is a reasonable configuration: they are independent modules, and default_search_mode decides only which one answers when the caller names no mode.

Need behavior beyond these search modules — multi-step reasoning, answer synthesis, custom prompting? That lives in the OSPREY agent layer; see ARIEL Contracts under “Extending the integration.”

Read-only role for the SQL tool#

The agent can run raw SQL against the logbook database. That query path connects as its own Postgres role — <username>_ro, derived from services.postgresql.username — which can log in, read the tables in the public schema, and do nothing else. It is not the role ingestion writes with, and it is not a superuser. A query naming a server-side function such as pg_read_file() is therefore refused by Postgres rather than by a pattern match over the query text. Only the agent’s ARIEL server opens that role; ingestion and the other osprey ariel commands connect as the owner role alone.

The role is created by an init script the Postgres entrypoint runs once, while it initializes a fresh data volume, the same window POSTGRES_PASSWORD is read in. Its password is ARIEL_DB_READONLY_PASSWORD in the deployment’s .env, minted by osprey up beside ARIEL_DB_PASSWORD. A new deployment gets all of this with no action from you.

Tables created later are covered. The script grants SELECT on the schema’s tables and sets default privileges for the owner. The enhanced_entries table the migrations create and the per-model text_embeddings_* tables an enhancement module adds are therefore readable without a second grant.

A deployment that names its own database with the uri key gets no <username>_ro role. Name a SELECT-only login on that database in ariel.database.readonly_uri; without it the SQL tool is not offered (see ariel.database.readonly_uri — the SQL tool’s login on your own database).

Adopting the role on an existing database#

A data volume created before the role existed never runs the init script, so it has no <username>_ro. The agent logs one warning at start-up, and the SQL tool refuses every query, naming the refused login, until the role exists. The other ARIEL tools keep working. To adopt the role without recreating the volume, run the same script by hand once:

docker exec -e ARIEL_DB_READONLY_PASSWORD="$(grep '^ARIEL_DB_READONLY_PASSWORD=' .env | cut -d= -f2-)" \
    <project>-ariel-postgres bash /docker-entrypoint-initdb.d/10-readonly-role.sh

Restart the agent afterwards so it picks up the role and the SQL tool answers. The script is idempotent — running it again is how you roll the role’s password after changing it in .env.

See Also#

Data Ingestion

How data gets into the system — facility adapters, enhancement modules, and database schema

ARIEL Contracts

MCP tools, service factory, and search result structure

Web Interface

Web interface architecture and capabilities API

Retrieval Paths

How these modes relate to OSPREY’s other retrieval stacks, and which of them need an embedding provider reachable at query time