Part 2: Building Your Channel Finder#

Step 3: Semantic Channel Finding Pipelines#

Academic Reference

For a comprehensive theoretical framework and analysis of semantic channel finding in complex experimental infrastructure, see Hellert et al. (2025), “From Natural Language to Control Signals: A Conceptual Framework for Semantic Channel Finding in Complex Experimental Infrastructure,” available on arXiv:2512.18779.

The Core Challenge: Bridging Human Language and Control System Addresses

Control systems at scientific facilities present a fundamental communication gap: operators and physicists think in terms of physical concepts (“beam current,” “terminal voltage,” “ion pump pressure”), while control systems use technical addresses (SR01C___DCCT1_AM00, TMVST, IP41Pressure). This gap becomes critical in large facilities with thousands to hundreds of thousands of channels—manually looking up addresses is impractical, and exact string matching fails because users don’t know the precise naming conventions.

Semantic channel finding solves this by using LLM-powered pipelines to translate natural language queries into specific control system addresses. Instead of requiring users to memorize technical names or browse through documentation, they can ask for what they need in their own words. The system understands the semantic meaning and finds the correct channels, even when query terminology doesn’t exactly match the database.

Why This Matters for Control Systems:

Accessibility: Non-experts can interact with complex systems without extensive training on naming conventions
Efficiency: Eliminates time spent searching documentation or asking colleagues for channel names
Reliability: Reduces errors from mistyped or misremembered addresses
Scalability: Makes systems with thousands of channels as easy to use as small systems
Integration: Enables natural language interfaces for automated procedures and diagnostics

Three Pipeline Implementations

This template provides three semantic channel finding implementations, each suited to different facility architectures:

In-Context Search: Direct semantic matching—best for small to medium systems (few hundred channels)
Hierarchical Navigation: Structured navigation through system hierarchy—scales to large systems with strict naming patterns (thousands+ channels)
Middle Layer Exploration: React agent with database query tools—scales to large systems organized by function rather than naming patterns (thousands+ channels)

All three pipelines share the same interface and capabilities, differing only in their matching strategy and database organization. Choose based on your system architecture, or try multiple and benchmark which performs better for your facility.

Switching Between Pipelines

To switch pipelines, edit config.yml and change the pipeline_mode setting:

channel_finder:
  pipeline_mode: in_context  # or "hierarchical" or "middle_layer"

That’s it—no code changes required. The template includes complete implementations of all three pipelines with example databases, CLI tools, and benchmark datasets. The tabs below detail each pipeline’s workflow, database format, and testing tools.

Customize prompts for your facility

After building your database, edit facility_description.py with your facility’s systems and terminology to dramatically improve matching accuracy. See Channel Finder Prompt Customization in Part 4 for step-by-step guidance.

In-Context Pipeline

Concept: Put the entire channel database in the LLM context and ask it to find matches.

Best for: Small to medium systems (few hundred channels), facilities with flat channel lists, rapid prototyping when database structure is still evolving.

Pipeline Flow:

Query splitting
Semantic matching against full database
Validation/correction (iterative)

Channel Names vs Addresses: Optimizing LLM Token Matching

The Challenge: Control systems use cryptic addresses (TMVST, Acc_I, IP41Pressure), but users query with natural language (“terminal voltage setpoint”, “grading resistor current”). The LLM needs to bridge this gap during semantic matching.

The Solution: When building your channel database, you choose how to create the searchable channel names:

# Default: use raw addresses as channel names
python build_channel_database.py

# LLM-powered: generate descriptive channel names (recommended)
python build_channel_database.py --use-llm --config config.yml

How It Works:

Your database stores three fields, but the LLM only sees two during semantic matching:

The LLM sees only: channel + description (formatted as "ChannelName: Description")

Why Descriptive Names Dramatically Improve Matching:

When a user queries “terminal voltage setpoint”:

Default mode (TMVST): LLM matches on description alone → returns cryptic token TMVST
LLM-powered mode (TerminalVoltageSetPoint): LLM matches on both name and description → returns semantically aligned token TerminalVoltageSetPoint

The descriptive channel name reinforces the semantic signal from the description, creating stronger, more consistent matches. The channel name becomes a searchable index aligned with how users think and query.

What the ``–use-llm`` flag does:

Extracts semantic tokens from descriptions (location, device type, property)
Generates unique PascalCase names that align with description content
Preserves original addresses for control system connections

Example results:

Channel (shown to LLM, searchable)	Address (for EPICS/LabView/etc.)
`TerminalVoltageSetPoint`	`TMVST`
`AcceleratingTubeEndIonPumpPressure`	`IP41Pressure`

Note: You can manually edit channel names in the database JSON file at any time to customize them for your facility’s terminology and conventions.

In-Context Channel Finder CLI in action — **Live Example**: In-context pipeline processing “What are the grading resistor currents and the terminal voltage?” — Shows query splitting into 2 atomic queries, full database mode with 255 channels, matching 4 results (2 grading resistor currents + 2 terminal voltages), and validation confirming all channels are valid.#

Try It Now: Interactive Channel Finder

The template includes a working example database (UCSB FEL accelerator, 255 channels). Try the channel finder immediately—no database setup required:

# From my-control-assistant directory
cd my-control-assistant

# Switch to in-context pipeline
# python config.yml: set pipeline_mode to "in_context"

python src/my_control_assistant/services/channel_finder/cli.py

Example queries to try:

🔍 Query: What is the terminal voltage?
🔍 Query: Get me the grading resistor currents and beam current

How It Works:

Pipeline Stages

The in-context pipeline uses a three-stage process to translate natural language into channel addresses.

Stage 1: Query Splitting

Complex queries are broken into atomic sub-queries to improve matching accuracy. For example, “grading resistor currents and terminal voltage” becomes two separate searches.

The LLM analyzes the query structure and identifies distinct intents. Single-intent queries pass through unchanged, while multi-intent queries are split for independent processing.

Stage 2: Semantic Matching

Each atomic query is matched against the channel database using semantic similarity. The pipeline operates in two modes:

Full database mode (default, recommended for <200 channels): The entire database is presented to the LLM in a single context window. Fast and simple, but limited by model context size. Controlled by chunk_dictionary: false in config.yml.
Chunked mode (for larger databases): The database is split into manageable chunks (default 50 channels per chunk, configurable via chunk_size in config.yml), and each chunk is searched independently. Prevents context overflow but may miss cross-chunk relationships. Enable with chunk_dictionary: true.

The LLM receives the formatted database and atomic query, then identifies all matching channels based on semantic meaning rather than exact string matching. This allows queries like “beam current” to match channels with descriptions containing corresponding concepts.

Stage 3: Validation & Correction

All matched channels are validated against the database to ensure they actually exist. This catches hallucinations or malformed channel names.

If invalid channels are detected, the pipeline enters a correction loop (maximum 2 iterations):

The LLM is presented with the validation results (which channels are valid/invalid)
It re-examines the database chunk with the original query
It either corrects the invalid channels or removes them
The corrected channels are re-validated

Only validated channels appear in the final result. This ensures the assistant never provides non-existent channel addresses to users or downstream capabilities.

Database Format

Simple Flat Structure - Plug and Play:

The in-context database uses a flat list where every channel has rich natural language descriptions for semantic matching:

{
  "channels": [
    {
      "template": false,
      "channel": "TerminalVoltageReadBack",
      "address": "TerminalVoltageReadBack",
      "description": "Actual value of the terminal potential where electron gun and collector are located measured capacitively"
    },
    {
      "template": false,
      "channel": "AcceleratingTubeGradingResistorDiagnosticCurrent",
      "address": "Acc_I",
      "description": "Diagnostic current through grading resistor chain in accelerating tube"
    },
    {
      "template": true,
      "base_name": "BPM",
      "instances": [1, 10],
      "sub_channels": ["XPosition", "YPosition"],
      "address_pattern": "BPM{instance:02d}{suffix}",
      "description": "Beam Position Monitors measure electron beam location",
      "channel_descriptions": {
        "XPosition": "Horizontal position from BPM {instance:02d} in millimeters",
        "YPosition": "Vertical position from BPM {instance:02d} in millimeters"
      }
    }
  ]
}

Key features:

Flat structure: All channels at the same level, no nested hierarchy
Rich descriptions: Natural language descriptions enable semantic matching
Template support: Device families (like BPM01-BPM10) defined once and expanded automatically
Separate naming: channel (searchable name shown to LLM) vs. address (actual PV address, not shown to LLM)
LLM sees: Only channel + description during semantic matching

Minimal setup: Just list your channels with descriptions. The pipeline handles the rest.

Build Your Database

From CSV to Template Database

The in-context pipeline provides tools to build template-based databases from simple CSV files. This workflow streamlines database creation, especially for facilities with device families.

Workflow Overview:

CSV File              →  build_channel_database.py  →  Template JSON Database
(raw data)               (with optional LLM naming)     (optimized format)
                                 ↓
                         validate_database.py
                         preview_database.py
                         (verify structure & presentation)

Step 1: Prepare Your CSV File

For CSV format reference, see data/raw/CSV_EXAMPLE.csv which includes inline documentation of all supported patterns. The template also includes data/raw/address_list.csv with real UCSB FEL channel data (53 standalone channels + 5 device families = 255 total expanded channels).

Create a CSV with your channel data (typically in src/my_control_assistant/data/raw/) in this format:

address,description,family_name,instances,sub_channel
TerminalVoltageReadBack,Actual value of terminal potential measured capacitively,,,
Acc_I,Diagnostic current through grading resistor chain in accelerating tube (monitors voltage distribution health),,,
DipoleMagnet{instance:02d}{sub_channel},Dipole magnets for beam bending - set point for magnet {instance},DipoleMagnet,9,SetPoint
DipoleMagnet{instance:02d}{sub_channel},Dipole magnets for beam bending - readback for magnet {instance},DipoleMagnet,9,ReadBack
SteeringCoil{instance:02d}{sub_channel},Steering coils for trajectory corrections - X direction setpoint,SteeringCoil,19,XSetPoint
SteeringCoil{instance:02d}{sub_channel},Steering coils for trajectory corrections - X direction readback,SteeringCoil,19,XReadBack

Column Definitions:

address: The actual PV address or pattern (required)
description: Natural language description for semantic matching (required)
family_name: Device family identifier for template grouping (optional)
instances: Number of device instances in the family (optional, for templates)
sub_channel: Sub-channel type (e.g., SetPoint, ReadBack) (optional, for templates)

Template Detection Logic:

Rows with family_name: Grouped into template entries (efficient for device families)
Rows without family_name: Standalone channel entries (one-off channels)

Step 2: Run the Database Builder

The builder tool processes your CSV and generates an optimized template-based database:

cd my-control-assistant

# Basic build (uses addresses as channel names)
python src/my_control_assistant/data/tools/build_channel_database.py

# With LLM-generated descriptive names (recommended)
python src/my_control_assistant/data/tools/build_channel_database.py --use-llm --config config.yml

The template includes working UCSB FEL accelerator data by default:

Format Reference: data/raw/CSV_EXAMPLE.csv (documented examples of all patterns)
Working Example: data/raw/address_list.csv (real UCSB FEL channels)
Default Output: data/channel_databases/in_context.json (pre-built database)

For your facility’s data, specify custom paths:

python src/my_control_assistant/data/tools/build_channel_database.py \
   --csv data/raw/your_address_list.csv \
   --output data/channel_databases/your_database.json \
   --use-llm --config config.yml

What the Builder Does:

Loads CSV: Reads your channel data, skipping comments and empty rows
Groups by Family: Identifies device families from family_name column
Extracts Common Descriptions: Automatically finds common description parts for template families
Creates Templates: Builds template-based entries with instance ranges
Generates Names (with --use-llm): Creates descriptive PascalCase names for standalone channels
Resolves Duplicates: Ensures all generated names are unique with location specificity
Adds Metadata: Includes generation date, stats, and tool provenance

LLM Name Generation Example:

The database stores three fields per channel, but the in-context pipeline only presents channel and description to the LLM during semantic matching. The address is preserved for control system connections but never shown to the LLM.

Without LLM (uses raw addresses):

{
  "channel": "Acc_I",
  "address": "Acc_I",
  "description": "Diagnostic current through grading resistor chain in accelerating tube"
}

LLM sees: "Acc_I: Diagnostic current through grading resistor chain in accelerating tube"

With LLM (descriptive, searchable names):

{
  "channel": "AcceleratingTubeGradingResistorDiagnosticCurrent",
  "address": "Acc_I",
  "description": "Diagnostic current through grading resistor chain in accelerating tube"
}

LLM sees: "AcceleratingTubeGradingResistorDiagnosticCurrent: Diagnostic current through grading resistor chain in accelerating tube"

The LLM uses facility-specific prompts to generate names that:

Use PascalCase format for consistency
Include location/position details (e.g., “AcceleratingTube”, “Top”, “Beginning”)
Use full words instead of abbreviations (e.g., “SetPoint” not “Set”)
Are unique across the entire database
Are self-documenting and semantic

Output Structure:

The builder creates a template-based JSON database with metadata:

{
  "_metadata": {
    "generated_from": "data/raw/address_list.csv",
    "generation_date": "2025-11-08",
    "generator": "build_channel_database.py",
    "llm_naming": {
      "enabled": true,
      "model": "anthropic/claude-haiku",
      "purpose": "Generated descriptive PascalCase names for 53 standalone channels"
    },
    "stats": {
      "template_entries": 5,
      "standalone_entries": 53,
      "total_entries": 58
    }
  },
  "channels": [
    { "template": false, "channel": "TerminalVoltageReadBack", ... },
    { "template": true, "base_name": "DipoleMagnet", "instances": [1, 9], ... }
  ]
}

Step 3: Validate Your Database

Check for structural issues and verify the database can be loaded:

# From my-control-assistant directory
python src/my_control_assistant/data/tools/validate_database.py

# Validate specific file
python src/my_control_assistant/data/tools/validate_database.py --database path/to/database.json

# Show detailed statistics
python src/my_control_assistant/data/tools/validate_database.py --verbose

Validation Checks:

Structure: Valid JSON format, correct schema, required fields
Templates: Instance ranges [start, end], sub-channels, patterns
Standalone: All required fields (channel, address, description)
Database Loading: Can be loaded by TemplateChannelDatabase class
Statistics: Channel counts, compression ratio, entry types

Step 4: Preview Database Presentation

See exactly how the database will appear to the LLM:

# From my-control-assistant directory
python src/my_control_assistant/data/tools/preview_database.py

# Show all channels (not just first 20)
python src/my_control_assistant/data/tools/preview_database.py --full

This prints the database presentation directly to your terminal, showing:

Database configuration and statistics
First 20 channels in LLM presentation format (or all with --full)
Template and standalone entry breakdown

Preview Output Example:

Configuration:

The build tool uses command-line arguments to specify input/output paths. The LLM configuration for name generation can optionally be added to your config.yml:

channel_finder:
  # LLM configuration for name generation (optional)
  channel_name_generation:
    llm_batch_size: 10
    llm_model:
      provider: cborg
      model_id: anthropic/claude-haiku
      max_tokens: 2000

The database path is configured in the pipeline settings:

channel_finder:
  pipeline_mode: in_context  # Active pipeline: "in_context" or "hierarchical"

  pipelines:
    in_context:
      database:
        type: template
        path: src/my_control_assistant/data/channel_databases/in_context.json
        presentation_mode: template

Tool Locations:

All tools are in src/my_control_assistant/data/tools/ and should be run from the my-control-assistant directory (for config.yml access):

build_channel_database.py: Main builder (Workflow A: CSV → JSON)
validate_database.py: Schema and loading validation
preview_database.py: LLM presentation preview
llm_channel_namer.py: LLM-based name generation library
README.md: Comprehensive tool documentation

Alternative: Manual JSON Creation (Workflow B)

For complete control, you can create the JSON database manually:

Create JSON file directly with template and standalone entries
Run validate_database.py to check correctness
Update config.yml with new database path
Proceed to testing (next tab)

Both workflows are supported and documented in the tools README.

Test Your Database

Interactive Testing & Benchmarking

Once your database is built and validated, test its functionality with the CLI for rapid iteration, then benchmark it for statistical evaluation.

Interactive CLI: Rapid Development Testing

Query your database interactively without running the full agent:

# From my-control-assistant directory
python src/my_control_assistant/services/channel_finder/cli.py

This launches an interactive terminal where you can:

Test queries against your channel database in real-time
Watch the pipeline stages execute with live progress updates
Verify which channels are matched and why
Measure query execution time to optimize performance
Iterate on your database and see results immediately

Example queries to try:

“What’s the terminal voltage?”
“Show me all dipole magnets”
“Find steering coil setpoints”

The CLI operates independently of the agent framework, making it ideal for rapid development cycles. You can quickly iterate on your channel descriptions and immediately test the results.

Benchmarking: Critical Validation Before Production

Benchmark datasets are essential for validating your channel finder before deployment. A benchmark that performs well during development with developer-created queries may completely fail in production when users with different mental models start querying the system.

Once your database is ready and you have a diverse benchmark dataset from domain experts, run systematic benchmarks to measure accuracy:

# Run all benchmark queries
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py

# Test specific queries (useful during development)
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --queries 0,1

# Compare model performance
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --model anthropic/claude-sonnet

Benchmark Capabilities:

Statistical metrics: Precision (accuracy of matches), recall (completeness), and F1 scores (harmonic mean)
Success categorization: Perfect matches (100% correct), partial matches (some correct), and failures
Parallel execution: Runs 5 queries concurrently by default to speed up large benchmark sets
Incremental saves: Results are saved after each query, so interruptions don’t lose progress
Consistency tracking: Measures whether repeated queries produce the same results

Example benchmark results:

╭───────────────────── Overall Metrics ─────────────────────╮
│   Queries Evaluated    30/30                               │
│   Precision            0.949                               │
│   Recall               0.950                               │
│   F1 Score             0.943                               │
╰────────────────────────────────────────────────────────────╯

╭───────────────────── Success Breakdown ────────────────────╮
│   ✓ Perfect Matches    27    90.0%                         │
│   ⚠ Partial Matches     2    6.7%                          │
│   ✗ No Matches          1    3.3%                          │
╰────────────────────────────────────────────────────────────╯

╭──────────────────────── Performance ───────────────────────╮
│   Avg Consistency    1.000                                 │
│   Avg Time/Query     2.343s                                │
╰────────────────────────────────────────────────────────────╯

Configuring Benchmarks:

The benchmark system is configured in config.yml with both global settings and pipeline-specific datasets:

channel_finder:
  # Active pipeline determines which benchmark dataset is used by default
  pipeline_mode: in_context

  pipelines:
    in_context:
      database:
        type: template
        path: src/my_control_assistant/data/channel_databases/in_context.json
        presentation_mode: template

      processing:
        chunk_dictionary: false
        chunk_size: 50
        max_correction_iterations: 2

  # Benchmark dataset for this pipeline
  benchmark:
    dataset_path: src/my_control_assistant/data/benchmarks/datasets/in_context_benchmark.json
     # In-context pipeline benchmark using UCSB FEL channels (30 queries)

# Global benchmarking configuration
benchmark:
  # Execution settings
  execution:
    runs_per_query: 1                 # Number of times to run each query
    delay_between_runs: 0             # Delay in seconds between runs
    continue_on_error: true           # Continue even if some queries fail
    max_concurrent_queries: 5         # Maximum parallel queries
    query_selection: all              # "all" or specific queries like [0,1,2]

  # Output settings
  output:
    results_dir: src/my_control_assistant/data/benchmarks/results
    save_incremental: true            # Save results after each query
    include_detailed_metrics: true    # Include detailed timing/cost metrics

  # Evaluation thresholds
  evaluation:
    exact_match_weight: 1.0
    partial_match_weight: 0.5

You can override the dataset and other settings via CLI:

# Use a different dataset
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py \
   --dataset src/my_control_assistant/data/benchmarks/datasets/custom_dataset.json

# Run specific queries
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --queries 0,1,5,10

# Show detailed logs
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --verbose

When to use these tools:

During development: Use the CLI to rapidly test queries as you build your database
Before deployment: Run full benchmarks with domain expert queries to validate accuracy meets production requirements (aim for >90% F1 score)
Model selection: Compare different LLM models to balance cost, speed, and accuracy for your specific use case
Continuous validation: Re-run benchmarks after database updates to ensure no regression in accuracy

Best Practices:

Rich Descriptions: Write detailed, semantic descriptions for better LLM matching
Consistent Naming: Use consistent patterns within device families
Family Grouping: Group related channels into templates for efficiency
LLM Naming: Use --use-llm for standalone channels to get searchable names
Validate Early: Run validation after building to catch structural issues
Preview Before Testing: Check LLM presentation to ensure clarity
Iterate with CLI: Test queries interactively before running full benchmarks
Benchmark Systematically: Run full benchmark suite before production deployment

See also

Next Step: Customize Prompts — Edit facility_description.py with your facility’s terminology to improve matching accuracy. See Channel Finder Prompt Customization.

Hierarchical Pipeline

Concept: Navigate through a structured hierarchy of systems → families → devices → fields.

Best for: Large systems (thousands+ channels), facilities with well-defined hierarchical organization, systems where context window size becomes a constraint.

Hierarchical Channel Finder CLI in action — **Live Example**: Hierarchical pipeline processing “whats the beam current?” through 5 navigation levels (system → family → device → field → subfield) to find `DIAG:DCCT[MAIN]:CURRENT:RB` from 1,050 channels. Shows the recursive branching at each level.#

Try It Now: Interactive Channel Finder

The template includes a working example database (accelerator with magnets, vacuum, RF, and diagnostics—1,050 channels). Try the hierarchical channel finder immediately:

# From my-control-assistant directory
cd my-control-assistant

# Switch to hierarchical pipeline
# Edit config.yml: set pipeline_mode to "hierarchical"

python src/my_control_assistant/services/channel_finder/cli.py

Example queries to try:

🔍 Query: Whats the beam current?
🔍 Query: Get me all focusing quadrupole currents

Database Format (v0.9.4+)

Hierarchical databases use a flexible schema that lets you define any hierarchy structure to match your control system. See the Database Format tab below for schema details and Building Your Own Database for a step-by-step guide.

How It Works:

Middle Layer Pipeline

Concept: Agent explores database using query tools to find channels by function.

Best for: Large systems organized by function (Monitor, Setpoint) rather than naming patterns, facilities using MATLAB Middle Layer (MML) style organization, systems requiring device/sector filtering.

Middle Layer Channel Finder CLI in action — **Live Example**: Middle layer pipeline processing “What’s the beam current?” using React agent with database query tools. Shows the agent calling list_systems(), list_families(), inspect_fields(), and list_channel_names() to explore the functional hierarchy (SR → DCCT → Monitor) and find `SR:DCCT:Current`.#

Try It Now: Interactive Channel Finder

The template includes a working example database (accelerator with Storage Ring, Booster, and Transfer Line).

# From my-control-assistant directory
cd my-control-assistant

# Switch to middle_layer pipeline
# Edit config.yml: set pipeline_mode to "middle_layer"

python src/my_control_assistant/services/channel_finder/cli.py

Example queries to try:

🔍 Query: What's the beam current?
🔍 Query: Get me BPM X positions in sector 1

How It Works:

Pipeline Stages

The middle layer pipeline uses a React agent with database query tools to find channels through functional exploration.

Stage 1: Query Splitting

Identical to the other pipelines’ query splitting stage, complex queries are decomposed into atomic sub-queries. Each atomic query then explores the database independently using the agent.

Stage 2: Agent-Based Database Exploration

The pipeline uses a LangGraph React agent with five database query tools. The agent autonomously explores the functional hierarchy (System → Family → Field → Subfield) to find matching channels.

Available Tools:

list_systems() - Get all available systems with descriptions
list_families(system) - Get device families within a system
inspect_fields(system, family, field) - Examine field structure and subfields
list_channel_names(…) - Retrieve actual PV addresses with optional filtering
get_common_names(system, family) - Get friendly device names

The agent follows this typical workflow:

Query: “What’s the beam current?”

Explore systems → Calls list_systems() → Finds SR, VAC, BR, BTS
Select relevant system → Chooses SR (Storage Ring) based on description
Explore families → Calls list_families('SR') → Finds BPM, HCM, DCCT, etc.
Identify beam current family → Chooses DCCT (DC Current Transformer)
Inspect fields → Calls inspect_fields('SR', 'DCCT') → Finds Monitor, FastMonitor, Lifetime
Retrieve channels → Calls list_channel_names('SR', 'DCCT', 'Monitor')
Report results → Returns SR:DCCT:Current

Key Features:

Autonomous exploration: Agent decides which tools to call and in what order
Description-aware: Uses optional _description fields when available for better matching
Subfield navigation: Automatically discovers nested structures (e.g., X/Y under BPM)
Device filtering: Supports optional sector/device filtering when DeviceList metadata present

Stage 3: Result Aggregation & Deduplication

Results from all atomic queries are combined and deduplicated to provide a clean final result.

Database Format

Functional Hierarchy Structure:

The middle layer database organizes channels by function rather than naming pattern, following the MATLAB Middle Layer (MML) convention used at accelerator facilities:

{
  "SR": {
    "_description": "Storage Ring: Main synchrotron light source...",
    "BPM": {
      "_description": "Beam Position Monitors: Non-invasive electrostatic pickups...",
      "X": {
        "_description": "Horizontal position readback in millimeters.",
        "ChannelNames": ["SR01C:BPM1:X", "SR01C:BPM2:X", "SR01C:BPM3:X", ...],
        "DataType": "Scalar",
        "Mode": "Online",
        "Units": "Hardware",
        "HWUnits": "mm",
        "PhysicsUnits": "Meters",
        "MemberOf": ["BPM", "Monitor", "PlotFamily", "Save", "Archive"]
      },
      "Y": {
        "_description": "Vertical position readback in millimeters.",
        "ChannelNames": ["SR01C:BPM1:Y", "SR01C:BPM2:Y", "SR01C:BPM3:Y", ...]
      },
      "setup": {
        "CommonNames": ["BPM 1-1", "BPM 1-2", "BPM 1-3", ...],
        "DeviceList": [[1, 1], [1, 2], [1, 3], ...]
      }
    },
    "HCM": {
      "_description": "Horizontal Corrector Magnets...",
      "Monitor": {
        "_description": "Current readback in Amperes.",
        "ChannelNames": ["SR01C:HCM1:Current", ...]
      },
      "Setpoint": {
        "_description": "Current setpoint in Amperes.",
        "ChannelNames": ["SR01C:HCM1:SetCurrent", ...]
      },
      "OnControl": {
        "_description": "Enable/disable control...",
        "ChannelNames": ["SR01C:HCM1:Enable", ...]
      }
    },
    "DCCT": {
      "_description": "DC Current Transformer: Measures beam current...",
      "Monitor": {
        "_description": "Beam current in milliamperes.",
        "ChannelNames": ["SR:DCCT:Current"]
      }
    }
  },
  "VAC": {
    "_description": "Vacuum System...",
    "IonPump": {
      "_description": "Ion Pumps for UHV pumping...",
      "Pressure": {
        "_description": "Vacuum Pressure readback (Torr).",
        "ChannelNames": ["SR01C:VAC:IP1:Pressure", ...]
      }
    }
  }
}

Key features:

Functional organization: Systems → Families → Fields → ChannelNames
Retrieved addresses: PV addresses stored in database (not built from patterns)
Optional descriptions: _description fields at all levels provide semantic context
Optional metadata: Preserve MML metadata (DataType, Mode, Units, MemberOf, etc.)
Device filtering: Optional DeviceList enables sector/device number filtering
Common names: Optional CommonNames for friendly device identifiers
Nested subfields: Fields can have subfields (e.g., RF PowerMonitor has Forward/Reflected)

Hierarchy Levels:

System: Top-level accelerator systems (SR=Storage Ring, BR=Booster, BTS=Transfer Line, VAC=Vacuum)
Family: Device families within systems (BPM, HCM, VCM, DCCT, IonPump, etc.)
Field: Functional categories (Monitor, Setpoint, X, Y, Pressure, Voltage, etc.)
Subfield (optional): Nested functional organization (e.g., PowerMonitor → Forward/Reflected)
ChannelNames: List of actual EPICS PV addresses

Build Your Database

From MATLAB Middle Layer Exports

If your facility already uses MATLAB Middle Layer (MML), you can convert existing MML data structures directly to the middle layer JSON format using the included converter utility.

Workflow Overview:

MML Python Export  →  mml_converter.py  →  Middle Layer JSON Database
(MML_ao_SR.py)        (conversion tool)     (optimized format)
                            ↓
                    validate_database.py
                    preview_database.py
                    (verify structure & presentation)

Step 1: Export Your MML Data

If you have MATLAB Middle Layer data, export the ao (accelerator object) structure to Python format. The structure typically contains system/family hierarchies with channel information.

Step 2: Use the MML Converter

The template includes a conversion utility that transforms MML Python exports into middle layer JSON format:

cd my-control-assistant

# Convert MML exports to JSON
python src/my_control_assistant/services/channel_finder/utils/mml_converter.py \
   --input path/to/mml_exports.py \
   --output src/my_control_assistant/data/channel_databases/middle_layer.json

Using the Converter Programmatically:

from mml_converter import MMLConverter

# Import your MML exports
from my_facility_mml import MML_ao_SR, MML_ao_BR, MML_ao_VAC

# Create converter and add systems
converter = MMLConverter()
converter.add_system("SR", MML_ao_SR)
converter.add_system("BR", MML_ao_BR)
converter.add_system("VAC", MML_ao_VAC)

# Save to JSON
converter.save_json("my_facility.json")

What the Converter Does:

Extracts hierarchy: Preserves System → Family → Field → ChannelNames structure
Preserves metadata: Keeps DataType, Mode, Units, HWUnits, PhysicsUnits, MemberOf
Handles subfields: Detects nested field structures automatically
Includes setup data: Transfers CommonNames and DeviceList for filtering
Validates structure: Ensures all required fields present
Reports statistics: Shows channel counts and organization

Collaboration Welcome

The MML converter was developed and tested with the Advanced Light Source (ALS) MATLAB Middle Layer format. If your facility uses MATLAB Middle Layer or another middle layer solution with a different structure or export format, we’d love to add native support for it! Please open an issue on GitHub describing your middle layer system and we can work together to implement a converter for your facility’s format.

Alternative: Manual JSON Creation

For facilities not using MML, create the JSON database manually following the structure shown in the “Database Format” tab:

Define systems: Top-level dictionary with system keys (SR, BR, VAC, etc.)
Add families: Device families within each system
Specify fields: Functional categories with ChannelNames lists
Add descriptions (optional): Include _description fields for better semantic matching
Include metadata (optional): Add setup information for filtering

Step 3: Validate Your Database

Check for structural issues and verify the database can be loaded:

# From my-control-assistant directory
python src/my_control_assistant/data/tools/validate_database.py \
   --database src/my_control_assistant/data/channel_databases/middle_layer.json

# Show detailed statistics
python src/my_control_assistant/data/tools/validate_database.py --verbose

Validation Checks:

Structure: Valid JSON format, correct hierarchy levels
ChannelNames: All terminal fields have ChannelNames lists
Metadata: Optional fields properly formatted
Database Loading: Can be loaded by MiddleLayerDatabase class
Statistics: System/family/channel counts

Step 4: Preview Database Presentation

See how your database will be presented to the LLM agent during exploration:

# From my-control-assistant directory
python src/my_control_assistant/data/tools/preview_database.py

This shows you what information the agent sees at each level of exploration, helping you verify descriptions are clear and helpful.

Preview Output Example:

Best Practices:

Rich Descriptions: Write clear descriptions at all levels—these are the primary guide for the LLM agent
Functional Organization: Group channels by what they do (Monitor, Setpoint) not by location
Complete Metadata: Include Units, DataType, and other metadata from your control system
Device Lists: Add DeviceList metadata if you need sector/device filtering
Common Names: Include friendly device names for better user experience
Consistent Structure: Use same field names across families (e.g., always “Monitor” not “Readback”)
Test Incrementally: Build one system at a time, test it, then add the next

Test Your Database

Interactive Testing & Benchmarking

Once your database is built and validated, test its functionality with the CLI for rapid iteration, then benchmark it for statistical evaluation.

Interactive CLI: Rapid Development Testing

Query your database interactively without running the full agent:

# From my-control-assistant directory
python src/my_control_assistant/services/channel_finder/cli.py

This launches an interactive terminal where you can:

Test queries against your channel database in real-time
Watch the agent explore the database with tool calls
Verify which channels are matched and why
Measure query execution time to optimize performance
Iterate on your database and see results immediately

Example queries to try:

Simple lookups: “What’s the beam current?” → SR:DCCT:Current
Functional queries: “All BPM X positions” → All horizontal BPM readbacks
Filtered queries: “BPM positions in sector 1” → Sector-filtered results
Multi-system: “Vacuum pressure in booster” → BR system vacuum channels

The CLI shows the agent’s tool calls and reasoning, making it easy to spot issues with descriptions or database organization.

Benchmarking: Critical Validation Before Production

Benchmark datasets are essential for validating your channel finder before deployment. A benchmark that performs well during development with developer-created queries may completely fail in production when users with different mental models start querying the system.

Once your database is ready and you have a diverse benchmark dataset from domain experts, run systematic benchmarks to measure accuracy:

# Run all benchmark queries
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py

# Test specific queries (useful during development)
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --queries 0,1

# Compare model performance
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --model anthropic/claude-sonnet

Benchmark Capabilities:

Statistical metrics: Precision (accuracy of matches), recall (completeness), and F1 scores (harmonic mean)
Success categorization: Perfect matches (100% correct), partial matches (some correct), and failures
Parallel execution: Runs 5 queries concurrently by default to speed up large benchmark sets
Incremental saves: Results are saved after each query, so interruptions don’t lose progress
Consistency tracking: Measures whether repeated queries produce the same results

Configuring Benchmarks:

The benchmark system is configured in config.yml with both global settings and pipeline-specific datasets:

channel_finder:
  # Active pipeline determines which benchmark dataset is used by default
  pipeline_mode: middle_layer

  pipelines:
    middle_layer:
      database:
        type: middle_layer
        path: src/my_control_assistant/data/channel_databases/middle_layer.json

   # Benchmark dataset for this pipeline
   benchmark:
     dataset_path: src/my_control_assistant/data/benchmarks/datasets/middle_layer_benchmark.json
     # Middle layer pipeline benchmark (35 queries)

# Global benchmarking configuration
benchmark:
  # Execution settings
  execution:
    runs_per_query: 1                 # Number of times to run each query
    delay_between_runs: 0             # Delay in seconds between runs
    continue_on_error: true           # Continue even if some queries fail
    max_concurrent_queries: 5         # Maximum parallel queries
    query_selection: all              # "all" or specific queries like [0,1,2]

  # Output settings
  output:
    results_dir: src/my_control_assistant/data/benchmarks/results
    save_incremental: true            # Save results after each query
    include_detailed_metrics: true    # Include detailed timing/cost metrics

  # Evaluation thresholds
  evaluation:
    exact_match_weight: 1.0
    partial_match_weight: 0.5

You can override the dataset and other settings via CLI:

# Use a different dataset
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py \
   --dataset src/my_control_assistant/data/benchmarks/datasets/custom_dataset.json

# Run specific queries
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --queries 0,1,5,10

# Show detailed logs
python src/my_control_assistant/services/channel_finder/benchmarks/cli.py --verbose

When to use these tools:

During development: Use the CLI to rapidly test queries as you build your database
Before deployment: Run full benchmarks with domain expert queries to validate accuracy meets production requirements (aim for >90% F1 score)
Model selection: Compare different LLM models to balance cost, speed, and accuracy for your specific use case
Continuous validation: Re-run benchmarks after database updates to ensure no regression in accuracy

Best Practices:

Rich Descriptions: Write detailed, semantic descriptions at all levels—they guide the agent
Functional Grouping: Organize by function (Monitor, Setpoint) not location
Consistent Naming: Use same field names across families for predictability
Iterative Testing: Test incrementally as you build each system
Domain Expert Queries: Gather test queries from multiple operators and experts
Regular Benchmarking: Re-run benchmarks after database changes to catch regressions
CLI First: Use interactive CLI for rapid iteration before running full benchmarks

2.1: OSPREY Framework Integration#

The channel finder integrates into OSPREY as a capability—a reusable component that the agent orchestrator can plan and execute. The capability acts as a thin orchestration layer that connects the framework to the service layer, while all channel finding logic lives in services/channel_finder/. This separation makes the service independently testable via CLI and benchmarks before agent integration. The template provides a complete implementation in src/my_control_assistant/capabilities/channel_finding.py.

Capability Architecture:

@capability_node
class ChannelFindingCapability(BaseCapability):
    name = "channel_finding"
    description = "Find control system channel addresses based on descriptions"
    provides = ["CHANNEL_ADDRESSES"]  # What this capability produces
    requires = []                      # What this capability needs (none)

How the Agent Uses It:

Task Classification: When the user asks about channels (e.g., “What’s the beam current?”), the classifier identifies this requires channel finding
- Customization point: Modify _create_classifier_guide() to change classification behavior with few-shot examples
- See: Classification and Routing for classification mechanics
- Example: Classifier Guide in Hello World tutorial
Planning: The orchestrator creates a plan that includes a channel_finding step
- Customization point: Modify _create_orchestrator_guide() to influence when and how the capability is planned
- See: Orchestrator Planning for orchestration logic
- Example: Orchestrator Guide in Hello World tutorial
Execution: The capability calls the service layer (ChannelFinderService) to find matching channels
Context Storage: Results are stored in the agent state as CHANNEL_ADDRESSES context
Downstream Use: Other capabilities (like channel_read) consume the CHANNEL_ADDRESSES context

Error Handling:

The capability classifies errors for appropriate agent responses:

ChannelNotFoundError: Triggers re-planning with suggestions to refine the query
ChannelFinderServiceError: Critical error, aborts execution with diagnostic information

Configuration:

The capability reads config.yml to determine which pipeline to use (in_context or hierarchical). No code changes needed—just update the config:

channel_finder:
  pipeline_mode: in_context  # or "hierarchical"

Step 4: The Service Layer Pattern#

A key architectural pattern: separate business logic from framework orchestration. This service layer pattern provides several critical benefits for production systems:

Clean Boundaries: Clear separation of concerns ensures that the service layer contains pure business logic and algorithms, while the capability layer handles state management and orchestration. No framework coupling exists in service code, making it easier to maintain and evolve.

Independent Testing: Service logic can be tested without framework overhead. For example, you can unit test the service directly:

service = ChannelFinderService()
result = await service.find_channels("beam current")
assert "SR:CURRENT:RB" in result.channels

Reusability: The same service can be used in multiple contexts—Osprey capabilities, standalone CLI tools, MCP servers, and web APIs. Your business logic stays portable across deployment scenarios. For example:

@mcp.tool()
async def find_channels(query: str):
    service = ChannelFinderService()
    return await service.find_channels(query)

Architecture Overview:

# ===== SERVICE LAYER: src/my_control_assistant/services/channel_finder/ =====
# Pure Python, no framework dependencies

class ChannelFinderService:
    """Standalone service for channel finding."""

    def __init__(self):
        # Load database based on config
        self.pipeline = self._load_pipeline_from_config()

    async def find_channels(self, query: str) -> ChannelFinderResult:
        """
        Find channels matching natural language query.

        This is pure business logic - no AgentState, no StateManager.
        """
        result = await self.pipeline.process_query(query)

        return ChannelFinderResult(
            query=query,
            channels=[ChannelInfo(address=ch) for ch in result.channels],
            total_channels=len(result.channels),
            processing_notes=result.notes
        )

# ===== CAPABILITY LAYER: src/my_control_assistant/capabilities/channel_finding.py =====
# Framework integration only

@capability_node
class ChannelFindingCapability(BaseCapability):
    """Thin wrapper connecting service to framework."""

    name = "channel_finding"
    provides = ["CHANNEL_ADDRESSES"]

    async def execute(self) -> Dict[str, Any]:
        search_query = self.get_task_objective()

        # Delegate to service layer
        service = ChannelFinderService()
        result = await service.find_channels(search_query)

        # Convert to framework context
        context = ChannelAddressesContext(
            channels=[ch.address for ch in result.channels],
            description=result.processing_notes
        )

        # Store in framework state
        return self.store_output_context(context)

Part 2: Building Your Channel Finder#

Step 3: Semantic Channel Finding Pipelines#

2.1: OSPREY Framework Integration#

Step 4: The Service Layer Pattern#

Navigation#