Skip to main content
Ctrl+K

Osprey Framework

  • Getting Started
  • How-To Guides
  • Architecture
  • Reference
  • Contributing to Osprey
  • GitHub
  • Getting Started
  • How-To Guides
  • Architecture
  • Reference
  • Contributing to Osprey
  • GitHub

Section Navigation

  • Build Profiles
  • Deploy a Facility
  • Container Deployment
    • Compose Templates and Images
    • Network Binding and Security
    • Environment Variables (the .env chain)
    • The Project Image
  • LLM Providers
    • Configure LLM Providers
    • Run Open & Local Models
  • Build Your Own OSPREY Apps
    • Build Your Own App
  • Register a measurement kind
  • Web Terminal
    • Run the Web Terminal
    • Theming
    • Panels
    • Notebooks
    • CANOPY
    • Share CANOPY Screens
    • Send and Retrieve Feedback
    • Multi-User Support
      • Privilege Tiers
      • Require a Login
  • Agent Interfaces
    • Run the Agent from the Command Line
    • Event Dispatch
    • Add an MCP Server
    • Chat Bridges
      • Nextcloud Talk
      • Google Chat
      • Microsoft Teams
      • Slack
  • Health and Monitoring
    • Configure Health Checks
    • The Agent Record
  • Control Systems
    • Use Connectors
    • Use the Connectors Without the Framework
    • Use the Virtual Accelerator
      • Serve the Virtual Accelerator from a Machine Repository
    • Switch the Control Target at Run Time
    • Connect the Agent to Phoebus Displays
    • Review the Agent’s Code
  • Safety watchdogs
  • Describe your facility
  • Measure with the pyAML tools
  • How to Use the Channel Finder
  • Import a Middle Layer export
  • Facility Knowledge
    • Facility Rules
    • The OKF Bundle
    • Search the Facility Graph
  • Logbook Search Service (ARIEL)
    • Data Ingestion
    • Search Modes
    • Web Interface
    • The Search Sidecar (qmd)
    • Picture Captions and Picture Search
  • Bluesky Plans
    • Run Your First Plan
    • Plans and the Queue
    • Write Your Own Plans
  • How-To Guides
  • Logbook Search Service (ARIEL)
  • Picture Captions and Picture Search

Picture Captions and Picture Search#

ARIEL can read the pictures attached to logbook entries in two ways, each an enhancement module of its own:

  • image_caption asks a vision model to describe each copied picture and list its visible text. The caption is folded into the entry’s searchable text, so keyword, semantic and hybrid search find an entry by what its plots show.

  • image_embedding turns each copied picture into a vector with a multimodal embedding model served by a site-run llama-server. hybrid search then also ranks pictures against the query text, so an entry known only by its pictures can still come back.

Both read the picture bytes ARIEL copied at ingest (ariel.attachments.copy_on_ingest), so they see only copied pictures. The agent’s attachment_view tool, which returns one stored picture to the agent, needs neither module. See ARIEL Contracts for the tool and result contract.

Agent tools for pictures#

  • attachment_view shows one stored picture to the agent, so it can answer what a plot shows. It is offered while ariel.attachments.view.enabled is on.

  • entry_open shows an entry to the operator: it brings the Logbook panel forward with the entry’s detail card open, and with an attachment_id it also opens that picture enlarged. The main agent calls it itself, with the ids a logbook subagent reported: a subagent hands back the entry id and attachment_id of every picture it viewed. attachment_id is required and may be null: whenever the conversation holds the id of one of the entry’s pictures, the agent passes it, even when the operator asks only for “the entry”. Opened without one, the result lists the entry’s viewable pictures, so the agent can repeat the call with the picture under discussion. The same view is a link, #entry?id=<entry_id>&attachment=<attachment_id>, on the Logbook web page. Its lightbox also links the stored original at full resolution (“Open original”).

  • attachment_to_artifact keeps one picture: it copies the picture’s stored display rendition into the artifact gallery, with the entry id, attachment id, filename and caption recorded, and selects it there. Saving the same picture twice in one session gives the same artifact. Like attachment_view it is offered only while ariel.attachments.view.enabled is on and only for a viewable picture. The agent is told to use it rather than redraw a logbook plot from its caption.

On by default, skipped when unavailable#

The demo preset and its logbook login turn captions, picture search and the view tool on, and they choose a service and model for each: Ollama with qwen3-vl:4b for captions, and a site-run llama-server with qwen3-vl-embedding-2b for picture search. These are preset defaults, not requirements. Every one of them is a key in your profile.yml, so you can point captions at any vision-capable chat provider, run a different embedding model, or turn a module off (see Captions and below). The names in the rest of this guide are the preset defaults.

Like semantic search without Ollama, each picture module degrades gracefully: it runs when its configured server and model answer, and otherwise it is skipped while the rest of ARIEL keeps working.

A deployment without its caption model (with the preset defaults: Ollama not running, or qwen3-vl:4b not pulled) sees:

  • osprey ariel status printing a line that names the skipped module and why, for example image_caption: skipped, model qwen3-vl:4b not available on ollama (pull it, or set ariel.enhancement_modules.image_caption.enabled: false);

  • entries searchable by their text alone, as before captions existed.

A deployment without llama-server sees:

  • osprey ariel status printing, for example, image_embedding: skipped, llama-cpp not reachable at http://localhost:8080 (start llama-server, see the picture-search guide, or set ariel.enhancement_modules.image_embedding.enabled: false);

  • hybrid_search answering on text only, with the diagnostic “Picture search unavailable — results are matched on text only, so entries known only by their pictures are missing.”;

  • capabilities.attachments.picture_search_unavailable set to the reason (unreachable, model, auth or config) once the picture lane has failed.

To turn a module off, set its enabled key to false in profile.yml:

config:
  ariel.enhancement_modules.image_caption.enabled: false    # no captions
  ariel.enhancement_modules.image_embedding.enabled: false  # no picture search
  ariel.attachments.view.enabled: false                     # no picture tools

Both osprey ariel watch and osprey ariel sync --watch (the command the ariel-sync service runs) caption and embed pictures on every poll. A deployment with either needs no osprey ariel enhance for them. Run by hand while the watcher is working on a module, osprey ariel enhance --module image_caption prints image_caption: skipped, another pass is running it (osprey ariel watch runs the picture modules on every poll) and does nothing for that module. The same holds for --retry-failed and --refresh-stale. Run it again once the watcher’s pass has ended.

Picture formats#

The format table below is the one ARIEL uses everywhere: what it copies, renders, captions, embeds and shows. Accepted formats are rendered and viewable; reserved formats are recognised by MIME type and recorded as skipped.

Format

Status

MIME types

png

accepted

image/png

jpeg

accepted

image/jpeg

gif

accepted

image/gif

webp

accepted

image/webp

bmp

accepted

image/bmp

tiff

accepted

image/tiff

svg

reserved

image/svg+xml

pdf

reserved

application/pdf

text

reserved

text/plain

heif

reserved

image/heic, image/avif

video

reserved

video/mp4, video/quicktime, video/webm, video/x-msvideo

audio

reserved

audio/mp4, audio/mpeg, audio/flac, audio/ogg, audio/wav

archive

reserved

application/zip, application/gzip, application/x-bzip2, application/x-xz, application/x-7z-compressed, application/vnd.rar

capabilities.attachments.formats reports the same two lists at run time.

Captions#

Captions come from any chat provider with a vision-capable model. The provider and model are the module’s own, never the deployment’s main model, and the presets set them in profile.yml as:

config:
  ariel.enhancement_modules.image_caption.provider: ollama
  ariel.enhancement_modules.image_caption.model.model_id: qwen3-vl:4b

Change either line to use another provider or model. With the preset defaults, pull the model once:

ollama pull qwen3-vl:4b

On a CPU, a local vision model can spend minutes on one picture; the measured values are in Measured values, and image_caption.timeout_seconds (default 1320) is sized from them.

The llama-server for picture search#

OSPREY ships no image or service for the embedding server: the site builds and runs llama-server itself, and the llama-cpp provider talks to it (see the provider table in Configure LLM Providers). It serves both picture vectors at enhancement time and query vectors at search time.

Build#

Picture search needs multimodal embeddings in llama-server, which llama.cpp added in PR #29556 (merged 2026-09-28). Until that support reaches the packaged llama.cpp your site installs from, build tag b11277, the build OSPREY is tested against.

Note

b11277 is commit eae11d2217fe9225d1aaba48773b6cca45ae4de9. Builds from before PR #29556, including Homebrew’s llama.cpp 0.5.0, do not have the support. A later build that carries the PR is untested here: run osprey ariel status and one picture search after switching, and treat it like a model change in the Upgrade notes if the picture vectors move.

git clone --depth 1 --branch b11277 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Release   # add -DGGML_CUDA=ON for an NVIDIA GPU
cmake --build build --target llama-server

-DBUILD_SHARED_LIBS=OFF makes build/bin/llama-server a single file that still runs after it is copied elsewhere (see Keeping it running). Metal is on by default on a Mac.

Model files#

Both files come from the Hugging Face repository DevQuasar/Qwen.Qwen3-VL-Embedding-2B-GGUF at revision 6a1b927414664e0e17dd379913e3416a1ae1b48d. The vectors depend on both, so check them:

File

Bytes

sha256

Qwen.Qwen3-VL-Embedding-2B.Q4_K_M.gguf

1107410528

42a4ebc629ecc6514649e12b1529b857f54900273bb854f853c970fb90edd09d

mmproj-Qwen.Qwen3-VL-Embedding-2B.f16.gguf

819395136

3f89a7768ffa6606935319f71bf56bb71871249ba549bf1080a0caea7a088613

sha256sum Qwen.Qwen3-VL-Embedding-2B.Q4_K_M.gguf mmproj-Qwen.Qwen3-VL-Embedding-2B.f16.gguf

Command#

llama-server --embedding --pooling last -m Qwen.Qwen3-VL-Embedding-2B.Q4_K_M.gguf --mmproj mmproj-Qwen.Qwen3-VL-Embedding-2B.f16.gguf --alias qwen3-vl-embedding-2b --host 127.0.0.1 --port 8080 --no-webui --no-slots -c 8192 -b 2048 -ub 2048 -np 2 --image-max-tokens 256

Add --n-gpu-layers 99 on a GPU build. The other flags matter as follows:

  • --alias qwen3-vl-embedding-2b is the model id the server advertises on /v1/models. It must equal ariel.enhancement_modules.image_embedding.model in the profile, or osprey ariel status reports the module skipped with reason model.

  • --image-max-tokens 256 caps each picture at 256 vision tokens, about 512x512 px of area for Qwen3-VL. On a CPU-only server, a query that arrives while a picture is being embedded waits about one picture time; without the cap the measured query p95 under bulk embedding failed the 2 s target, with it the p95 passes (see Measured values).

    At start-up the server logs Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks. It prints this for every Qwen-VL model started without --image-min-tokens 1024, whatever the cap. Grounding means answering with box coordinates, which picture search does not do, so the warning does not apply here; do not add --image-min-tokens. The cap does lower the resolution the model sees: a 1024x768 rendition goes from about 770 to about 240 tokens. Its effect on ranking has not been measured.

  • --host 127.0.0.1 binds the server to the loopback interface. See the next section for why.

  • The command passes no --media-path, so the server reads no local files.

Keeping it running#

Run the server from an installed copy, not from the build tree. Copy build/bin/llama-server and both model files to a directory a service may read, such as /usr/local/llama.cpp/ or ~/Library/Application Support/llama.cpp/ on macOS and /opt/llama.cpp/ on Linux, and give the full paths to -m and --mmproj. On macOS a LaunchAgent cannot read ~/Desktop, ~/Documents or ~/Downloads: the system’s privacy protection refuses it without asking, and the server fails to start or to load its model. The copy is self-contained only when built static (see Build). OSPREY ships no LaunchAgent or service unit for the server.

Why the server listens on localhost only#

Build b11277 has no switch to disable remote image-URL fetch. Its handle_media function (tools/server/server-common.cpp:1088-1102) downloads any http or https URL a request names as an image_url, up to 10 MiB with a 10 s timeout; only file:// URLs are gated, by --media-path. A request that names an http URL makes the server fetch it. Probed live against b11277, the server sent this request to the URL a client named:

GET /orbit_kick.png HTTP/1.1
Host: 127.0.0.1:18999
User-Agent: llama-cpp/b1-eae11d2
Connection: close
Accept: */*

Anyone who can reach the server can therefore make it issue requests from the host’s network position. Binding to 127.0.0.1 limits that to processes on the same host. OSPREY itself sends only data: URLs.

How containers reach it#

The llama-cpp provider’s default address is http://localhost:8080; LLAMA_CPP_HOST overrides it.

  • Docker Desktop (macOS, Windows). A container reaches a server on the host’s loopback through the same fallback table OSPREY uses for Ollama: a server configured as localhost is also tried as host.docker.internal from a Docker container (host.containers.internal from a Podman one). Nothing needs to be configured.

  • Linux host. A localhost-bound server is reachable only from containers on host networking. Run ariel-sync with host networking by setting services.ariel_sync.network: host, and do the same for any bridge-networked container that serves hybrid_search; web terminals already use host networking. In profile.yml that is network: host under the service’s config: key:

    services:
      ariel_sync:
        config:
          network: host
    

    On the default bridge network, osprey ariel status inside ariel-sync reports image_embedding unreachable and picture_search_unavailable: "unreachable". See Network Binding and Security for what host networking changes.

Limits#

Limits you can change#

ariel.attachments.max_file_mb (default 10)

The largest file one entry may attach, per file. A larger picture is not copied, so neither module sees it, and its file card in the web panel says why. See Data Ingestion for what it costs in storage.

ariel.enhancement_modules.image_caption.max_images_per_entry (default 8)

Pictures captioned per entry, in attachment order. The rest are recorded as over the cap and are not sent to the model; osprey ariel status counts them. After raising the key, run osprey ariel enhance --module image_caption --retry-failed to caption the pictures now within it. Lowering it keeps the captions already made.

ariel.enhancement_modules.image_caption.prompt_template

Replaces the caption prompt; {text} is replaced by the entry text. A replacement must keep asking for the Visible text: list the reply is split at. Data Ingestion has an example that adds a site’s device names. Each caption records the prompt it was made with, and a new prompt captions nothing again by itself: osprey ariel status reports how many captions were made with an older prompt, and osprey ariel enhance --module image_caption --refresh-stale captions those pictures again. The old caption stays searchable until the new one replaces it; when the model refuses the new prompt, the picture keeps its old caption and osprey ariel status counts it as a refresh that could not be made. Captions made before captions recorded their prompt are counted apart and left as they are; to caption every picture again, change model.model_id.

ariel.enhancement_modules.image_caption.timeout_seconds (default 1320)

Seconds one vision call may take; see Measured values.

Fixed limits#

  • At most 20 files are copied per entry, and at most 4 x max_file_mb bytes in all; a file past either budget is recorded as per_entry_limit.

  • The picture the agent, both modules and the web panel view is a rendition at most 1024 px on its longest side and 3.5 MiB. The original is stored as copied: the web panel’s lightbox links it (“Open original”) and /api/attachments/<attachment_id> serves it.

The 10 MiB limit in Why the server listens on localhost only is llama-server’s own limit for pictures it downloads from an http URL. It does not apply to OSPREY, which sends each picture in the request as a data: URL, and it is unrelated to max_file_mb.

Fusion of picture and text results#

hybrid search merges the picture lane into the text results with two thresholds: a picture match below a cosine similarity of 0.45 (min_similarity) is never trusted, and an entry found only through its pictures is admitted only within 0.08 (relative_margin) of the best picture match. These defaults are uncalibrated: no labelled set of logbook pictures and queries existed to calibrate them against, so no set size or precision can be stated. The one check they pass is that 0.45 is below the 1024-dimension cosine (0.540) between the query “orbit kick near BPM 7” and the matching probe picture in the recorded fixtures. scripts/benchmark/fusion_calibration.py --set <set.json> --dsn <store> --threshold <precision> reruns the calibration once a labelled set exists.

Measured values#

The values below were measured with the command above on one native x86_64 Linux host, twice: once on its CPU alone, once on its GPU. They describe that host; different hardware gives different times. The defaults OSPREY ships (the query timeout, image_embedding.timeout_seconds and image_caption.timeout_seconds) are sized from the CPU run, the slower of the two, so they hold on a host without a GPU.

Host: uname -m x86_64; CPU AMD EPYC 7313 16-Core Processor (2 sockets x 16 cores x 2 threads, 64 logical CPUs); 503 GiB RAM; docker host architecture x86_64; Rocky Linux 8.10, docker 28.0.4.

  • CPU run: llama-server built without CUDA and started without GPU access, 32 threads and 2 slots; Ollama with no GPU visible. Other users kept the host lightly loaded (load average 1.3–2.9 on 64 threads).

  • GPU run: NVIDIA H100 PCIe 80 GB, driver 590.48.01. llama-server b11277 built with CUDA 12.8 for compute capability 9.0, started with the same command on another port and no GPU flags; Ollama 0.15.2 offloaded all 37 layers of qwen3-vl:4b to the GPU. Another job shared the GPU throughout (36.4 GiB in use and 43% utilisation before the run), so these times are an upper bound for an idle H100.

Value

CPU (AMD EPYC 7313)

GPU (NVIDIA H100)

Time per picture (s/picture, 1024x768 rendition)

3.77 s mean (771 tokens) without the cap; 1.14 s (237 tokens) with --image-max-tokens 256

0.21 s without the cap; 0.071 s with the cap

Pictures per hour (derived, 3600 / s/picture)

about 3,150 with the cap; about 950 without

about 50,800 with the cap; about 16,900 without

llama-server peak RSS

4.74 GiB with the cap; 7.57 GiB without; 3.44 GiB loaded and idle

1.86 GiB with the cap; 3.65 GiB without; plus 4.8 GiB of GPU memory

Idle query embed

p50 0.059 s, p95 0.072 s

p50 0.008 s, p95 0.020 s

Query p95 under bulk image_embedding (2 s target)

3.83 s without the cap (fails); 0.94 s with --image-max-tokens 256 (passes). The latency gate failed, so the documented command carries the cap

0.18 s without the cap; 0.059 s with it. Both pass

Query timeout chosen

5 s (max(5, 2 x 0.94)); image_embedding.timeout_seconds 120

the same defaults, sized from the CPU run

Render worker VmSize

30.6 MiB after start-up (target: at most 512 MiB), under a 1 GiB address-space limit; a 48 Mpx JPEG renders in 0.43 s with a peak of 119 MiB

not GPU-dependent

qwen3-vl:4b with think:false (Ollama)

75.7 s, 199.4 s and 119.4 s per caption, mean 131.5 s; resident 4.25 GiB. think:false is not honoured: every reply still carries thinking output

59.6 s for the first caption, 49.0 s of it loading the model; then 2.6 s and 9.4 s. 5.0 GiB of GPU memory. think:false is still not honoured

Captions per hour at the defaults (derived, 3600 / mean)

about 27

about 600 once the model is loaded (mean 6.0 s)

Upgrade fold, 135,000 rows (half captioned)

57.6 s (PostgreSQL 16.15)

not GPU-dependent

v2 full-text index build, 135,000 rows

27.2 s (99 MB index)

not GPU-dependent

Trigram index build, 135,000 rows

2.6 s (24 MB index)

not GPU-dependent

Fusion calibration

Uncalibrated: no labelled set; defaults relative_margin 0.08 and min_similarity 0.45 kept

the same

Upgrade notes#

Profile keys. An existing deployment’s profile.yml is explicit and does not gain the new keys. Add the lines shown below, or run osprey init --revalidate (it writes every lacking leaf, with the side effect named below); until then image_caption and image_embedding keep their code defaults (off) and the view tool its default (on).

config:
  ariel.attachments.copy_on_ingest: images
  ariel.attachments.view.enabled: true
  ariel.enhancement_modules.image_caption.enabled: true
  ariel.enhancement_modules.image_caption.provider: ollama
  ariel.enhancement_modules.image_caption.model.model_id: qwen3-vl:4b
  ariel.enhancement_modules.image_embedding.enabled: true
  ariel.enhancement_modules.image_embedding.provider: llama-cpp
  ariel.enhancement_modules.image_embedding.model: qwen3-vl-embedding-2b
  ariel.enhancement_modules.image_embedding.dimensions: 1024

The llama-cpp provider entry. A deployment that keeps its own providers.yml does not need a llama-cpp entry: the adapter’s default address and LLAMA_CPP_HOST already work. To declare it anyway, paste the packaged entry:

providers:
  llama-cpp:
    api_key: llama-cpp              # ignored by a server started without --api-key
    base_url: ${LLAMA_CPP_HOST:-http://localhost:8080}
    default_model: qwen3-vl-embedding-2b
    models:                         # the name llama-server advertises (--alias)
      - qwen3-vl-embedding-2b

osprey init --revalidate also writes it, with a side effect: it fills in every key the profile leaves to its preset and stamps provenance.

Search indexes. The upgrade builds a v2 full-text index and a trigram index over attachment text, and keeps the v1 indexes (the full-text index and, with the semantic processor on, its search index); a later release drops them. The build times on 135,000 rows are in Measured values.

Captions already present upstream. The upgrade folds captions that came with the upstream entries into each entry’s searchable text and marks those entries’ text embeddings as owed again. Count them with the fold’s own predicate, so null captions are not counted:

SELECT count(*) FROM enhanced_entries WHERE attachments @? '$[*] ? (@.caption != null && @.caption != "")'

With a paid embedding provider, drain that backlog in a maintenance window, passing the count as the limit (the default limit is 100):

osprey ariel enhance --module text_embedding --limit <N>

Emptied attachment lists. An upstream entry whose attachment list becomes empty keeps its stored attachment rows. This is a known limitation.

Re-running the picture modules. --force does not apply to them:

–force does not re-run image_caption/image_embedding: their results are kept per picture and model. Change model.model_id (captions) or model/dimensions (embeddings) to re-run, or use –retry-failed for per-picture failures.

Captions made before this release. They record no prompt, so osprey ariel status counts them as captions that record no prompt, and --refresh-stale leaves them as they are.

Changing the server. Changing --image-max-tokens or the model files changes every picture vector while the table name stays the same, so re-embed with osprey ariel purge --embeddings-only followed by a catch-up (which re-embeds text too).

previous

The Search Sidecar (qmd)

next

Bluesky Plans

On this page
  • Agent tools for pictures
  • On by default, skipped when unavailable
  • Picture formats
  • Captions
  • The llama-server for picture search
    • Build
    • Model files
    • Command
    • Keeping it running
    • Why the server listens on localhost only
    • How containers reach it
  • Limits
    • Limits you can change
    • Fixed limits
  • Fusion of picture and text results
  • Measured values
  • Upgrade notes
Edit on GitHub

© Copyright 2026, Osprey Developer Team.

Created using Sphinx 9.0.4.

Built with the PyData Sphinx Theme 0.16.1.