Picture Captions and Picture Search#
ARIEL can read the pictures attached to logbook entries in two ways, each an enhancement module of its own:
image_captionasks a vision model to describe each copied picture and list its visible text. The caption is folded into the entry’s searchable text, so keyword, semantic and hybrid search find an entry by what its plots show.image_embeddingturns each copied picture into a vector with a multimodal embedding model served by a site-runllama-server.hybridsearch then also ranks pictures against the query text, so an entry known only by its pictures can still come back.
Both read the picture bytes ARIEL copied at ingest
(ariel.attachments.copy_on_ingest), so they see only copied pictures.
The agent’s attachment_view tool, which returns one stored picture to the
agent, needs neither module. See ARIEL Contracts for the
tool and result contract.
Agent tools for pictures#
attachment_viewshows one stored picture to the agent, so it can answer what a plot shows. It is offered whileariel.attachments.view.enabledis on.entry_openshows an entry to the operator: it brings the Logbook panel forward with the entry’s detail card open, and with anattachment_idit also opens that picture enlarged. The main agent calls it itself, with the ids a logbook subagent reported: a subagent hands back the entry id andattachment_idof every picture it viewed.attachment_idis required and may be null: whenever the conversation holds the id of one of the entry’s pictures, the agent passes it, even when the operator asks only for “the entry”. Opened without one, the result lists the entry’s viewable pictures, so the agent can repeat the call with the picture under discussion. The same view is a link,#entry?id=<entry_id>&attachment=<attachment_id>, on the Logbook web page. Its lightbox also links the stored original at full resolution (“Open original”).attachment_to_artifactkeeps one picture: it copies the picture’s stored display rendition into the artifact gallery, with the entry id, attachment id, filename and caption recorded, and selects it there. Saving the same picture twice in one session gives the same artifact. Likeattachment_viewit is offered only whileariel.attachments.view.enabledis on and only for aviewablepicture. The agent is told to use it rather than redraw a logbook plot from its caption.
Picture formats#
The format table below is the one ARIEL uses everywhere: what it copies, renders, captions, embeds and shows. Accepted formats are rendered and viewable; reserved formats are recognised by MIME type and recorded as skipped.
Format |
Status |
MIME types |
|---|---|---|
|
accepted |
|
|
accepted |
|
|
accepted |
|
|
accepted |
|
|
accepted |
|
|
accepted |
|
|
reserved |
|
|
reserved |
|
|
reserved |
|
|
reserved |
|
|
reserved |
|
|
reserved |
|
|
reserved |
|
capabilities.attachments.formats reports the same two lists at run time.
Captions#
Captions come from any chat provider with a vision-capable model. The
provider and model are the module’s own, never the deployment’s main model,
and the presets set them in profile.yml as:
config:
ariel.enhancement_modules.image_caption.provider: ollama
ariel.enhancement_modules.image_caption.model.model_id: qwen3-vl:4b
Change either line to use another provider or model. With the preset defaults, pull the model once:
ollama pull qwen3-vl:4b
On a CPU, a local vision model can spend minutes on one picture; the measured
values are in
Measured values, and
image_caption.timeout_seconds (default 1320) is sized from them.
The llama-server for picture search#
OSPREY ships no image or service for the embedding server: the site builds and
runs llama-server itself, and the llama-cpp provider talks to it (see
the provider table in Configure LLM Providers). It
serves both picture vectors at enhancement time and query vectors at search
time.
Build#
Picture search needs multimodal embeddings in llama-server, which llama.cpp
added in PR #29556 (merged
2026-09-28). Until that support reaches the packaged llama.cpp your site
installs from, build tag b11277, the build OSPREY is tested against.
Note
b11277 is commit eae11d2217fe9225d1aaba48773b6cca45ae4de9. Builds
from before PR #29556, including Homebrew’s llama.cpp 0.5.0, do not
have the support. A later build that carries the PR is untested here: run
osprey ariel status and one picture search after switching, and treat
it like a model change in the Upgrade notes if the picture vectors move.
git clone --depth 1 --branch b11277 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Release # add -DGGML_CUDA=ON for an NVIDIA GPU
cmake --build build --target llama-server
-DBUILD_SHARED_LIBS=OFF makes build/bin/llama-server a single file that
still runs after it is copied elsewhere (see Keeping it running). Metal is
on by default on a Mac.
Model files#
Both files come from the Hugging Face repository
DevQuasar/Qwen.Qwen3-VL-Embedding-2B-GGUF at revision
6a1b927414664e0e17dd379913e3416a1ae1b48d. The vectors depend on both, so
check them:
File |
Bytes |
sha256 |
|---|---|---|
|
1107410528 |
|
|
819395136 |
|
sha256sum Qwen.Qwen3-VL-Embedding-2B.Q4_K_M.gguf mmproj-Qwen.Qwen3-VL-Embedding-2B.f16.gguf
Command#
llama-server --embedding --pooling last -m Qwen.Qwen3-VL-Embedding-2B.Q4_K_M.gguf --mmproj mmproj-Qwen.Qwen3-VL-Embedding-2B.f16.gguf --alias qwen3-vl-embedding-2b --host 127.0.0.1 --port 8080 --no-webui --no-slots -c 8192 -b 2048 -ub 2048 -np 2 --image-max-tokens 256
Add --n-gpu-layers 99 on a GPU build. The other flags matter as follows:
--alias qwen3-vl-embedding-2bis the model id the server advertises on/v1/models. It must equalariel.enhancement_modules.image_embedding.modelin the profile, orosprey ariel statusreports the module skipped with reasonmodel.--image-max-tokens 256caps each picture at 256 vision tokens, about 512x512 px of area for Qwen3-VL. On a CPU-only server, a query that arrives while a picture is being embedded waits about one picture time; without the cap the measured query p95 under bulk embedding failed the 2 s target, with it the p95 passes (see Measured values).At start-up the server logs
Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks. It prints this for every Qwen-VL model started without--image-min-tokens 1024, whatever the cap. Grounding means answering with box coordinates, which picture search does not do, so the warning does not apply here; do not add--image-min-tokens. The cap does lower the resolution the model sees: a 1024x768 rendition goes from about 770 to about 240 tokens. Its effect on ranking has not been measured.--host 127.0.0.1binds the server to the loopback interface. See the next section for why.The command passes no
--media-path, so the server reads no local files.
Keeping it running#
Run the server from an installed copy, not from the build tree. Copy
build/bin/llama-server and both model files to a directory a service may
read, such as /usr/local/llama.cpp/ or
~/Library/Application Support/llama.cpp/ on macOS and /opt/llama.cpp/
on Linux, and give the full paths to -m and --mmproj. On macOS a
LaunchAgent cannot read ~/Desktop, ~/Documents or ~/Downloads:
the system’s privacy protection refuses it without asking, and the server
fails to start or to load its model. The copy is self-contained only when
built static (see Build). OSPREY ships no LaunchAgent or service unit for
the server.
Why the server listens on localhost only#
Build b11277 has no switch to disable remote image-URL fetch. Its
handle_media function (tools/server/server-common.cpp:1088-1102)
downloads any http or https URL a request names as an image_url,
up to 10 MiB with a 10 s timeout; only file:// URLs are gated, by
--media-path. A request that names an http URL makes the server fetch
it. Probed live against b11277, the server sent this request to the URL a
client named:
GET /orbit_kick.png HTTP/1.1
Host: 127.0.0.1:18999
User-Agent: llama-cpp/b1-eae11d2
Connection: close
Accept: */*
Anyone who can reach the server can therefore make it issue requests from the
host’s network position. Binding to 127.0.0.1 limits that to processes on
the same host. OSPREY itself sends only data: URLs.
How containers reach it#
The llama-cpp provider’s default address is http://localhost:8080;
LLAMA_CPP_HOST overrides it.
Docker Desktop (macOS, Windows). A container reaches a server on the host’s loopback through the same fallback table OSPREY uses for Ollama: a server configured as
localhostis also tried ashost.docker.internalfrom a Docker container (host.containers.internalfrom a Podman one). Nothing needs to be configured.Linux host. A localhost-bound server is reachable only from containers on host networking. Run
ariel-syncwith host networking by settingservices.ariel_sync.network: host, and do the same for any bridge-networked container that serveshybrid_search; web terminals already use host networking. Inprofile.ymlthat isnetwork: hostunder the service’sconfig:key:services: ariel_sync: config: network: host
On the default bridge network,
osprey ariel statusinsideariel-syncreportsimage_embeddingunreachable andpicture_search_unavailable: "unreachable". See Network Binding and Security for what host networking changes.
Limits#
Limits you can change#
ariel.attachments.max_file_mb(default 10)The largest file one entry may attach, per file. A larger picture is not copied, so neither module sees it, and its file card in the web panel says why. See Data Ingestion for what it costs in storage.
ariel.enhancement_modules.image_caption.max_images_per_entry(default 8)Pictures captioned per entry, in attachment order. The rest are recorded as over the cap and are not sent to the model;
osprey ariel statuscounts them. After raising the key, runosprey ariel enhance --module image_caption --retry-failedto caption the pictures now within it. Lowering it keeps the captions already made.ariel.enhancement_modules.image_caption.prompt_templateReplaces the caption prompt;
{text}is replaced by the entry text. A replacement must keep asking for theVisible text:list the reply is split at. Data Ingestion has an example that adds a site’s device names. Each caption records the prompt it was made with, and a new prompt captions nothing again by itself:osprey ariel statusreports how many captions were made with an older prompt, andosprey ariel enhance --module image_caption --refresh-stalecaptions those pictures again. The old caption stays searchable until the new one replaces it; when the model refuses the new prompt, the picture keeps its old caption andosprey ariel statuscounts it as a refresh that could not be made. Captions made before captions recorded their prompt are counted apart and left as they are; to caption every picture again, changemodel.model_id.ariel.enhancement_modules.image_caption.timeout_seconds(default 1320)Seconds one vision call may take; see Measured values.
Fixed limits#
At most 20 files are copied per entry, and at most 4 x
max_file_mbbytes in all; a file past either budget is recorded asper_entry_limit.The picture the agent, both modules and the web panel view is a rendition at most 1024 px on its longest side and 3.5 MiB. The original is stored as copied: the web panel’s lightbox links it (“Open original”) and
/api/attachments/<attachment_id>serves it.
The 10 MiB limit in Why the server listens on localhost only is
llama-server’s own limit for pictures it downloads from an http URL. It
does not apply to OSPREY, which sends each picture in the request as a
data: URL, and it is unrelated to max_file_mb.
Fusion of picture and text results#
hybrid search merges the picture lane into the text results with two
thresholds: a picture match below a cosine similarity of 0.45
(min_similarity) is never trusted, and an entry found only through its
pictures is admitted only within 0.08 (relative_margin) of the best
picture match. These defaults are uncalibrated: no labelled set of
logbook pictures and queries existed to calibrate them against, so no set size
or precision can be stated. The one check they pass is that 0.45 is below
the 1024-dimension cosine (0.540) between the query “orbit kick near BPM 7”
and the matching probe picture in the recorded fixtures.
scripts/benchmark/fusion_calibration.py --set <set.json> --dsn <store>
--threshold <precision> reruns the calibration once a labelled set exists.
Measured values#
The values below were measured with the command above on one native x86_64
Linux host, twice: once on its CPU alone, once on its GPU. They describe that
host; different hardware gives different times. The defaults OSPREY ships
(the query timeout, image_embedding.timeout_seconds and
image_caption.timeout_seconds) are sized from the CPU run, the slower of
the two, so they hold on a host without a GPU.
Host: uname -m x86_64; CPU AMD EPYC 7313 16-Core Processor
(2 sockets x 16 cores x 2 threads, 64 logical CPUs); 503 GiB RAM; docker host
architecture x86_64; Rocky Linux 8.10, docker 28.0.4.
CPU run: llama-server built without CUDA and started without GPU access, 32 threads and 2 slots; Ollama with no GPU visible. Other users kept the host lightly loaded (load average 1.3–2.9 on 64 threads).
GPU run: NVIDIA H100 PCIe 80 GB, driver 590.48.01. llama-server
b11277built with CUDA 12.8 for compute capability 9.0, started with the same command on another port and no GPU flags; Ollama 0.15.2 offloaded all 37 layers ofqwen3-vl:4bto the GPU. Another job shared the GPU throughout (36.4 GiB in use and 43% utilisation before the run), so these times are an upper bound for an idle H100.
Value |
CPU (AMD EPYC 7313) |
GPU (NVIDIA H100) |
|---|---|---|
Time per picture (s/picture, 1024x768 rendition) |
3.77 s mean (771 tokens) without the cap; 1.14 s (237 tokens) with
|
0.21 s without the cap; 0.071 s with the cap |
Pictures per hour (derived, 3600 / s/picture) |
about 3,150 with the cap; about 950 without |
about 50,800 with the cap; about 16,900 without |
llama-server peak RSS |
4.74 GiB with the cap; 7.57 GiB without; 3.44 GiB loaded and idle |
1.86 GiB with the cap; 3.65 GiB without; plus 4.8 GiB of GPU memory |
Idle query embed |
p50 0.059 s, p95 0.072 s |
p50 0.008 s, p95 0.020 s |
Query p95 under bulk |
3.83 s without the cap (fails); 0.94 s with |
0.18 s without the cap; 0.059 s with it. Both pass |
Query timeout chosen |
5 s ( |
the same defaults, sized from the CPU run |
Render worker |
30.6 MiB after start-up (target: at most 512 MiB), under a 1 GiB address-space limit; a 48 Mpx JPEG renders in 0.43 s with a peak of 119 MiB |
not GPU-dependent |
|
75.7 s, 199.4 s and 119.4 s per caption, mean 131.5 s; resident 4.25
GiB. |
59.6 s for the first caption, 49.0 s of it loading the model; then
2.6 s and 9.4 s. 5.0 GiB of GPU memory. |
Captions per hour at the defaults (derived, 3600 / mean) |
about 27 |
about 600 once the model is loaded (mean 6.0 s) |
Upgrade fold, 135,000 rows (half captioned) |
57.6 s (PostgreSQL 16.15) |
not GPU-dependent |
v2 full-text index build, 135,000 rows |
27.2 s (99 MB index) |
not GPU-dependent |
Trigram index build, 135,000 rows |
2.6 s (24 MB index) |
not GPU-dependent |
Fusion calibration |
Uncalibrated: no labelled set; defaults |
the same |
Upgrade notes#
Profile keys. An existing deployment’s profile.yml is explicit and does
not gain the new keys. Add the lines shown below, or run
osprey init --revalidate (it writes every lacking leaf, with the side effect
named below); until then
image_caption and image_embedding keep their code defaults (off) and the
view tool its default (on).
config:
ariel.attachments.copy_on_ingest: images
ariel.attachments.view.enabled: true
ariel.enhancement_modules.image_caption.enabled: true
ariel.enhancement_modules.image_caption.provider: ollama
ariel.enhancement_modules.image_caption.model.model_id: qwen3-vl:4b
ariel.enhancement_modules.image_embedding.enabled: true
ariel.enhancement_modules.image_embedding.provider: llama-cpp
ariel.enhancement_modules.image_embedding.model: qwen3-vl-embedding-2b
ariel.enhancement_modules.image_embedding.dimensions: 1024
The llama-cpp provider entry. A deployment that keeps its own
providers.yml does not need a llama-cpp entry: the adapter’s default
address and LLAMA_CPP_HOST already work. To declare it anyway, paste the
packaged entry:
providers:
llama-cpp:
api_key: llama-cpp # ignored by a server started without --api-key
base_url: ${LLAMA_CPP_HOST:-http://localhost:8080}
default_model: qwen3-vl-embedding-2b
models: # the name llama-server advertises (--alias)
- qwen3-vl-embedding-2b
osprey init --revalidate also writes it, with a side effect: it fills in
every key the profile leaves to its preset and stamps provenance.
Search indexes. The upgrade builds a v2 full-text index and a trigram index over attachment text, and keeps the v1 indexes (the full-text index and, with the semantic processor on, its search index); a later release drops them. The build times on 135,000 rows are in Measured values.
Captions already present upstream. The upgrade folds captions that came with the upstream entries into each entry’s searchable text and marks those entries’ text embeddings as owed again. Count them with the fold’s own predicate, so null captions are not counted:
SELECT count(*) FROM enhanced_entries WHERE attachments @? '$[*] ? (@.caption != null && @.caption != "")'
With a paid embedding provider, drain that backlog in a maintenance window, passing the count as the limit (the default limit is 100):
osprey ariel enhance --module text_embedding --limit <N>
Emptied attachment lists. An upstream entry whose attachment list becomes empty keeps its stored attachment rows. This is a known limitation.
Re-running the picture modules. --force does not apply to them:
–force does not re-run image_caption/image_embedding: their results are kept per picture and model. Change model.model_id (captions) or model/dimensions (embeddings) to re-run, or use –retry-failed for per-picture failures.
Captions made before this release. They record no prompt, so
osprey ariel status counts them as captions that record no prompt, and
--refresh-stale leaves them as they are.
Changing the server. Changing --image-max-tokens or the model files
changes every picture vector while the table name stays the same, so re-embed
with osprey ariel purge --embeddings-only followed by a catch-up (which
re-embeds text too).