Four tools let an agent open something that is not text: a photo, a scanned invoice, a recording, a video. Each one turns its input into something a model can reason about, and which tool you want depends on what you need out of the file rather than on its extension.
In the SoAI WebUI, enable a media tool by selecting Chat configuration in the chat header, then
Tools. Turn on Tools enabled, choose the tool in the required mode, and select
Save. The account needs MCP usage (MCP_USE) and the selected model must support tool
calling. read_image also needs a vision-capable model; read_audio and audio extraction
from read_video need a local speech-to-text plugin backend and model.
See Set up speech-to-text.
The tool runtime must be enabled (TOOLS.MCP.ENABLED).
Picking the right one
| Tool | Use it when | Needs |
|---|---|---|
read_audio |
You want a recording transcribed | A local speech-to-text plugin backend and model |
read_document |
You want the text out of a PDF, an office file, or a scan | Nothing for text PDFs; OCR components for scans |
read_image |
The model should look at a picture and describe or judge it | A model with vision |
read_video |
You want frames, and usually the spoken audio with them | ffmpeg, plus a local speech-to-text plugin backend and model for the audio track |
The distinction between the first two catches people out. A photograph of a whiteboard goes to
read_image if you want the model to interpret the diagram, and to read_document if you
want the words typed out. The tool descriptions say the same thing: use read_document only when you
need OCR or text extraction.
Images
read_image takes a file_path inside the conversation's folder and passes the image to the
model as an image, not as a description of one. Three limits apply at different stages.
-
SoAI rejects a file bigger than
TOOLS.MCP.READ_IMAGE.MAX_SOURCE_BYTES, 128 MiB by default. The agent gets an error rather than a truncated picture. -
SoAI scales down an image with more pixels than
TOOLS.MCP.READ_IMAGE.MAX_PIXELS, 4 million by default, before encoding. A 6000 by 4000 photo is 24 million pixels, so it arrives shrunk. This works for describing a scene but can lose small print in a corner. -
The JPEG starts at
TOOLS.MCP.READ_IMAGE.JPEG_QUALITY, 85 by default. If its Base64 payload exceedsTOOLS.MCP.READ_IMAGE.MAX_ENCODED_CHARS, 16,777,216 characters by default, SoAI lowers JPEG quality as far as 60 and then shrinks the dimensions. It rejects the image if 16 preparation attempts still cannot fit the payload under the limit.
When the detail matters, crop before you hand it over, or use read_document so OCR runs at full
resolution.
To change these limits, an administrator with Configuration changes (CONFIG_PATCH) selects
the Settings button at the right of the header, then Settings. Select
Advanced in the Settings page header. The page becomes Advanced settings and the
control changes to Basic. Search for TOOLS.MCP.READ_IMAGE, edit the required key, and select Save.
Documents
read_document extracts text and accepts exactly one of four sources: a file_path in the
conversation folder, a file_id from SoAI storage, a document_id from the conversation's
Knowledge, or a url. Passing more than one is an error, which keeps the agent from silently reading the
wrong thing.
Long documents come back in chunks. Set max_chars and offset_chars, and when the response
reports that it was truncated, continue from the offset it hands back. That keeps a 400-page PDF from filling the
context window in one call.
For PDFs with no text layer, scans, and photographs of text, the pipeline falls back to OCR. SoAI provisions its managed Tesseract runtime during startup; if that cannot complete, raster-only document OCR reports as unavailable. This is also the path that reads text out of video frames.
Audio
read_audio transcribes through a local speech-to-text plugin and returns timing, container and source
metadata, speech pacing, silence gaps, and any confidence figures the model supplies. Set language
when known. The task argument selects transcription in the original language or translation to English,
when the selected model supports it. Missing word probabilities are returned as null. A model that cannot return
the requested timing or language reports an unsupported-option error.
include_word_timestamps adds per-word timing, which is what you want for subtitles or for quoting an
exact moment. It makes the response considerably longer, so leave it off otherwise.
The model argument accepts auto, a released Whisper size selector, or a concrete model
reference such as a universal ID or a plugin-qualified reference. An omitted or null argument follows
TOOLS.MCP.READ_VIDEO.WHISPER_MODEL.
Auto chooses an eligible local model. A specific selection never falls back to another
model or a remote provider. Install a backend and download its model through
Set up speech-to-text.
Video
read_video pulls frames out of a local video and, by default, transcribes the audio alongside them.
Narrow the work with start_seconds and end_seconds rather than feeding in an hour-long
file: frame extraction is the slow part.
Sampling runs at one frame per second, which is both the default and the ceiling. The initial call records a job and
blocks until the selected range finishes or is cancelled. Its result includes a job_id and the first
page of up to 60 frames. The agent passes the returned cursor to read later pages, or uses
cancel with a job ID or cursor to request cancellation. Two jobs may run at once, and completed results
are kept for a week.
SoAI caps frames at 2 million pixels and encodes them as JPEG at quality 85. It transcribes audio in five-minute
chunks with the local model selected by TOOLS.MCP.READ_VIDEO.WHISPER_MODEL.
Fresh installations default to Auto, while existing saved selections are preserved. Prepare a plugin backend and
download its model before enabling offline mode. With include_audio=false, or a video without audio,
speech setup is unnecessary and frame extraction continues normally.
An administrator with Configuration changes (CONFIG_PATCH) can change these controls in Advanced settings. Select
the Settings button at the right of the header, then Settings, select
Advanced in the Settings page header, search for the key, and select Save after
editing.
In the chat transcript, a completed video read shows its extracted Frames and transcript inline, so you can check the material sent to the model.