Capacity planning starts from the work SoAI must finish, not from the largest queue the host can hold. Describe the models, request shapes, concurrent owners, tools, file processing, storage growth, and recovery target, then measure that workload on the host that will run it and set admission limits below the point where it stopped improving. Scheduler and runtime tuning holds the individual controls and their factory values, Storage paths and host capacity covers where the data goes, and Resilience and self-healing describes what SoAI does once a limit is reached, so this page is about arriving at the numbers.
Describe the workload
| Input | Record |
|---|---|
| Inference | Model and quantization, context and output length, streaming, image or audio inputs, reasoning, and backend. |
| Demand | Concurrent users or API keys, burst size, sustained request rate, automation schedules, and acceptable wait. |
| Tools | Browser sessions, shell work, web or provider calls, subagents, mail and calendar sync, and image generation. |
| Data | Model downloads, managed files, Knowledge collections, logs, histories, backups, and expected monthly growth. |
| Recovery | Required backup frequency, restore time, update headroom, and behavior during one plugin or provider failure. |
Measure one representative unit
Run one request with the intended model and backend, and record model load time, time to first output, total duration, input and output tokens, peak RAM and VRAM, and storage activity. Repeat with the longest ordinary context and each media or tool path that changes resource use, because a short text prompt is not a safe proxy for a document-heavy agent turn.
For remote providers, record provider concurrency and rate limits, network latency, timeout behavior, and cost policy. Local compute capacity does not remove a remote quota, and remote inference does not remove local agent, file, queue, or database work.
SoAI answers part of the question before anything is downloaded. In the SoAI WebUI, open Models, select Add model, and select Check, which compares the selected variant's declared download size, memory, and VRAM requirements against the disk, VRAM, and combined memory it currently detects, then reports whether each one fits, whether the model is runnable on this system, and estimated download and load times. Models catalog covers the download steps. Read the result as a fit check for one model sitting alone on an idle host, since it knows nothing about context length, concurrent requests, or the other backends you intend to keep resident.
Increase concurrency until a limit appears
- Start at one running request and a small bounded queue.
- Increase concurrent owners and requests in controlled steps.
- Watch completed throughput, latency, queue depth, rejections, plugin state, RAM, VRAM, temperature, disk, and network.
- Stop increasing when throughput stops improving, latency exceeds the service goal, errors rise, or recovery margin disappears.
Read completed throughput rather than queue depth, because a queue that keeps growing is evidence of the limit and not evidence of capacity. Metrics charts Queue depth beside requests, latency, and tokens, while Hardware shows what the host was doing at the same moment. The ceiling belongs to the whole workload rather than to a single setting.
When every plugin slot is occupied
MODELS.ROUTING.MAX_CONCURRENT_PLUGINS
bounds how many non-persistent plugin backends stay resident, and reaching it does not refuse work on its own,
because SoAI first looks for a plugin it can evict safely. A non-persistent plugin in Error or Quarantined state
goes first whether or not it is busy, and after that SoAI takes an idle plugin that holds a loaded model, has no
active task, has an empty plugin queue, and is in a state that can be interrupted. A model loaded within the last 30
seconds is left alone so a backend that has just paid its load cost is not evicted before it serves anything.
When no candidate exists, the task is deferred with a reason recording that the instance is at capacity, and it stays queued for the next scheduling cycle instead of failing. Plan around that, because a slot limit set too low turns into waiting and repeated model loading rather than a visible error, and the model load time you measured earlier is the price of each swap.
Persistent plugins do not count toward the resident plugin limit and are not unloaded to make room. The bundled speech to text, text to speech, and external provider plugins are persistent, so account for their memory separately when enabled. The embedding plugin uses a resident plugin slot and can be unloaded to make room. Browser workers, Knowledge processing, and scheduled integrations also use resources alongside inference.
Turn the measurements into limits
Set the per-plugin task limit at the concurrency where the backend stopped gaining throughput rather than above it, since a backend that batches internally can get slower as external concurrency rises. Keep the owner limit low enough that one account, script, or key cannot hold every running slot while the rest of the workload waits. Size the queue from the wait a caller will tolerate rather than from what memory allows.
Durable admission is where storage meets capacity, and it runs two checks in a fixed order.
MODELS.ROUTING.DURABLE_QUEUE_HARD_LIMIT_TASKS
is evaluated first, with its factory value of 0 leaving the count unbounded, and then
MODELS.ROUTING.MIN_FREE_DISK_BYTES_FOR_ACCEPT
refuses acceptance when the volume holding the system database has less than its 256 MiB reserve free. Set both from
the storage and recovery capacity the host actually has, and treat a reserve rejection as a capacity signal rather
than an obstacle to work around. Scheduler and runtime tuning holds each path and
its factory value.
Plan for the degraded case
A virtual model adds capacity across requests rather than within one. Each request is dispatched to a single eligible member while the remaining members stay as its failover order, so more members raise throughput when several requests are in flight and do nothing for the latency of one. Test normal routing and the loss of each required member, and expect a failover target to have a lower quota, a different capability, or higher latency. Keep enough queue and timeout margin for a bounded transition, without promising that every failed request is safe to replay.
Storage is the part that fails slowly. Models, plugin runtimes, managed files, Knowledge indexes, the database, logs, retained backups, and browser data each grow at their own rate, and Storage paths and host capacity plans their placement while Data storage, retention, and relocation owns retention and the safe move sequence. Include the temporary space a download, extraction, update, or relocation needs, and keep a verified backup outside the host, because another directory on the same volume is not recovery capacity.
Set the operating boundary
Choose limits that leave memory, disk, and scheduler margin for cancellation, logging, health checks, backups, updates, and a degraded backend. Refusing new work with an explicit capacity outcome is safer than accepting a backlog the instance cannot finish. Revisit the numbers after changing a model, context limit, backend variant, driver, provider plan, tool selection, retention policy, or user population, and keep the workload definition and its measurements with the deployment so a later operator can tell a planned limit from a fault.
Deployment readiness checklist
- The normal and peak workload has a recorded test.
- Admission limits are below measured saturation.
- One member or provider can fail without making accepted work permanently ambiguous.
- Storage includes growth, update, relocation, and backup headroom.
- Backups are verified and a restore has been tested.
- Monitoring retains enough evidence to identify the limiting resource.