Scheduler and runtime tuning

Runtime tuning controls how SoAI admits work, how much of it runs at once, how long an attempt may take, and when idle model memory is released. This page holds those controls and their factory values, while Resilience and self-healing describes what the scheduler does with them under failure and Capacity planning covers choosing values for a measured workload. Raising a limit moves pressure somewhere else, usually to memory, a provider quota, or a backend that was already saturated, so change one control at a time and read completed throughput rather than queue depth.

Change a routing setting in the SoAI WebUI

Opening Advanced settings requires an Administrator with Configuration changes (CONFIG_PATCH).

1. Select the Settings button at the right of the header, then Settings.
2. Select Advanced in the page header.
3. Paste the full configuration path into Search settings..., change the value, and select Save.
4. Select Restart SoAI if the WebUI reports a pending restart.

MODELS.ROUTING is one of the three sections SoAI reloads in place, so a save under it usually applies at once: SoAI rebuilds the routing configuration and hands it to the queue, the scheduler, the executor, the capacity service, and the Plugin Guardian. The exceptions matter more than the rule.

MODELS.ROUTING.HTTP_CLIENT_LIMITS and MODELS.ROUTING.HTTP_CLIENT_TIMEOUTS record a pending restart, because the shared HTTP client is built once at startup, and SoAI compares them against the values the process started with rather than against the previous save, so restoring them clears the pending restart they created. Lowering MODELS.ROUTING.HEALTH_CHECKS.NUM_PLANNER_WORKERS logs that it will apply on restart, while raising it spawns the extra workers immediately.

A second group is easier to miss, because those settings are read once while SoAI starts and never picked up by a reload, and since they live under a reloadable section the WebUI reports nothing. Restart the process yourself after changing MODELS.ROUTING.DURABLE_QUEUE_HARD_LIMIT_TASKS, MODELS.ROUTING.MIN_FREE_DISK_BYTES_FOR_ACCEPT, MODELS.ROUTING.ACCEPTANCE_DB_BUSY_TIMEOUT_SEC, MODELS.ROUTING.PLUGIN_PREFETCH_WINDOW_DEFAULT, or MODELS.ROUTING.HTTP_CLIENT_TRUST_ENV.

Everything under SYSTEM.INACTIVITY sits outside the reloadable set and does record a pending restart. Configuration scopes and restart behavior covers the reload rules for the rest of the configuration.

Bound what SoAI accepts

MODELS.ROUTING.TASK_QUEUE_MAX_SIZE defaults to 1000 and bounds the in-memory scheduling queue. A full queue produces an explicit refusal rather than an unbounded wait, which Resilience and self-healing describes. The value applies live, although shrinking it while the queue is fuller than the new limit only takes the limit down to the current occupancy and logs that it did. Keep it at 1 or above. Neither this key nor MODELS.ROUTING.MAX_CONCURRENT_PLUGINS has a minimum in the settings schema, so a 0 saves without complaint and then fails when SoAI rebuilds the routing configuration, which makes it a way to break routing rather than a way to remove a cap.

Durable admission is checked before any of that, inside the database write that records the task. MODELS.ROUTING.DURABLE_QUEUE_HARD_LIMIT_TASKS caps non-terminal durable tasks and its factory value of 0 disables that ceiling, MODELS.ROUTING.MIN_FREE_DISK_BYTES_FOR_ACCEPT defaults to 268435456, or 256 MiB, measured on the volume that holds the system database, and MODELS.ROUTING.ACCEPTANCE_DB_BUSY_TIMEOUT_SEC gives that write 30 seconds to reach the database before acceptance fails. Set a positive durable limit when the instance needs a queue ceiling, keep the three aligned with storage and recovery capacity, and restart after changing any of them, because accepting more durable work than the instance can bring to a truthful terminal state is worse than refusing it.

MODELS.ROUTING.HTTP_ASYNC_ACCEPT_DEFAULT is false, so an HTTP inference request holds its connection unless the client sends Prefer: respond-async. An asynchronous acceptance answers 202 with a body carrying the accepted status and the task id, plus a Preference-Applied: respond-async header, and the caller collects the result later. Turning the default on applies it to streaming requests as well and there is no header that asks for the synchronous behavior back, so it suits automation that polls and breaks a client waiting on the response body. A request that asks for its completion to be stored stays synchronous either way.

Limit what runs at once

The plugin limit has a shortcut in the WebUI: open Plugins, select Configure concurrent plugins on the Concurrent plugins card, change Maximum concurrent plugins, and select Save. It accepts a whole number from 1 through 1000, and saving it is a configuration change like any other, so it needs the same permission. Persistent plugins do not consume the limit, and Capacity planning explains how SoAI looks for a safe eviction candidate when every slot is occupied.

The per-plugin task limit is the one to read carefully, because a loaded plugin that declares its own concurrency wins over the configured value, which then applies only to a plugin that declares none. Every bundled plugin declares one: 1 for Ollama, MeloTTS, and Whisper, 2 for CTranslate2, 4 for Embedding and llama.cpp, 16 for vLLM, and 100 for External. Open Plugins, select the plugin, and read Max concurrent in Plugin information to see what a given plugin is actually enforcing.

MODELS.ROUTING.PROMPT_QUEUING is false and adds a fourth cap when you turn it on, admitting at most 4 tasks at a time out of the priority queue and into scheduling. That slot count is fixed and has no configuration path, so treat the setting as on or off rather than as a number you can size.

Override one plugin or one owner type

Use MODELS.ROUTING.HEALTH_CHECKS.MAX_CONCURRENT_TASKS_PER_PLUGIN.PER_PLUGIN only for a measured plugin-specific difference, and only for a plugin that leaves the limit to SoAI. In Advanced settings, search for that complete path and enter a JSON object keyed by plugin name:

{"my-plugin": 2}

An owner is the thing SoAI counts running work against, and there are four kinds: conversation, http_request, mcp_client, and system. MODELS.ROUTING.OWNER_RUNNING_LIMITS.PER_OWNER_TYPE takes an object keyed by those names, so you can give API traffic a different ceiling from WebUI conversations without touching the default. A limit of 0 or less removes the check for that owner type, which is rarely what you want, since the limit exists so one account, script, or key cannot hold every running slot.

Size each plugin's queue

Each plugin also has its own bounded queue in front of its workers, and overflowing it refuses the dispatch rather than growing. SoAI divides MODELS.ROUTING.TASK_QUEUE_MAX_SIZE by the number of plugins to get a share, raises that share to MODELS.ROUTING.HEALTH_CHECKS.PLUGIN_QUEUE_MIN_SIZE, which is 16, or to the plugin's concurrency limit multiplied by MODELS.ROUTING.HEALTH_CHECKS.PLUGIN_QUEUE_BURST_FACTOR, which is 4, whichever is largest, and caps the result at the global budget. Raise the burst factor when a plugin with a small concurrency limit needs to absorb bursts, and raise the minimum size when many plugins are dividing one queue budget between them.

The worker count follows the concurrency limit rather than the queue: a plugin runs one dispatcher worker per unit of concurrency, capped by MODELS.ROUTING.HEALTH_CHECKS.PLUGIN_QUEUE_MAX_WORKERS at 32, so External with its declared 100 gets 32 workers rather than 100. A plugin that declares unlimited concurrency instead gets MODELS.ROUTING.HEALTH_CHECKS.PLUGIN_QUEUE_UNLIMITED_WORKERS, which is 8.

Keep one busy plugin from crowding the others

MODELS.ROUTING.HEALTH_CHECKS.FAIR_DISPATCH_ENABLED is true and engages only when every plugin slot is already occupied and another plugin is waiting for capacity. While that holds, a non-persistent plugin that has reached its share stops receiving new dispatches until the waiting plugins get theirs. MODELS.ROUTING.HEALTH_CHECKS.FAIR_DISPATCH_MAX_PER_PLUGIN is 32, but the share is the lower of that ceiling and the plugin's own concurrency limit, so on a default instance the plugin limit is what binds and raising the ceiling changes nothing. Persistent plugins are never throttled this way, and turning fairness off lets the priority order run unmodified, which favors whichever plugin is already warm.

Plan and prefetch queued work

MODELS.ROUTING.HEALTH_CHECKS.NUM_PLANNER_WORKERS defaults to 16 and sets how many workers pull tasks from the priority queue and turn them into scheduling decisions. The minimum is 1. Raising it helps only when planning is the bottleneck rather than the backends, and it is the one control here where the direction of the change matters.

MODELS.ROUTING.PLUGIN_PREFETCH_WINDOW_DEFAULT defaults to 8 and limits how many claimed durable items SoAI keeps in memory per priority class ahead of execution. A larger window gives the scheduler more to choose from at the cost of holding more leases, and the drain loop reads it at startup along with its lease and sweep intervals, so the change lands on the next restart.

Set the deadlines an inference must meet

Several controls under HEALTH_CHECKS bound ordinary inference rather than probes, and these are the ones that decide when a slow backend produces a timeout instead of an answer.

MODELS.ROUTING.HEALTH_CHECKS.NON_STREAMING_TIMEOUT_SEC defaults to 300.0 and bounds a non-streaming attempt. MODELS.ROUTING.HEALTH_CHECKS.STREAMING_IDLE_TIMEOUT_SEC also defaults to 300.0 and bounds the gap between streamed chunks, so it governs a stalled stream rather than a slow one.

The deadline for the first streamed chunk is not fixed, because a long prompt legitimately takes longer to prefill. SoAI divides the prompt token count by MODELS.ROUTING.HEALTH_CHECKS.STREAMING_PREFILL_MIN_TOKENS_PER_SEC, which defaults to 8.0, adds MODELS.ROUTING.HEALTH_CHECKS.STREAMING_PREFILL_OVERHEAD_SEC, which is 60.0, and uses whichever is larger, that result or the non-streaming timeout. A per-request timeout from the client caps the outcome. Lower the assumed rate if a host needs more time to prefill long prompts. Existing configured values stay unchanged after an update. The admitted execution deadline covers dispatch and the wait for meaningful output across retries; configuration changes apply to later requests. Keepalives and empty chunks do not count as meaningful output.

Handle disconnects and duplicate work

MODELS.ROUTING.CANCEL_ON_CLIENT_DISCONNECT defaults to false. When it is on and a streaming client goes away, SoAI publishes a cancellation for that task, so the disconnect becomes a cancellation outcome instead of work that continues with nobody reading it. Each stream reads the setting as it opens, so the change reaches new streams at once. Leave it off when a caller is expected to reconnect or to collect accepted asynchronous work later, and remember that the request still ends only as fast as the selected plugin honors cancellation.

MODELS.ROUTING.DEDUPLICATION_ENABLED defaults to true, letting eligible identical requests share one lead execution while the rest wait on its result. Streaming requests are excluded outright. By default the match covers the request payload, the resolved execution targets, the priority, and the parameter snapshot, and MODELS.ROUTING.DEDUPLICATION_KEYS narrows the payload half to named fields when a field that does not affect the output keeps identical work apart. Keep ordinary admission limits either way, because deduplication reduces only duplicate work.

Release idle model resources

SYSTEM.INACTIVITY.MODELS.ENABLED is off by default, and what SoAI counts as activity is worth knowing before you turn it on. The monitor watches for a model load, an inference request, or an embedding request, but only the MCP paths publish those two request events, so a conversation in the WebUI and a call to the compatible API leave the timer running. On an instance whose traffic arrives that way, the clock measures time since the last model load rather than time since the last request.

When the setting is on, SoAI checks every SYSTEM.INACTIVITY.CHECK_INTERVAL_SEC seconds, 15 by default, and acts once the instance has been idle for SYSTEM.INACTIVITY.MODELS.TIMEOUT_MINUTES, which is 10. The action is the same force stop the administrator command performs rather than a quiet unload, so it cancels running work and drains the queue, finalizing those tasks as failed with the reason that all plugins are being stopped, and only then stops the plugins. Read that together with the activity rule above and leave the setting off unless the deployment is genuinely idle between sessions or its traffic arrives over MCP.

System inactivity is a separate mechanism, also off by default, and it reads the same activity signal. SYSTEM.INACTIVITY.SYSTEM.TIMEOUT_MINUTES is 30, SYSTEM.INACTIVITY.SYSTEM.ACTION accepts shutdown or unload_models, and shutdown is the factory value, so enabling SYSTEM.INACTIVITY.SYSTEM.ENABLED without changing the action stops the SoAI application after 30 minutes. It stops SoAI rather than the host, which is a separate operation described in System power and application lifecycle, and recovery means starting the service again. Choosing unload_models runs the same force stop that model inactivity uses, and system inactivity is evaluated first, so its action replaces the model action rather than running beside it.

Verify a change

1. Record completed throughput, latency distribution, queue depth, rejections, plugin concurrency, memory, and provider limits.
2. Change one control, and restart only if that control needs it.
3. Repeat the same workload through steady state and recovery.
4. Keep the change only if terminal outcomes and operating margin are still acceptable.

Related documentation

• SoAI system prompt
• Capacity planning
• Resilience and self-healing
• Make SoAI faster on low-end servers
• Configuration scopes and restart behavior
• System power and application lifecycle
• Monitoring and observability