SoAI stays usable under load and during failures because it refuses to hold more work than it can finish, and because every accepted request exists as a durable record before anything schedules it. Failures stay where they happen: a dispatch circuit stops new work from reaching a failing plugin, while the Plugin Guardian watches runtime health and performs bounded recovery. The WebUI reports those two states separately, since either one can make a plugin unavailable for a different reason.
Admission under load
An inference request becomes a durable task and queue item in the database before the scheduler ever sees it, which
is what lets accepted work outlive a crash or a restart. Acceptance can also fail, and it fails before SoAI takes
responsibility for the request: free space below
MODELS.ROUTING.MIN_FREE_DISK_BYTES_FOR_ACCEPT
returns a disk-space error, and a positive
MODELS.ROUTING.DURABLE_QUEUE_HARD_LIMIT_TASKS
that the durable queue has already reached returns a capacity error.
Past acceptance the request enters a bounded in-memory priority queue. When that queue is full a new request waits up to 10 seconds for a slot, and if none frees in that window SoAI answers 503 Service Unavailable with a queue-full error, marked transient so a client that retries on transient failures can send it again. That 10-second admission wait is fixed and not configurable. Either outcome is explicit, so a request holds a durable record or it received a refusal it can act on.
Reaching the queue is not the same as running. SoAI limits how many local backends run at once, how many tasks one
plugin runs in parallel, and how many tasks one owner can have running, where an owner is an account, an
automation, or an API key, so a single heavy client cannot occupy every slot. Waiting requests gain priority as
they age, after 25 seconds for standard requests
(MODELS.ROUTING.STANDARD_PRIORITY_AGING_SEC) and 120 seconds for flex requests
(MODELS.ROUTING.FLEX_PRIORITY_AGING_SEC), which keeps low-priority work from starving behind a steady stream of higher-priority traffic.
Scheduler and runtime tuning holds the controls and their factory values, and
Capacity planning covers choosing them for a workload.
SoAI has been tested through bursts of thousands of simultaneous requests across one plugin and across several, and the instance stays available and usable: the WebUI keeps responding, accepted work keeps reaching a terminal state, and requests beyond the admission limits receive a capacity error rather than an unbounded wait. Longer request timeouts let more queued requests finish when the workload justifies them. A queue absorbs a burst and does not add capacity, so size the deployment for completed throughput rather than for queue depth.
Retries and rerouting
An attempt that fails for a reason the retry policy treats as transient, such as a transport error or an upstream
5xx, is retried up to 3 times with exponential backoff that starts at 1 second, grows to a 30-second ceiling, and
carries jitter so retries from concurrent requests do not land together. A provider that answers 429 gets a longer
budget of 10 attempts, and SoAI honors the Retry-After header when the provider sends one, still
capped at 30 seconds per wait. A failure the policy does not classify as retryable ends the request
instead of spending the budget on it.
Virtual models add a second path, because the router can select another eligible target when one fails, without interrupting the connected client. Virtual models, routing, and failover describes target scoring and failover behavior.
Recovery after a restart
Workers claim durable queue items on leases that last 30 seconds
(MODELS.ROUTING.DURABLE_QUEUE_LEASE_TTL_SEC) and a sweep runs every 5 seconds
(MODELS.ROUTING.DURABLE_QUEUE_RECOVERY_SWEEP_SEC) to reclaim items whose lease expired, so work that was in flight when its worker died returns to the queue
instead of staying claimed by a process that no longer exists.
At startup SoAI inspects the tasks still marked active and resolves each one. A task that still carries its orchestration context is requeued and runs again, a task that lost it is finalized as failed and reports that the server restarted while the task was in flight, and a task whose cancellation was already requested is finalized as cancelled with its recorded reason. When systemd recorded that the previous service invocation was stopped by an out-of-memory kill, a failed task last updated before that kill reports that system memory was exhausted instead. The same pass repeats periodically while SoAI runs, which covers tasks orphaned by a worker failure rather than by a restart.
The WebUI handles a restart from the other side. It shows that SoAI is offline or restarting, asks you to keep the
page open, and reconnects on its own with a delay that starts at 1 second and grows to at most 30. A script
can check /api/v1/system/health, which reports ok or degraded and does not
require authentication; the HTTP API reference documents it with the other management
endpoints.
After an unclean shutdown
SoAI writes a marker file named soai_process.dirty in
SYSTEM.PATHS.SYSTEM_DATA
when the process starts and removes it only after a clean shutdown has finished. Finding that marker at the next
start is how SoAI knows the previous run ended without one, which is the case after a crash, a host power loss, a
forced host restart, or a process killed from a terminal. It then runs a
SQLite PRAGMA quick_check over the core database before the
application starts serving, because the run that ended abruptly may have been in the middle of a write.
That check reads the whole database rather than sampling it, so on a large instance it can take several minutes, and the WebUI and the API are not available for any of them. Startup is not stuck: the lifecycle log records that the previous run did not shut down cleanly when the check begins, and records its successful completion before startup continues. Wait for it. Stopping SoAI or power-cycling the host while it runs leaves the marker in place, so the next start finds an unclean shutdown again and repeats the same check from the beginning, and each interruption also adds another unclean shutdown to recover from.
When SoAI runs as a systemd system service, startup logs the previous process's shutdown cause when it
can be determined. If it reports an out-of-memory kill, free RAM or reduce loaded models, context length, batch size,
or concurrency before repeating the workload. A signal alone does not establish memory exhaustion; unavailable
shutdown information is reported as unknown.
If the check fails, SoAI stops instead of continuing, and it does not modify the damaged database on its own. Treat that as a data-recovery situation rather than a startup problem, and follow Recover a database that failed its integrity check.
An unclean shutdown has a second consequence away from the database. When GPU settings were applied before the crash, the same start returns power, fan, and clock control to safe defaults and clears Apply at boot on every saved device, so a tuning profile that may have contributed to the crash is not reapplied on the way back up. GPU controls and tuning covers the saved slots and when to re-enable the boot profile.
Circuit breakers
Each plugin has a circuit breaker for inference dispatch, and it starts closed. SoAI counts qualifying failures
inside the window set by
MODELS.ROUTING.HEALTH_CHECKS.FAILURE_WINDOW_SEC, 1,440 seconds by default, and drops failures older than that window from the count. Reaching
MODELS.ROUTING.HEALTH_CHECKS.FAILURE_THRESHOLD, 3 by default, opens the circuit and blocks new dispatch to that plugin, while a success clears the
accumulated count.
An open circuit holds until
MODELS.ROUTING.HEALTH_CHECKS.RECOVERY_TIMEOUT_SEC
has passed since the last failure, 1,440 seconds by default, and then moves to half-open and admits one probe. A
successful probe closes the circuit and clears its history, and a failed probe opens it again and restarts the
cooldown.
SoAI writes changed breaker state to the database every 15 seconds. Startup resets persisted open and half-open circuits to closed, so a restarted backend evaluates plugin health again rather than inheriting an old verdict.
Plugin Guardian
The Guardian schedules health checks about every
MODELS.ROUTING.HEALTH_CHECKS.INTERVAL_SEC
seconds, 20 by default, with jitter spreading them across time. It pings idle plugins, watches long-running work,
and detects plugins left in a transitional state. The default ping timeout is 15 seconds and the default
transitional-state timeout is 900 seconds. It also detects repeated state changes through
MODELS.ROUTING.HEALTH_CHECKS.FLAPPING_WINDOW_SEC
and
MODELS.ROUTING.HEALTH_CHECKS.FLAPPING_MIN_TRANSITIONS.
Recovery attempts are counted consecutively rather than inside a time window. A successful health ping clears the
counter, and so does resetting the circuit from Plugins. When the attempts exceed
MODELS.ROUTING.HEALTH_CHECKS.MAX_RECOVERY_ATTEMPTS, 3 by default, SoAI stops the backend where the runtime requires it, trips the circuit, and marks the plugin
quarantined.
Each recovery attempt purges that plugin's queued and running tasks before it revalidates or restarts the backend, so requests already in flight on the affected plugin end with a failure you can see instead of waiting on a runtime that is being rebuilt. Work for other plugins is untouched.
What users see
In the SoAI WebUI, open Dashboard for overall health and current queue depth, and open Plugins to see a plugin's runtime status and circuit state. Select the plugin to open Plugin information, where the circuit state, failure count, last failure, and timeout are shown; a plugin with an open circuit also carries a banner saying the switch resets it. For the output behind a failed health check, open Logs from the Settings menu at the right of the header and select the Core or plugin source.
An open circuit and a quarantine are not interchangeable. An open circuit is a dispatch decision based on recent failures, while a quarantine is a runtime state the plugin reaches after recovery escalation or an integrity failure. Package compatibility, backend state, worker state, quarantine, and circuit state stay separate so the WebUI reports the actual failure instead of collapsing it into one generic status.
Correct the reported cause before you clear either state, since resetting a circuit while its failure remains only sends work back into it. Reset an open circuit gives the WebUI procedure, including the distinct paths for an open circuit, an ordinary quarantine, and clone integrity recovery, and Troubleshooting maps a queue-full 503 and other symptoms to their causes.