How SoAI works

SoAI sits between the applications that request model work and the engines or providers that perform it. A client sends an OpenAI-compatible or Anthropic-compatible request to SoAI. The orchestrator checks the request against the model catalog, current plugin state, capabilities, and capacity, then dispatches it to an eligible target. The response returns through the same API, so the client does not need a separate endpoint for every engine.

The names you will see

A plugin is SoAI's adapter package for an engine or provider. A backend is the process or service that handles inference. A model is a catalog entry associated with a plugin. The External plugin represents a remote OpenAI-compatible provider; it does not install that provider on the host. A virtual model gives several model targets one client-facing ID. The Glossary defines these and other terms used throughout the documentation.

Plugins are distributed as .soaiplugin packages. They declare their capabilities and lifecycle operations through the public plugin contract. SoAI runs plugin workers in their own processes and keeps package, backend, model, and runtime state distinct. See Plugin architecture when you need the boundary details.

What happens to a request

  1. A client calls an endpoint under /v1, such as /v1/chat/completions or /v1/responses.
  2. SoAI validates the request and resolves the requested catalog or virtual model ID into an execution plan.
  3. The scheduler admits the work when the queue, owner, model, and plugin limits allow it.
  4. The selected plugin starts or prepares its backend when the target needs it, then sends the request to that backend.
  5. SoAI returns the result through the selected compatibility contract. OpenAI-compatible streams use SSE data records and end with data: [DONE]; Anthropic-compatible streams use Anthropic event names and end with message_stop.

The WebUI and management scripts use the separate /api/v1 REST surface. Live WebUI events and interactive sessions, including the browser terminal, use /api/v1/system/ws. MCP server mode uses Streamable HTTP at /mcp. The HTTP API reference lists the surfaces and their authentication rules.

Routing, queues, and idle resources

SoAI persists admitted inference work in a durable queue before the scheduler dispatches it. The scheduler considers target availability, queue state, model residency, declared capabilities, and routing configuration. With MODELS.ROUTING.DEDUPLICATION_ENABLED enabled, eligible matching non-streaming requests can wait for one lead execution and receive its result. Streaming requests are not deduplicated.

A virtual model puts two or more catalog models behind one client-facing ID. Clients keep using that ID while the scheduler selects an eligible member at dispatch. See Virtual models, routing, and failover for creation, eligibility, strategies, cooldowns, and permissions. The catalog can list a model before its local backend is running; SoAI prepares it when a request needs it. Read Install and manage plugins and Resilience and self-healing for operational controls.

Model unloading and system inactivity actions are separate opt-in settings. When enabled, SoAI can unload idle model resources or perform the configured system action after its timeout. The switches are SYSTEM.INACTIVITY.MODELS.ENABLED and SYSTEM.INACTIVITY.SYSTEM.ENABLED. Both are disabled by default. In the SoAI WebUI, an administrator with the Configuration changes (CONFIG_PATCH) permission selects the Settings button at the right of the header, chooses Settings, selects Advanced in the page header, opens System, edits the fields, then selects Save.

Where tools and knowledge fit

The assistant layer uses the same model request path, then adds tool execution around a conversation turn. Built-in tools can work with files, browsers, the knowledge base, hardware, and connected services. SoAI can also connect to external MCP servers as a host, or expose its own tools and resources to an MCP client at /mcp. The MCP overview explains the two directions and their permissions.

RAG collections use storage and search paths distinct from the model catalog. A conversation can retrieve indexed passages and provide them as context to the selected model. That keeps model routing independent from document storage and lets you change the inference target without rebuilding the collection. See Knowledge and document collections (RAG) for collection behavior.

A skill is a reusable set of instructions available to the chats where you select it. Its description helps the model decide when to load it; an explicit name request loads it directly when allowed. A saved prompt inserts text into a message, while a plugin connects a model backend or provider. Skills use the tools and permissions already available to the conversation.

Related documentation