Adding your first backend

Your first backend should match the work you want to do and the host that will run it. SoAI does not require one engine for every installation. Use the bundled plugin descriptions in Bundled plugins reference to choose a local engine, or add an External provider when the model already runs behind an OpenAI-compatible API.

If you used SoAI setup in the first-run wizard, check Plugins and Models before adding another backend. When Chat offers Finish SoAI setup, select it to continue the remaining steps.

Pick a starting point

  • Ollama suits a host where you already use Ollama or want SoAI to manage an Ollama process and its model lifecycle.
  • llama.cpp is the portable choice for GGUF models on Linux, Windows, or macOS (experimental).
  • vLLM offers CPU and accelerator variants for workloads that benefit from higher concurrent throughput. Open Plugins > vLLM > Manage backend to see the builds available for the current host.
  • External connects an existing OpenAI-compatible service. It does not install or supervise that provider's software.

CTranslate2, Llama.cpp Embeddings, Whisper, and Melo TTS serve translation, RAG embeddings, speech recognition, and speech generation rather than general chat. Add those plugins when your workflow needs the corresponding API capability.

Install or connect the plugin

Viewing Plugins requires Plugin read access (PLUGIN_READ). Installing a backend or adding a provider requires Plugin administration (PLUGIN_ADMIN). Viewing the model catalog requires Model read access (MODEL_READ), while downloading or changing a managed model requires Model administration (MODEL_ADMIN). The Administration overview explains accounts and ACL actions.

  1. In the SoAI WebUI, open Plugins after completing the setup wizard. A plugin card shows Install backend only when that plugin supports managed installation, is not permanently disabled, does not require a compatibility override, and has the status Not installed or an installation error.
  2. For a local engine, select Install backend, choose a build supported by your operating system and hardware, then select Start installation. SoAI reports the local installation in Tasks.
  3. For a remote service, open Models > Add model > Add provider. The Add provider tab appears when the catalog includes a plugin that supports user-managed providers. A choice in Plugin becomes usable only when it is enabled, available, compatible, and in a usable runtime state. Select External, then enter a Provider name, API URL, API key, and any Models filter.
  4. For a local engine, wait for its task in Tasks to finish. After a successful backend installation, SoAI triggers model discovery.
  5. For an External provider, SoAI creates the provider and queues model discovery; it calls GET /models at the configured API URL. Discovery is skipped for a non-local provider when SYSTEM.RUNTIME.STAY_OFFLINE is enabled.

Add a model and test it

Open Models and select Add model. Use Download model or Add model manually when the selected plugin offers that flow. The manual tab appears only for an Administrator account. Select a variant when the plugin offers one, then wait for the task to finish. A local model appears after discovery finds it; a remote model appears after provider discovery accepts it.

Open Chat, choose the new model, and send a short prompt. A successful answer confirms the complete path from the catalog through the plugin and backend. You can also call /v1/models and /v1/chat/completions from a client. See HTTP API reference for endpoint details and Authentication and API keys before sharing an endpoint with another client.

If the first request fails

Check the model status in Models, the plugin runtime in Plugins, and the operation result in Tasks. A backend can be installed while its process is stopped, loading, unavailable, or quarantined. Select the Settings button at the right of the header, then Logs, and inspect the plugin log before retrying. With the default sidebar layout, Logs is hidden, but an account with Log access (LOG_ACCESS) can open it from that menu. See Monitoring and observability for log controls, then follow Operational runbooks for circuit and quarantine recovery. If the model does not appear at all, confirm its files, model ID, plugin capability, and storage path before changing runtime settings.

Related documentation