SoAI includes eight bundled plugins. Seven provide managed local backends or local media capabilities, while External connects to remote OpenAI-compatible providers. To use a local backend in the SoAI WebUI, open Plugins, select its card, open Manage backend, choose an available backend variant, and start installation. Then open Models > Add model to add a model for that plugin.
The availability of a backend variant is based on the current operating system, hardware, drivers, and required runtime. Read the compatibility message in Manage backend before installing. The capability badges on a plugin or model describe what the selected backend and model can handle; a model does not automatically support every operation its plugin can proxy.
Local inference backends
Ollama is a managed local inference backend and can also connect to Ollama Cloud. It supports chat completions, completions, Responses, embeddings, vision, tool calling, structured output, and JSON Schema when the selected model supports the operation. It can search for and download models, and its plugin can be cloned when you need a separately configured instance.
For separate local instances, select Clone plugin on the Ollama card. Give the clone a different configuration, including its GPU binding or port where appropriate, before running both instances. Use this only when the host has capacity for the additional backend and model files.
The local Ollama Responses endpoint cannot apply saved native execution settings
such as GPU offload, batch size, thread count or context length. SoAI rejects requests containing those settings
before sending inference to the endpoint. Use Chat or /v1/chat/completions with the model to apply them.
Responses requests can use supported sampling, output limits and thinking settings; explicit
reasoning.effort overrides the saved thinking setting. The remote provider path uses its provider's
request contract.
vLLM
vLLM is a managed local backend for plugin-declared GPTQ, AWQ, GGUF, and transformer models. Its capabilities include chat completions, completions, Responses, embeddings, vision, input audio, tool calling, structured output, and JSON Schema where the chosen model supports them. vLLM can also be cloned.
The bundled variants include CPU builds for Linux x64 and arm64, plus an Apple Silicon CPU build. GPU-oriented variants are available for NVIDIA CUDA on Linux, AMD ROCm on Linux x64, and an Intel XPU container on Linux x64. The Apple GPU variant and the Windows-through-WSL variant are currently marked unavailable in SoAI. Select the variant in Plugins > vLLM > Manage backend; do not rely on a generic statement that every vLLM installation requires the same operating system or accelerator.
The bundled Linux x64 CPU wheel requires AVX2; the arm64 wheel has its own requirements. If the backend reports a missing compiler or an illegal instruction, install or update it to repair its prerequisites, or select a runtime compatible with your CPU.
Llama.cpp
Llama.cpp is the bundled GGUF backend. It supports chat completions, completions, Responses, embeddings, vision, tool calling, structured output, and JSON Schema where the model supports them. SoAI can install a suitable backend variant, search and download models, and clone the plugin.
The managed variants cover CPU and selected CUDA, ROCm, Vulkan, OpenVINO, SYCL, OpenCL, and Metal paths across the platforms shown in Manage backend. Pick from that live list instead of copying a backend port or build setting from another host.
Auto chooses an artifact compatible with the selected GPUs. If discovery is incomplete, wait and retry. An explicit variant must also meet launch requirements, including those of GPUs initialized with zero model offload. Select a CPU runtime if you need to avoid GPU initialization. Compatibility can differ between operating systems.
A Snapdragon variant is available for Arm Linux hosts in both the chat and embedding plugins. It supports CPU, Adreno GPU, and Hexagon NPU execution and requires the corresponding Qualcomm runtimes. Select a compatible variant in Manage backend.
Mixed-vendor hosts can run separate compatible plugin instances or backends for NVIDIA, AMD, and Intel GPUs. CUDA, ROCm, and XPU bindings validate that every selected GPU belongs to the matching vendor. Clone a plugin when separate instances need different runtime variants or GPU bindings; one vendor-specific backend instance cannot combine CUDA, ROCm, and XPU devices. See Mixed-vendor GPU systems for the hardware boundary.
Task-specific local backends
CTranslate2 is an offline CPU neural machine translation backend, not a general-purpose chat model engine. It uses CTranslate2 model entries from the Helsinki-NLP repository and exposes translation through chat-completions and completions compatibility surfaces. It does not provide embeddings, Responses, vision, tool calling, or audio operations.
Use it when the job is translation. You can install its backend, download a translation model, and clone the plugin for an independently configured instance from the same Plugins workflow.
Llama.cpp Embeddings
Llama.cpp Embeddings is a separate GGUF embedding runtime. It uses a resident plugin slot and can be unloaded to make room. It supports the embeddings endpoint and not chat completions, completions, Responses, media, or tool calling. It can be installed, configured, searched, downloaded, and cloned like the other local model plugins.
Use this plugin when you want a dedicated local embedding model for Knowledge collections or embedding API work. In Chat, open Chat configuration > Knowledge, then choose Embedding model after the model is installed and appears in the catalog.
Melo TTS
Melo TTS is a persistent local text-to-speech backend. It supplies the audio speech capability, downloads its language packs as models, and allows one active request at a time. It is not a chat, completion, embedding, or transcription backend.
The plugin offers CPU, CUDA, Apple MPS, and container variants where the current host can use them. Choose the displayed variant in Manage backend, then add the language pack needed for the voice workflow.
Whisper
Whisper is a persistent local speech-to-text backend. It provides audio transcription and audio translation, can search for and download compatible models, and allows one active request at a time. It is not a chat, completion, embedding, image, or text-to-speech backend.
Use Plugins > Whisper > Manage backend to choose an available CPU or accelerator variant, then add a Whisper model from Models > Add model. The displayed backend variants show the choices compatible with the current platform and detected drivers.
External providers
External is a persistent connector for remote OpenAI-compatible APIs, including OpenAI, OpenRouter, Anthropic, Groq, and Together AI. It does not install, download, delete, or run a provider on the SoAI host. Instead, it stores a provider connection and imports the models that provider reports.
Open Models > Add model > Add provider, choose External, then enter the
Provider name and API URL. Enter an API key when the provider
requires one, and leave it empty for a local or open endpoint that accepts requests without a key. A
Models filter is optional. After the provider is saved, SoAI discovers models through the
provider's OpenAI-compatible /models response and applies that filter. Provider and model capability
data can vary, so confirm the imported model's capabilities before using media, tools, structured output, or
Responses features.
External leaves unset sampling settings to the provider. Values supplied in a request or saved model parameters still apply, and Chat can set its own output-token budget. Set the context length to match the provider's limit; otherwise, SoAI assumes 65,536 tokens.