Using AnythingLLM with SoAI

AnythingLLM can use SoAI's OpenAI-compatible model API and can attach SoAI MCP tools to AnythingLLM agents.

Prepare the server

  1. In the SoAI WebUI, select the Settings button at the right of the header, choose Settings, and open OpenAI. This tab appears only to an Administrator with API key management (OPENAI_API_ADMIN). Select Create key and copy the key when it is shown. Open Models and copy the exact model ID; this requires Model read access (MODEL_READ).
  2. For MCP, select the Settings button at the right of the header, choose Settings, then open MCP > MCP tokens. The MCP tab appears only to an Administrator with MCP administration (MCP_ADMIN), and creating a token also requires MCP usage (MCP_USE). Changing the MCP runtime, server mode, or exposed capabilities under MCP configuration also requires Configuration changes (CONFIG_PATCH).

Add the server as the language model

  1. Open AnythingLLM Settings > AI Providers > LLM.
  2. Choose OpenAI (Generic).
  3. Enter https://SOAI_HOST:5090/v1, the SoAI API key, and an exact model ID. Set the context window and token limits to the selected model's real values.
  4. Save the settings, select the provider for the workspace, and send a short chat message.

Add SoAI MCP

Add the remote server to anythingllm_mcp_servers.json in AnythingLLM's storage plugins directory. In Docker, edit the file in the mounted storage volume, not in the disposable container layer. 

{
  "mcpServers": {
    "soai": {
      "type": "streamable",
      "url": "https://SOAI_HOST:5090/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_MCP_TOKEN"
      }
    }
  }
}
  1. Open Settings > Agent Skills.
  2. Under MCP Servers, select Refresh, then confirm the restart and tool reload.
  3. Confirm the server is healthy and inspect its discovered tools.
  4. Open an Agent chat, enable the required MCP tools, and approve a harmless call.

Related documentation