Using LiteLLM with SoAI

LiteLLM can place another OpenAI-compatible gateway in front of SoAI and can also aggregate its MCP server. Use this only when you need LiteLLM's key, budget, or routing layer; clients can otherwise connect directly.

Connect and use the model API

  1. In the SoAI WebUI, select the Settings button at the right of the header, choose Settings, and open OpenAI. This tab appears only to an Administrator with API key management (OPENAI_API_ADMIN). Select Create key and copy the key when it is shown. Open Models and copy the exact model ID; this requires Model read access (MODEL_READ).
  2. For MCP, select the Settings button at the right of the header, choose Settings, then open MCP > MCP tokens. The MCP tab appears only to an Administrator with MCP administration (MCP_ADMIN), and creating a token also requires MCP usage (MCP_USE). Changing the MCP runtime, server mode, or exposed capabilities under MCP configuration also requires Configuration changes (CONFIG_PATCH).

Add a LiteLLM model

In config.yaml, route an alias to the server with LiteLLM's custom OpenAI provider:

model_list:
  - model_name: soai-model
    litellm_params:
      model: openai/YOUR_MODEL_ID
      api_base: os.environ/SOAI_BASE_URL
      api_key: os.environ/SOAI_OPENAI_API_KEY
export SOAI_BASE_URL='https://SOAI_HOST:5090/v1'
export SOAI_OPENAI_API_KEY='YOUR_SOAI_API_KEY'

LiteLLM resolves both os.environ references from the proxy's own environment, so set them before starting the proxy with that configuration, then send an OpenAI Chat Completions request to LiteLLM using the alias soai-model.

Use the LiteLLM interface

  1. Sign in to LiteLLM's Admin UI and open Models + Endpoints.
  2. Select Add Model, choose the custom OpenAI provider, and enter the model ID, base URL, and key.
  3. Save, run the model health/test action, and use the alias from a LiteLLM client.

Add the MCP server to LiteLLM

  1. Open MCP Servers in the LiteLLM UI and select Add New MCP Server.
  2. Enter https://SOAI_HOST:5090/mcp, choose Streamable HTTP, set authentication to Bearer token, and supply the personal MCP token.
  3. Save and confirm that LiteLLM discovers the selected tools. Grant access only to the required LiteLLM keys, teams, or users.
mcp_servers:
  soai:
    url: https://SOAI_HOST:5090/mcp
    transport: http
    auth_type: bearer_token
    authentication_token: os.environ/SOAI_MCP_TOKEN

LiteLLM clients authenticate to LiteLLM separately. Do not confuse a LiteLLM virtual key with either the upstream SoAI API key or MCP token. Store the MCP token through LiteLLM's protected configuration or Admin UI rather than committing it to source control.

Related documentation