Monitoring and observability

Use the SoAI WebUI to check SoAI's state, workload, hardware, and logs. The monitoring pages draw from the same live services, but each one answers a different operational question. Use the HTTP API reference when a script needs the compatible client APIs or the System API.

Choose a monitoring page

A Regular user account does not receive SYSTEM_STATUS_READ or HARDWARE_READ by default, although an Administrator with ACL_ADMIN can grant either action to the Standard user ACL group. In the WebUI, select the Settings button at the right of the header, select Settings, then ACL, expand Standard user in Access control policy, enable the required action, and select Save. LOG_ACCESS, HW_PROCESS_VIEW, and HW_GPU_TUNING remain administrator-only actions. See Access control for the full access policy.

Use the monitoring pages

Dashboard

Open Dashboard from the side navigation for a broad view of the instance. Its movable sections can include Status, Requests, Plugins, Models, Hardware widgets, Network, Storage, Logs, Image, Memo, Clock, Throughput, and Quick actions. SoAI OS also adds Capabilities for host-level capability checks. SYSTEM_STATUS_READ grants access to Dashboard but does not override the permission required by an individual card, so a user may open the page without seeing every protected card.

Select Edit layout in the page header to rearrange or hide sections. Hardware and Metrics use the same layout control. To disable layout editing, select Settings, Theme, Dashboard, then enable Lock layouts.

Open Plugins when a card identifies a problem with one backend or circuit, and open Models when the problem is limited to a model. Use Operational runbooks for recovery procedures after you identify the affected resource.

Hardware

1. Open Hardware from the side navigation.
2. Review the CPU, memory, storage, network, and GPU data reported by the host.
3. In Chart, choose Device and Metric, then select a chart type and time range.

An OHLC or Heikin Ashi chart also provides Interval to choose how much time each candle represents. The page shows History is disabled by the backend. when history collection is off, while the values and controls that appear depend on the detected device, driver, and vendor tooling. Continue with Hardware monitoring for chart, history, and export details, or GPU setup and troubleshooting when telemetry or controls are missing.

GPU control appears only for an account with HW_GPU_TUNING. It shows No GPU controls available when no detected GPU reports an actionable control; otherwise each panel contains only the controls supported by that GPU. Read GPU controls and tuning before changing power, clocks, fan settings, or saved slots.

Metrics

Open Metrics from the side navigation. In Chart, select Requests, Latency, Tokens, Token throughput, or Queue depth, then choose the chart type and time range. Request distribution, Plugin usage, Model usage, System performance, and Frontend telemetry provide more focused views, while More opens Advanced metrics. API key usage appears only for an Administrator, and loading its data also requires OPENAI_API_ADMIN.

Select Export to open a preview, then download the metrics-history CSV. Available history intervals and the metrics stored in history come from the server's current metrics configuration.

Drag or swipe the chart to move through time, and use the mouse wheel or a pinch gesture to inspect a different span. Moving into older history loads stored samples until the oldest available reading. The selected time range follows the visible span, including while earlier samples load.

Logs

1. Open Logs from the Settings button at the right of the header. If Logs has been added to the sidebar, that link opens the same page.
2. Select Core or an available plugin source.
3. Choose the number of visible entries, then use Increase text size or Decrease text size if needed.

The visible-line choices are 100, 250, 500, 1,000, 2,000, and 5,000. With no saved preference, the page defaults to 250 lines, and the browser remembers the selected source and line count. Clear empties the browser view but does not delete a log file, and new stream entries continue to arrive. The page follows incoming entries while you remain at the bottom of the view. If server logging is disabled, no log sources are available.

The main log directory and file name come from OBSERVABILITY.LOGGING.LOGS_PATH and OBSERVABILITY.LOGGING.MAIN_LOG. By default, SoAI rotates the main log daily at local midnight, retains 5 rotated files, and leaves compression off. The settings are OBSERVABILITY.LOGGING.MAIN_LOG_BACKUP_COUNT and OBSERVABILITY.LOGGING.LOGGING_SYSTEM.MAIN_LOG_COMPRESS_LOGS.

Connection freshness and recovery

Live monitoring uses the authenticated WebSocket at /api/v1/system/ws, with no polling fallback. During a disconnect or recovery, the WebUI can retain the last valid snapshot while its connection state changes and it requests fresh data. Treat retained values as last-known state, not current state, until an update arrives. Logs shows Reconnecting to log stream... while its stream reconnects.

If live data does not recover, check the browser's WebSocket connection and any proxy or firewall that handles the WebUI connection. Use Troubleshooting for the WebUI and real-time diagnostic checks.

Sampling, history, and retention

To change these settings, an Administrator with CONFIG_PATCH selects the Settings button at the right of the header, selects Settings, then Advanced. Metrics and logging settings are under Observability, while hardware collection is under System and Hardware. Select Save after an edit, then select Restart SoAI when the WebUI reports that the change requires a restart.

Live broadcasts and stored history are independent, so disabling metrics history does not stop current Metrics updates. An empty OBSERVABILITY.METRICS.METRICS_HISTORY.TRACKED_METRICS list uses SoAI's built-in history set for total, completed, and failed requests, generated tokens, request latency, queue size, and effective token throughput.

Related documentation

• Administration overview
• Resilience and self-healing
• Tasks and notifications
• Scheduler and runtime tuning
• Reset and account recovery