Operational runbooks

These runbooks cover routine administration and recovery through the SoAI WebUI wherever it provides the required control, while the HTTP API reference covers the same management surfaces for scripts.

Starting and stopping SoAI

If SoAI is stopped, start it with the launcher for its installation. Once it is running, use the WebUI for routine power actions. Open Power to show application and system controls over your current page. Select Restart application to reload SoAI, or Shut down application to stop it while leaving the host running. Read the confirmation before continuing because either action disconnects current WebUI sessions.

Each action opens one confirmation with its description, restart notice when relevant, delay and any available force setting. Application restart defaults to no delay; other actions default to 10 seconds. Review the options and select the button named for the action to submit it. A restart-required notice can require immediate execution. Selecting an action from search or a saved Power link only focuses its control.

Use Back, Cancel, Esc or click the background to leave Power. When a confirmation is open, Esc closes that confirmation first; Cancel returns to the action chooser. After acceptance, the countdown's Cancel button requests server cancellation. Failed cancellation keeps the operation visible. Once execution begins, the operation screen shows its progress and reconnection guidance.

After a restart, reconnect to the WebUI and confirm that the Dashboard reports a ready application state. After a shutdown, start SoAI with the same launcher used for the installation.

When a saved HTTP or HTTPS change requires a different address, the restart overlay shows the planned address. Select Open new address to open it in another tab, or Copy address to keep it. The original tab retains its drafts. The destination may still be starting after the connection is interrupted; normal certificate checks and sign-in apply. If a proxy, changed listener or fallback port makes the address uncertain, follow the manual address guidance and use the address configured for the server or reverse proxy.

Shutdown stops new work and allows API requests to drain for the budget set by SYSTEM.SHUTDOWN.SERVER_DRAIN_TIMEOUT_SEC. It then stops services and resolves accepted tasks before closing shared resources. Wait for shutdown to finish before replacing application files or starting another instance.

Scripts can use the power control endpoints in the System API. Restart follows the same teardown sequence and then relaunches the application.

Recover file uploads and attachment reads

Check an uncertain or cancelled upload

1. Open Files at the selected upload destination. If the result reports an unverified final state, inspect that destination before retrying: the file may already have been saved.
2. Review every retained file named in the result. Keep files you need, or remove an unwanted file through the normal Delete action after checking its contents.
3. For a batch, distinguish successful files, failed or uncertain files, and unattempted items. Unattempted items were never published by that batch; retry them separately after resolving the uncertain item.
4. Choose a different name or destination for a retry when an existing entry must be kept. Uploads do not overwrite an existing file or folder.

Retry a temporarily unavailable attachment read

A busy verification worker or a verification timeout can make one attachment temporarily unavailable while the message text and other usable attachments remain readable. Reload the conversation to retry the saved attachment read. A temporary verification notice does not mean the file changed or that its earlier processing was lost. If the source is missing, changed or inaccessible on the next read, check its current contents and access before attaching it again. Keep the original message while resolving the source.

Recover an unsuccessful application update

If the WebUI does not return after an update, check the installed application's status with the matching launcher from Launcher and CLI reference. Keep the error reference and any diagnostics folder named by the update dialog. Read application_updater.log and application_updater.stderr.log in the configured logs directory (OBSERVABILITY.LOGGING.LOGS_PATH) to identify the failure, and correct its cause before retrying.

Wait for installation or rollback to finish before starting another update. If SoAI reports unfinished recovery, follow the reported recovery action; do not delete transaction files or rollback data to force another installation. If that action cannot complete, preserve the diagnostics and follow Getting help and reporting an issue.

When SoAI is available again, create a backup through Settings > Settings > Backup > Create backup as an Administrator with Backup administration (BACKUP_ADMIN). If it cannot start, preserve the installation and its configured data folders before attempting manual recovery. Keep any existing backups. A copy of a failed installation preserves evidence and recoverable files, but does not replace a verified backup.

These manual procedures apply to SoAI Core. Use a newer release for the same platform and architecture, and authenticate its artifact with Download and verify SoAI releases. Do not use a generated source snapshot or a Core installer on SoAI OS. For an appliance, follow Updating SoAI for application updates and SoAI OS package updates for host packages.

Retry with the Windows setup program

Run the launcher commands below in an administrator Command Prompt from the existing installation folder.

1. Download and verify the new release's setup executable for Windows.
2. Stop the existing instance with soai.exe --stop from its installation folder and confirm soai.exe --status reports it stopped. Closing a window alone does not confirm that shutdown has finished.
3. Run the verified setup executable and approve its administrator prompt.
4. Select the existing installation folder. Check the path even if setup has selected it automatically; choosing a different folder would not update the existing installation.
5. Wait for setup to finish, then start SoAI from the finish page or the installed launcher.

Setup updates the application in place and preserves its data directory. You do not need to uninstall SoAI or remove its data first. If setup fails, keep the diagnostics folder it identifies and address that error before another attempt.

Update Core from a complete archive

1. Download and verify the new release's complete archive for the installed platform and architecture.
2. Extract it into a separate staging folder outside the existing installation and open its SoAI folder. Do not unpack the archive over the installed files.
3. Stop the existing installation using its launcher or service and confirm that it is stopped. Use the commands below for the default paths, replacing them if your installation is elsewhere.

From the extracted release's SoAI folder, run the appropriate installer below with --target set to the existing installation folder. The examples use default paths. On Linux, run as the account that owns the existing installation, using sudo for a root-owned target. On Windows, use an administrator Command Prompt. Keep the target path quoted when it contains spaces.

Wait for the installer to finish successfully before starting SoAI again. It replaces managed application files while preserving user data, saved configuration, models, and plugin selections. If it refuses a running target, stop that instance before retrying; if it reports unfinished recovery, complete the recovery action first. Start a service installation through its existing service, or use the installed launcher for an installation without a service.

After either procedure, sign in and confirm the installed version on Settings > Updates. Check that accounts, conversations, settings, models, and plugins are present, then send a test message. If startup or a migration fails, keep the reported error and logs rather than resetting the data or installing an older release over it. Follow Restore if recovery requires a verified backup.

Recover a database that failed its integrity check

After an unclean shutdown SoAI runs a SQLite PRAGMA quick_check over the core database before it starts serving, and a failed check ends startup at that point. The WebUI never becomes available, so the backup and restore controls inside it are out of reach and this recovery happens in a terminal against the files on disk. SoAI does not repair or rewrite the damaged database on its own, so it stays exactly as the failed run left it until you act on it. After an unclean shutdown describes when the check runs.

The commands below use the default Linux installation folder /opt/soai, where the database is data/database/soai.db, resolved from SYSTEM.PATHS.SYSTEM_DATA and DATA.DATABASE.PATHS.SYSTEM_DB. macOS installs to /Applications/SoAI and Windows to the folder chosen during setup, and the layout inside is the same on all three. A service installation runs as root and its database files are readable only by their owner, so run every command as that account or through sudo.

You also need the SQLite command-line shell, which a SoAI installation does not include. Debian and Ubuntu install it with sudo apt install sqlite3, macOS already ships it, and Windows needs the precompiled command-line tools from the download page at SQLite. Run sqlite3 --version and confirm 3.29 or later, because the .recover command this runbook depends on does not exist in earlier releases.

Stop SoAI and copy the database

Stop the service before copying or replacing database files, and wait for shutdown to finish. A failed or interrupted integrity check runs again at the next startup. For an installation you start yourself, use its edition and platform launcher from Launcher and CLI reference.

sudo systemctl stop soai.service

# An installation without the service
cd /opt/soai
./soai.sh --stop
./soai.sh --status

# SoAI OS
./soai_os/soai_os.sh --stop
./soai_os/soai_os.sh --status

Now take two copies of the database and its write-ahead log companions, onto a filesystem with room for both. The copy under original is the one version of your data that nothing later in this runbook touches, so leave it closed even to read it. Everything else happens under work.

mkdir -p /var/tmp/soai-db-rescue/original /var/tmp/soai-db-rescue/work
cd /opt/soai/data/database
cp -p soai.db soai.db-wal soai.db-shm /var/tmp/soai-db-rescue/original/
cp -p soai.db soai.db-wal soai.db-shm /var/tmp/soai-db-rescue/work/

The -wal and -shm files exist only while a write session was open, so cp reporting one of them as missing is normal and the copy is still complete. Keeping the whole set together matters because the -wal file holds committed transactions that are not in the database file yet.

Confirm the failure on the copy

Run the check SoAI ran, followed by the slower full check, against the working copy rather than the instance. Opening a database that still has a -wal file makes SQLite replay that log into it, which is a write, so both the instance and the original copy stay closed.

cd /var/tmp/soai-db-rescue/work
sqlite3 soai.db "PRAGMA quick_check;"
sqlite3 soai.db "PRAGMA integrity_check;"

A healthy database answers ok to both, and anything else is the damage that stopped startup. The critical message SoAI wrote before it exited is in the main log, named by OBSERVABILITY.LOGGING.LOGS_PATH and OBSERVABILITY.LOGGING.MAIN_LOG.

Restarting SoAI does not repair database corruption. A failed or interrupted check leaves soai_process.dirty in place and verification runs again on the next startup. Keep the instance stopped while you investigate the copy and prepare a verified replacement. Shutdown backup and database-dependent shutdown reconciliation remain paused while recovery verification is incomplete.

Restore the database from a backup

A verified backup is the only path that returns a database whose contents are known to be good. Backups live under DATA.BACKUP.BACKUPS_PATH as one directory per backup named soai_backup_<date>_<time>, each holding its database at data/database/soai.db beside a manifest.json recording every captured file with its size and SHA-256. SoAI writes the backup as a consistent database copy rather than a raw file copy, so it arrives without a -wal companion to carry over.

Each manifest also records the integrity result SoAI observed while writing that backup, which narrows the candidates without reading a single database. Verify the one you choose anyway, since the manifest describes the file as it was written rather than as it is now. That verification reads the whole database, so on a large instance give it time.

cd /opt/soai/data/backups
grep -l '"sqlite_integrity_ok": true' soai_backup_*/manifest.json
sqlite3 soai_backup_20260903_142824/data/database/soai.db "PRAGMA integrity_check;"

Work backwards from the newest backup until one answers ok, then put that file in place. The damaged files are already preserved in /var/tmp/soai-db-rescue, so removing them from the instance costs nothing. The -wal and -shm files belong to the database you are removing, and SQLite can replay a leftover log into whatever database takes its place, so they go with it.

cd /opt/soai
rm -f data/database/soai.db data/database/soai.db-wal data/database/soai.db-shm
cp -p data/backups/soai_backup_20260903_142824/data/database/soai.db data/database/soai.db

Do not substitute a symbolic link for the database file, because SoAI rejects one. It also restores the restrictive file mode on the database, its companions, and their directory each time it opens them, so the copy needs no permission fixing afterwards.

This returns the database alone. Configuration, plugin configuration, GPU settings, the encryption key that protects secrets stored in the database, and user files are all still whatever the failed instance left behind, and they can be newer than the database you just installed. Once SoAI is running again, use Restore on the same backup to bring all of it back to one consistent point.

Salvage the database when no backup is usable

The SQLite .recover command reads whatever pages it can still parse and writes them out as SQL statements, which you load into a new file. Run it against the working copy and never in place over a database you want to keep, allow room for a second full-size database, and expect a large instance to take a while. The target file has to be absent or empty, so the rm below clears anything an earlier attempt left behind.

cd /var/tmp/soai-db-rescue/work
rm -f soai-recovered.db
sqlite3 soai.db ".recover" | sqlite3 soai-recovered.db
sqlite3 soai-recovered.db "PRAGMA integrity_check;"
sqlite3 soai-recovered.db "SELECT name FROM sqlite_master WHERE name = 'lost_and_found';"

On Windows, redirect through a file instead of piping between the two commands.

del recovered.sql soai-recovered.db
sqlite3.exe soai.db ".recover" > recovered.sql
sqlite3.exe soai-recovered.db < recovered.sql
sqlite3.exe soai-recovered.db "PRAGMA integrity_check;"

The recovered file passing integrity_check means only that it is now a well-formed database, not that it still holds your data. Rows on a page SQLite could not read are gone, and rows it could read but could not return to their own table land in a lost_and_found table, which is what the third command looks for. If .recover produces nothing usable, .dump is worth one attempt, although it stops at the first page it cannot read rather than working around it. This step can cost data permanently, so decide what the recovered file is worth before you install it.

cd /opt/soai
rm -f data/database/soai.db data/database/soai.db-wal data/database/soai.db-shm
cp -p /var/tmp/soai-db-rescue/work/soai-recovered.db data/database/soai.db

Return the instance to service

Start SoAI with the launcher for the installation, or start the service again. The launcher starts SoAI when you give it no command at all.

cd /opt/soai
./soai.sh

# SoAI OS
./soai_os/soai_os.sh

# Service installations
sudo systemctl start soai.service

If the recovery marker remains, this start verifies the replacement database before serving requests. The file you installed must still answer ok before it goes into place. Sign in, confirm that the Dashboard reports a ready application state, and confirm that the users, conversations, models, and plugin configuration you expect are all there. A restored backup should be complete as of the moment it was taken. A recovered database is the one to read carefully, because that is where .recover may have dropped rows.

Corruption reported here is usually a symptom of the host rather than of the database. Before the instance goes back into normal use, check for an unstable overclock or undervolt, failing memory, and storage device or filesystem errors, because a host that damaged the database once will do it again. Backup covers the policy that decides how much work a future failure can cost.

Working with plugins

Open Plugins to manage a plugin and its backend. Use the plugin switch to enable or disable it. Use Stop plugin when you need to halt its running backend without uninstalling anything. For a local engine, use Install backend or Manage backend. Use Edit configuration for plugin-specific settings and GPU binding.

Disable a plugin when you need the scheduler to keep it offline during maintenance. Use Stop plugin to end its current backend process. A stopped plugin remains enabled, so SoAI may start it again for later work. Follow the task in the Tasks panel and wait for a terminal result before continuing. The plugin stop command has a 300-second default budget, set by MODELS.ROUTING.HEALTH_CHECKS.COMMAND_TIMEOUTS_SEC.PLUGIN_STOP.

Reset an open circuit

An open circuit blocks new dispatch after repeated qualifying failures. Reset it from Plugins after you correct the cause.

1. Open Plugins and select the affected plugin to open Plugin information.
2. Read the status and circuit-breaker details. Then correct the reported backend, provider, configuration, storage, or package problem.
3. Return to the plugin card. Its circuit-breaker banner states that the switch will reset the circuit. Turn on the switch and wait for the reset task to finish.
4. Confirm that the banner is gone. If the plugin owns a model, open that model from Models, select Test model, and confirm that the test succeeds.

Do not reset a circuit before fixing the underlying failure. The default threshold is three consecutive failures, controlled by MODELS.ROUTING.HEALTH_CHECKS.FAILURE_THRESHOLD.

Make a quarantined plugin available again

A quarantined plugin with no circuit-breaker banner has a locked switch and no reset control on its card. Restarting SoAI clears an ordinary recovery quarantine and runs the startup checks again. A clone integrity quarantine remains in place until the failed clone is recovered or removed.

1. Open Plugins and select the quarantined plugin to open Plugin information.
2. Confirm its status, then open Logs from the Settings menu at the right of the header and select that plugin's source. Check Tasks for a failed clone operation.
3. Correct the backend, provider, configuration, storage, package, or clone problem shown there.
4. Open Power and select Restart application. Read the confirmation and select Restart application again.
5. Wait for the WebUI to reconnect, then return to Plugins.
6. Confirm that the plugin no longer shows Quarantined. If its switch is off, turn it on. If the backend needs attention, use Install backend or Manage backend on its card.
7. If the plugin owns a model, open that model from Models, select Test model, and confirm that the test succeeds.

If the plugin still shows Quarantined after restart and Tasks contains a failed clone operation, treat it as clone integrity recovery. A circuit reset does not clear that state, and startup preserves it while the clone remains inconsistent. The WebUI has no force-clear action. Keep the clone if you need its files for diagnosis or recovery.

If you do not need the damaged clone, open Plugins, select Delete plugin on that clone, review the deletion warning, and select Delete. Then open the source plugin, select Clone plugin, choose whether to copy its models, and select Clone. Verify that the replacement appears on Plugins without a quarantine status.

Model downloads, discovery, and deletion

Open Models and select Add model to download a model. Choose the plugin, enter the model download ID, select a variant when the plugin offers one, and select Download model. Follow progress in the Tasks panel. SYSTEM.RUNTIME.STAY_OFFLINE blocks downloads when enabled.

To refresh the catalog after you place model files in the models folder, open Models, select Add model, switch to Add model manually, and select Discover models. SoAI also runs discovery after backend installation, model downloads, and plugin enable operations.

To remove a model, open it from Models, select Delete model, and read the warning. Confirm when you intend to remove the listed data. The owning plugin decides which local files or provider records the operation removes.

Watching logs and cancelling tasks

Open Logs from the Settings menu at the right of the header to inspect current Core and plugin output. Select the Core or plugin source, choose how many lines to display, and keep the view open while reproducing a failure. Use the Tasks panel in the WebUI header to inspect running operations.

To cancel an operation, open Tasks and select Cancel running task when that control is available for the entry. Cancellation is a request, so wait for the final status before retrying. Stop all activity appears only when the panel contains a stoppable non-persistent plugin or visible cancelable operation; persistent plugins cannot be stopped from this panel. See Tasks and notifications for the panel controls and outcomes.

Authenticating a management script

Most System API routes at /api/v1/* require authentication. Explicit public routes include system health, information, limits, and license data; the setup-status endpoint; login; MCP OAuth metadata and callback routes; and the Telegram and WhatsApp messaging webhooks. The System API schema lists the available routes.

The WebUI creates soai-http-token and soai-http-csrf cookies over HTTP. Over HTTPS, it uses __Host-soai-token and __Host-soai-csrf. A script that signs in through the WebUI login flow must retain both cookies. For cookie-authenticated POST, PUT, PATCH, and DELETE requests to protected routes, copy the CSRF cookie value into the X-SoAI-CSRF header.

API keys created by selecting the Settings button at the right of the header, then Settings > OpenAI, apply to OpenAI-compatible and Anthropic-compatible routes under /v1/*, not the System API. After setup, those compatible routes are open while no API keys exist. Creating the first key makes a key mandatory for every /v1/* call. Send it as a Bearer token to OpenAI-compatible endpoints; Anthropic-compatible endpoints also accept x-api-key.

Related documentation

• Monitoring and observability
• Resilience and self-healing
• Backup
• Restore
• Troubleshooting