update documentation for sdk, vault, gitea integration
This commit is contained in:
228
docs/deployment.md
Normal file
228
docs/deployment.md
Normal file
@@ -0,0 +1,228 @@
|
||||
# Sol — Kubernetes Deployment
|
||||
|
||||
Sol runs as a single-replica Deployment in the `matrix` namespace. SQLite is the persistence backend, so only one pod can run at a time (Recreate strategy).
|
||||
|
||||
## resource relationships
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph OpenBao
|
||||
vault[("secret/sol<br/>matrix-access-token<br/>matrix-device-id<br/>mistral-api-key")]
|
||||
end
|
||||
|
||||
subgraph "matrix namespace"
|
||||
vss[VaultStaticSecret<br/>sol-secrets]
|
||||
secret[Secret<br/>sol-secrets]
|
||||
cm[ConfigMap<br/>sol-config<br/>sol.toml + system_prompt.md]
|
||||
pvc[PVC<br/>sol-data<br/>1Gi RWO]
|
||||
deploy[Deployment<br/>sol]
|
||||
init[initContainer<br/>fix-permissions]
|
||||
pod[Container<br/>sol]
|
||||
end
|
||||
|
||||
vault --> |VSO sync| vss
|
||||
vss --> |creates| secret
|
||||
vss --> |rolloutRestartTargets| deploy
|
||||
deploy --> init
|
||||
init --> pod
|
||||
secret --> |env vars| pod
|
||||
cm --> |subPath mounts| pod
|
||||
pvc --> |/data| init
|
||||
pvc --> |/data| pod
|
||||
```
|
||||
|
||||
## manifests
|
||||
|
||||
All manifests are in `infrastructure/base/matrix/`.
|
||||
|
||||
### Deployment (`sol-deployment.yaml`)
|
||||
|
||||
```yaml
|
||||
strategy:
|
||||
type: Recreate # SQLite requires single-writer
|
||||
replicas: 1
|
||||
```
|
||||
|
||||
**initContainer** — `busybox` runs `chmod -R 777 /data && mkdir -p /data/matrix-state` to ensure the nonroot distroless container can write to the Longhorn PVC.
|
||||
|
||||
**Container** — `sol` image (distroless/cc-debian12:nonroot)
|
||||
|
||||
- Resources: 256Mi request / 512Mi limit memory, 100m CPU request
|
||||
- `enableServiceLinks: false` — avoids injecting service env vars that could conflict
|
||||
|
||||
**Environment variables** (from Secret `sol-secrets`):
|
||||
|
||||
| Env Var | Secret Key |
|
||||
|---------|-----------|
|
||||
| `SOL_MATRIX_ACCESS_TOKEN` | `matrix-access-token` |
|
||||
| `SOL_MATRIX_DEVICE_ID` | `matrix-device-id` |
|
||||
| `SOL_MISTRAL_API_KEY` | `mistral-api-key` |
|
||||
|
||||
Fixed env vars:
|
||||
|
||||
| Env Var | Value |
|
||||
|---------|-------|
|
||||
| `SOL_CONFIG` | `/etc/sol/sol.toml` |
|
||||
| `SOL_SYSTEM_PROMPT` | `/etc/sol/system_prompt.md` |
|
||||
|
||||
**Volume mounts:**
|
||||
|
||||
| Mount | Source | Details |
|
||||
|-------|--------|---------|
|
||||
| `/etc/sol/sol.toml` | ConfigMap `sol-config` | subPath: `sol.toml`, readOnly |
|
||||
| `/etc/sol/system_prompt.md` | ConfigMap `sol-config` | subPath: `system_prompt.md`, readOnly |
|
||||
| `/data` | PVC `sol-data` | read-write |
|
||||
|
||||
### PVC (`sol-deployment.yaml`, second document)
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: sol-data
|
||||
namespace: matrix
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
```
|
||||
|
||||
Uses the default StorageClass (Longhorn).
|
||||
|
||||
### VaultStaticSecret (`vault-secrets.yaml`)
|
||||
|
||||
```yaml
|
||||
apiVersion: secrets.hashicorp.com/v1beta1
|
||||
kind: VaultStaticSecret
|
||||
metadata:
|
||||
name: sol-secrets
|
||||
namespace: matrix
|
||||
spec:
|
||||
vaultAuthRef: vso-auth
|
||||
mount: secret
|
||||
type: kv-v2
|
||||
path: sol
|
||||
refreshAfter: 60s
|
||||
rolloutRestartTargets:
|
||||
- kind: Deployment
|
||||
name: sol
|
||||
destination:
|
||||
name: sol-secrets
|
||||
create: true
|
||||
overwrite: true
|
||||
```
|
||||
|
||||
The `rolloutRestartTargets` field means VSO will automatically restart the Sol deployment when secrets change in OpenBao.
|
||||
|
||||
Three keys synced from OpenBao `secret/sol`:
|
||||
|
||||
- `matrix-access-token`
|
||||
- `matrix-device-id`
|
||||
- `mistral-api-key`
|
||||
|
||||
## `/data` mount layout
|
||||
|
||||
```
|
||||
/data/
|
||||
├── sol.db SQLite database (conversations + agents tables, WAL mode)
|
||||
└── matrix-state/ Matrix SDK sqlite state store (E2EE keys, sync tokens)
|
||||
```
|
||||
|
||||
Both are created automatically. The initContainer ensures directory permissions are correct for the nonroot container.
|
||||
|
||||
## secrets in OpenBao
|
||||
|
||||
Store secrets at `secret/sol` in OpenBao KV v2:
|
||||
|
||||
```sh
|
||||
# Via sunbeam seed (automated), or manually:
|
||||
openbao kv put secret/sol \
|
||||
matrix-access-token="syt_..." \
|
||||
matrix-device-id="DEVICE_ID" \
|
||||
mistral-api-key="..."
|
||||
```
|
||||
|
||||
These are synced to K8s Secret `sol-secrets` by the Vault Secrets Operator.
|
||||
|
||||
## build and deploy
|
||||
|
||||
```sh
|
||||
# Build only (local Docker image)
|
||||
sunbeam build sol
|
||||
|
||||
# Build + push to registry
|
||||
sunbeam build sol --push
|
||||
|
||||
# Build + push + deploy (apply manifests + rollout restart)
|
||||
sunbeam build sol --push --deploy
|
||||
```
|
||||
|
||||
The Docker build cross-compiles to `x86_64-unknown-linux-gnu` on macOS. The final image is `gcr.io/distroless/cc-debian12:nonroot` (~30MB).
|
||||
|
||||
## startup sequence
|
||||
|
||||
1. Initialize `tracing_subscriber` with `RUST_LOG` env filter (default: `sol=info`)
|
||||
2. Load config from `SOL_CONFIG` path
|
||||
3. Load system prompt from `SOL_SYSTEM_PROMPT` path
|
||||
4. Read 3 secret env vars (`SOL_MATRIX_ACCESS_TOKEN`, `SOL_MATRIX_DEVICE_ID`, `SOL_MISTRAL_API_KEY`)
|
||||
5. Build Matrix client with E2EE sqlite store, restore session
|
||||
6. Connect to OpenSearch, ensure archive + memory indices exist
|
||||
7. Initialize Mistral client
|
||||
8. Build components: Personality, ConversationManager, ToolRegistry, Indexer, Evaluator, Responder
|
||||
9. Backfill conversation context from archive (if `backfill_on_join` enabled)
|
||||
10. Open SQLite database (fallback to in-memory on failure)
|
||||
11. Initialize AgentRegistry + ConversationRegistry (load persisted state from SQLite)
|
||||
12. If `use_conversations_api` enabled: ensure orchestrator agent exists on Mistral server
|
||||
13. Backfill reactions from Matrix room timelines
|
||||
14. Start background index flush task
|
||||
15. Start Matrix sync loop
|
||||
16. If SQLite failed: send `*sneezes*` to all joined rooms
|
||||
17. Log "Sol is running", wait for SIGINT
|
||||
|
||||
## monitoring
|
||||
|
||||
Sol uses `tracing` with structured fields. Default log level: `sol=info`.
|
||||
|
||||
Key log events:
|
||||
|
||||
| Event | Level | Fields |
|
||||
|-------|-------|--------|
|
||||
| Response sent | info | `room`, `len`, `is_dm` |
|
||||
| Tool execution | info | `tool`, `id`, `args` |
|
||||
| Engagement evaluation | info | `sender`, `rule`, `relevance`, `threshold` |
|
||||
| Memory extraction | debug | `count`, `user` |
|
||||
| Conversation created | info | `room`, `conversation_id` |
|
||||
| Agent restored/created | info | `agent_id`, `name` |
|
||||
| Backfill complete | info | `rooms`, `messages` / `reactions` |
|
||||
|
||||
Set `RUST_LOG=sol=debug` for verbose output including tool results, evaluation prompts, and memory details.
|
||||
|
||||
## troubleshooting
|
||||
|
||||
**Pod won't start / CrashLoopBackOff:**
|
||||
|
||||
```sh
|
||||
sunbeam logs matrix/sol
|
||||
```
|
||||
|
||||
Common causes:
|
||||
- Missing secrets (env vars not set) — check `sunbeam k8s get secret sol-secrets -n matrix -o yaml`
|
||||
- ConfigMap not applied — check `sunbeam k8s get cm sol-config -n matrix`
|
||||
- PVC not bound — check `sunbeam k8s get pvc -n matrix`
|
||||
|
||||
**SQLite recovery failure (*sneezes*):**
|
||||
|
||||
If Sol sends `*sneezes*` on startup, it means the SQLite database at `/data/sol.db` couldn't be opened. Sol falls back to in-memory state. Check PVC mount and file permissions:
|
||||
|
||||
```sh
|
||||
sunbeam k8s exec -n matrix deployment/sol -- ls -la /data/
|
||||
```
|
||||
|
||||
**Matrix sync errors:**
|
||||
|
||||
Sol auto-joins rooms on invite (3 retries with exponential backoff). If it can't join, check homeserver connectivity and access token validity.
|
||||
|
||||
**Agent creation failure:**
|
||||
|
||||
If the orchestrator agent can't be created, Sol falls back to model-only conversations (no agent). Check Mistral API key and quota.
|
||||
Reference in New Issue
Block a user