sol/README.md

# sol

a virtual librarian for Matrix. sol lives in your chat rooms, archives conversations in OpenSearch, and responds with the help of Mistral AI — with end-to-end encryption, tool use, per-user memory, and a multi-agent architecture.

sol is built by [sunbeam studios](https://sunbeam.pt) as part of our self-hosted collaboration stack for a three-person game studio.

## what sol does

- **Matrix presence** — joins rooms, reads the vibe, decides when to speak. direct messages always get a response; in group rooms, sol evaluates relevance before jumping in.
- **message archive** — every message is indexed in OpenSearch with full-text and semantic search. sol can search its own archive via tools.
- **tool use** — Mistral calls tools mid-conversation: archive search, room context retrieval, room info, and a sandboxed TypeScript/JavaScript runtime (deno_core) for computation.
- **per-user memory** — sol remembers things about the people it talks to. memories are extracted automatically after conversations, injected into the system prompt before responding, and accessible from scripts via `sol.memory.get/set`.
- **user impersonation** — sol acts on behalf of users when calling external services. PATs are auto-provisioned via admin APIs and stored securely in OpenBao (Vault). OIDC-to-service username mappings handle identity mismatches.
- **gitea integration** — first domain agent (sol-devtools): list repos, search issues, create issues, list PRs, get file contents — all as the requesting user.
- **multi-agent architecture** — an orchestrator agent with personality + tools + web search. domain agent delegation is dynamic — only active agents appear in instructions. agent state persisted in SQLite with instructions hash for automatic recreation on prompt changes.
- **conversations API** — persistent conversation state per room via Mistral's Conversations API, with automatic compaction at token thresholds. per-message context headers inject timestamps, room info, and memory notes.
- **multimodal** — m.image messages are downloaded from Matrix via mxc://, converted to base64 data URIs, and sent as `ContentPart::ImageUrl` to Mistral vision models.
- **reactions** — sol can react to messages with emoji when it has something to express but not enough to say.
- **E2EE** — full end-to-end encryption via matrix-sdk with sqlite state store.

## architecture

```mermaid
flowchart TD
    subgraph Matrix
        sync[Matrix Sync Loop]
    end

    subgraph Engagement
        eval[Evaluator]
        rules[Rule Checks]
        llm_eval[LLM Evaluation<br/>ministral-3b]
    end

    subgraph Response
        legacy[Legacy Path<br/>manual messages + chat completions]
        convapi[Conversations API Path<br/>ConversationRegistry + agents]
        tools[Tool Execution]
    end

    subgraph Persistence
        sqlite[(SQLite<br/>conversations + agents)]
        opensearch[(OpenSearch<br/>archive + memory)]
    end

    sync --> |message event| eval
    eval --> rules
    rules --> |MustRespond| Response
    rules --> |no rule match| llm_eval
    llm_eval --> |MaybeRespond| Response
    llm_eval --> |React| sync
    llm_eval --> |Ignore| sync
    legacy --> tools
    convapi --> tools
    tools --> opensearch
    legacy --> |response text| sync
    convapi --> |response text| sync
    sync --> |archive| opensearch
    convapi --> sqlite
    sync --> |memory extraction| opensearch
```

## source tree

```
src/
├── main.rs                 entrypoint, Matrix client setup, backfill, orchestrator init
├── sync.rs                 event loop — messages, reactions, redactions, invites
├── config.rs               TOML config (5 sections) with serde defaults
├── context.rs              ResponseContext — per-message sender identity threading
├── conversations.rs        ConversationRegistry — room→conversation mapping, SQLite-backed
├── persistence.rs          SQLite store (WAL mode, 2 tables: conversations, agents)
├── agent_ux.rs             AgentProgress — reaction lifecycle (🔍→⚙️→✅) + thread posting
├── matrix_utils.rs         message extraction, reply/edit/thread detection, image download
├── archive/
│   ├── schema.rs           ArchiveDocument, OpenSearch index mapping
│   └── indexer.rs          batched indexing, reactions, edits, redactions
├── brain/
│   ├── conversation.rs     sliding-window context per room (configurable group/DM windows)
│   ├── evaluator.rs        engagement decision (MustRespond/MaybeRespond/React/Ignore)
│   ├── personality.rs      system prompt templating ({date}, {room_name}, {members}, etc.)
│   └── responder.rs        both response paths, tool iteration loops, memory loading
├── memory/
│   ├── schema.rs           MemoryDocument, index mapping
│   ├── store.rs            query (topical), get_recent, set — OpenSearch operations
│   └── extractor.rs        post-response fact extraction via ministral-3b
├── agents/
│   ├── definitions.rs      orchestrator config + 8 domain agent definitions (dynamic delegation)
│   └── registry.rs         agent lifecycle with instructions hash staleness detection
├── sdk/
│   ├── mod.rs              SDK module root
│   ├── vault.rs            OpenBao/Vault client (K8s auth, KV v2 read/write/delete)
│   ├── tokens.rs           TokenStore — Vault-backed secrets + SQLite username mappings
│   └── gitea.rs            GiteaClient — typed Gitea API v1 with PAT auto-provisioning
└── tools/
    ├── mod.rs              ToolRegistry — 12 tool definitions + dispatch (5 core + 7 gitea)
    ├── search.rs           archive search (keyword + semantic via embedding pipeline)
    ├── room_history.rs     context around a timestamp or event
    ├── room_info.rs        room listing, member queries
    ├── script.rs           deno_core sandbox with sol.* host API, TS transpilation
    ├── devtools.rs         Gitea tool handlers (repos, issues, PRs, files)
    └── bridge.rs           ToolBridge — generic async handler map for future SDK integration
```


## engagement pipeline

```mermaid
sequenceDiagram
    participant M as Matrix
    participant S as Sync Handler
    participant E as Evaluator
    participant LLM as ministral-3b
    participant R as Responder

    M->>S: m.room.message
    S->>S: archive message
    S->>S: update conversation context
    S->>E: evaluate(sender, body, is_dm, recent)

    alt own message
        E-->>S: Ignore
    else @mention or matrix.to link
        E-->>S: MustRespond (DirectMention)
    else DM
        E-->>S: MustRespond (DirectMessage)
    else "sol" or "hey sol"
        E-->>S: MustRespond (NameInvocation)
    else no rule match
        E->>LLM: relevance evaluation (JSON)
        LLM-->>E: {relevance, hook, emoji}
        alt relevance >= spontaneous_threshold (0.85)
            E-->>S: MaybeRespond
        else relevance >= reaction_threshold (0.6) + emoji
            E-->>S: React (emoji)
        else below thresholds
            E-->>S: Ignore
        end
    end

    alt MustRespond or MaybeRespond
        S->>S: check in-flight guard
        S->>S: check cooldown (15s default)
        S->>R: generate response
    end
```

## response generation

Sol has two response paths, controlled by `agents.use_conversations_api`:

### legacy path (`generate_response`)

1. Apply response delay (random within configured range)
2. Send typing indicator
3. Load memory notes (topical query + recent backfill, max 5)
4. Build system prompt via `Personality` (template substitution: `{date}`, `{room_name}`, `{members}`, `{memory_notes}`, `{room_context_rules}`, `{epoch_ms}`)
5. Assemble message array: system → context messages (with timestamps) → trigger (multimodal if image)
6. Tool iteration loop (up to `max_tool_iterations`, default 5):
   - If `finish_reason == ToolCalls`: execute tools, append results, continue
   - If text response: strip "sol:" prefix, return
7. Fire-and-forget memory extraction

### conversations API path (`generate_response_conversations`)

1. Apply response delay
2. Send typing indicator
3. Format input: raw text for DMs, `<@user:server> text` for groups
4. Send through `ConversationRegistry.send_message()` (creates or appends to Mistral conversation)
5. Function call loop (up to `max_tool_iterations`):
   - Execute tool calls locally via `ToolRegistry`
   - Send `FunctionResultEntry` back to conversation
6. Extract assistant text, strip prefix, return

## tool system

| Tool | Parameters | Description |
|------|-----------|-------------|
| `search_archive` | `query` (required), `room`, `sender`, `after`, `before`, `limit`, `semantic` | Search the message archive (keyword or semantic) |
| `get_room_context` | `room_id` (required), `around_timestamp`, `around_event_id`, `before_count`, `after_count` | Get messages around a point in time or event |
| `list_rooms` | *(none)* | List all rooms Sol is in with names and member counts |
| `get_room_members` | `room_id` (required) | Get members of a specific room |
| `run_script` | `code` (required) | Execute TypeScript/JavaScript in a sandboxed deno_core runtime |

### run_script sandbox

The script runtime is a fresh V8 isolate per invocation with:

- **TypeScript support** — code is transpiled via `deno_ast` before execution
- **Timeout** — configurable via `behavior.script_timeout_secs` (default 5s), enforced by V8 isolate termination
- **Heap limit** — configurable via `behavior.script_max_heap_mb` (default 64MB)
- **Output** — `console.log()` + last expression value, truncated to 4096 characters
- **Temp filesystem** — sandboxed `sol.fs.read/write/list` with path traversal protection
- **Network** — `sol.fetch(url)` restricted to `behavior.script_fetch_allowlist` domains

Host API (`sol.*`):

```typescript
sol.search(query, opts?)         // search message archive
sol.rooms()                      // list joined rooms → [{name, id, members}]
sol.members(roomName)            // get room members → [{name, id}]
sol.fetch(url)                   // HTTP GET (allowlisted domains only)
sol.memory.get(query?)           // retrieve memories relevant to query
sol.memory.set(content, category?) // save a memory note
sol.fs.read(path)                // read file from sandbox
sol.fs.write(path, content)      // write file to sandbox
sol.fs.list(path?)               // list sandbox directory
```

All `sol.*` methods are async — use `await`.

## memory system

### extraction (post-response, fire-and-forget)

After each response, a background task sends the exchange to `ministral-3b` with a structured extraction prompt. The model returns `{"memories": [{"content": "...", "category": "preference|fact|context"}]}`. Categories are normalized via `normalize_category()` — valid categories are `preference`, `fact`, `context`; anything else falls back to `general`.

### storage (OpenSearch)

Each memory is a `MemoryDocument` with: `id`, `user_id`, `content`, `category`, `created_at`, `updated_at`, `source` (`"auto"` or `"script"`). The index name defaults to `sol_user_memory`. User isolation is enforced at the Rust level via `user_id` filtering on all queries.

### pre-response loading

Before generating a response, the responder loads up to 5 memories:

1. **Topical query** — semantic search against the trigger message
2. **Recent backfill** — if fewer than 3 topical results, fill remaining slots with most recent memories

Memory notes are injected into the system prompt as a `## notes about {display_name}` block with instructions to use them naturally without mentioning their existence.

## archive

Every message event is archived as an `ArchiveDocument` in OpenSearch:

- **Batch indexing** — messages are buffered and flushed periodically (`opensearch.batch_size` default 50, `opensearch.flush_interval_ms` default 2000)
- **Embedding pipeline** — configurable via `opensearch.embedding_pipeline` for semantic search
- **Edit tracking** — `m.replace` events update the original document's content
- **Redaction** — `m.room.redaction` sets `redacted: true` on the original
- **Reactions** — `m.reaction` events append `{sender, emoji, timestamp}` to the document's reactions array
- **Backfill** — on startup, conversation context is backfilled from the archive; reactions are backfilled from Matrix room timelines (last 500 events per room)

## agent architecture

```mermaid
stateDiagram-v2
    [*] --> CheckMemory: startup
    CheckMemory --> CheckServer: agent_id in SQLite?
    CheckMemory --> SearchByName: not in SQLite

    CheckServer --> Ready: exists on Mistral server
    CheckServer --> SearchByName: gone from server

    SearchByName --> Ready: found by name
    SearchByName --> Create: not found

    Create --> Ready: agent created
    Ready --> [*]
```

### orchestrator

The orchestrator agent carries Sol's full personality (system prompt) plus all 5 tool definitions converted to `AgentTool` format. It's created on startup if `agents.use_conversations_api` is enabled. Temperature: 0.5.

### domain agents (8 definitions)

| Agent | Domain |
|-------|--------|
| `sol-observability` | Metrics, logs, dashboards, alerts (Prometheus, Loki, Grafana) |
| `sol-data` | Full-text search, object storage (OpenSearch, SeaweedFS) |
| `sol-devtools` | Git repos, issues, PRs, kanban boards (Gitea, Planka) |
| `sol-infrastructure` | Kubernetes, deployments, certificates, builds |
| `sol-identity` | User accounts, sessions, OAuth2 (Kratos, Hydra) |
| `sol-collaboration` | Contacts, documents, meetings, files, email, calendars (La Suite) |
| `sol-communication` | Chat rooms, messages, members (Matrix) |
| `sol-media` | Video/audio rooms, recordings, streams (LiveKit) |

Domain agents are defined in `agents/definitions.rs` as `DOMAIN_AGENTS` (name, description, instructions). Temperature: 0.3.

### ToolBridge

`tools/bridge.rs` provides a generic async handler map (`ToolBridge`) for mapping Mistral tool call names to handler functions. This is scaffolding for future SDK-based tool integration where domain agents will have their own tool sets.

## persistence

SQLite database at `/data/sol.db` (configurable via `matrix.db_path`), WAL mode.

### tables

**conversations** — room_id (PK), conversation_id, estimated_tokens, created_at

**agents** — name (PK), agent_id, model, created_at

### recovery behavior

On startup, if the database fails to open:

1. Log error
2. Fall back to in-memory SQLite (conversations won't survive restarts)
3. After sync loop starts, send `*sneezes*` to all joined rooms to signal the hiccup

## multimodal

When an `m.image` message arrives:

1. Extract media source from event (`MessageType::Image`)
2. Download bytes from Matrix media API via `matrix_sdk::media::get_media_content`
3. Base64-encode as `data:{mime};base64,{data}` URI
4. Pass to Mistral as `ContentPart::ImageUrl` alongside any text caption

Encrypted images are not supported (the `MediaSource::Encrypted` variant is skipped).

## configuration reference

Config is loaded from `SOL_CONFIG` (default: `/etc/sol/sol.toml`).

### `[matrix]`

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `homeserver_url` | string | *required* | Matrix homeserver URL |
| `user_id` | string | *required* | Bot's Matrix user ID |
| `state_store_path` | string | *required* | Path for Matrix SDK sqlite state |
| `db_path` | string | `/data/sol.db` | SQLite database for persistent state |

### `[opensearch]`

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `url` | string | *required* | OpenSearch cluster URL |
| `index` | string | *required* | Archive index name |
| `batch_size` | usize | `50` | Messages per flush batch |
| `flush_interval_ms` | u64 | `2000` | Flush interval in milliseconds |
| `embedding_pipeline` | string | `tuwunel_embedding_pipeline` | Ingest pipeline for semantic embeddings |
| `memory_index` | string | `sol_user_memory` | Memory index name |

### `[mistral]`

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `default_model` | string | `mistral-medium-latest` | Model for response generation |
| `evaluation_model` | string | `ministral-3b-latest` | Model for engagement evaluation + memory extraction |
| `research_model` | string | `mistral-large-latest` | Model for research tasks |
| `max_tool_iterations` | usize | `5` | Max tool call rounds per response |

### `[behavior]`

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `response_delay_min_ms` | u64 | `100` | Min delay before direct response |
| `response_delay_max_ms` | u64 | `2300` | Max delay before direct response |
| `spontaneous_delay_min_ms` | u64 | `15000` | Min delay before spontaneous response |
| `spontaneous_delay_max_ms` | u64 | `60000` | Max delay before spontaneous response |
| `spontaneous_threshold` | f32 | `0.85` | LLM relevance score to trigger spontaneous response |
| `reaction_threshold` | f32 | `0.6` | LLM relevance score to trigger emoji reaction |
| `reaction_enabled` | bool | `true` | Enable emoji reactions |
| `room_context_window` | usize | `30` | Messages to keep in group room context |
| `dm_context_window` | usize | `100` | Messages to keep in DM context |
| `backfill_on_join` | bool | `true` | Backfill context from archive on startup |
| `backfill_limit` | usize | `10000` | Max messages to backfill |
| `instant_responses` | bool | `false` | Skip response delays (for testing) |
| `cooldown_after_response_ms` | u64 | `15000` | Cooldown before another spontaneous response |
| `evaluation_context_window` | usize | `25` | Recent messages sent to evaluation LLM |
| `detect_sol_in_conversation` | bool | `true` | Use active/passive evaluation prompts based on Sol's participation |
| `evaluation_prompt_active` | string? | *(built-in)* | Custom prompt when Sol is in conversation |
| `evaluation_prompt_passive` | string? | *(built-in)* | Custom prompt when Sol hasn't spoken |
| `script_timeout_secs` | u64 | `5` | Script execution timeout |
| `script_max_heap_mb` | usize | `64` | V8 heap limit for scripts |
| `script_fetch_allowlist` | string[] | `[]` | Domains allowed for `sol.fetch()` |
| `memory_extraction_enabled` | bool | `true` | Enable post-response memory extraction |

### `[agents]`

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `orchestrator_model` | string | `mistral-medium-latest` | Model for orchestrator agent |
| `domain_model` | string | `mistral-medium-latest` | Model for domain agents |
| `compaction_threshold` | u32 | `118000` | Token estimate before conversation reset (~90% of 131K context) |
| `use_conversations_api` | bool | `false` | Enable Conversations API path (vs legacy chat completions) |

## environment variables

| Variable | Required | Description |
|----------|----------|-------------|
| `SOL_MATRIX_ACCESS_TOKEN` | yes | Matrix access token |
| `SOL_MATRIX_DEVICE_ID` | yes | Matrix device ID (for E2EE state) |
| `SOL_MISTRAL_API_KEY` | yes | Mistral API key |
| `SOL_CONFIG` | no | Config file path (default: `/etc/sol/sol.toml`) |
| `SOL_SYSTEM_PROMPT` | no | System prompt file path (default: `/etc/sol/system_prompt.md`) |

## dependencies

sol talks to five external services:

- **Matrix homeserver** — [tuwunel](https://github.com/tulir/tuwunel) (or any Matrix server)
- **OpenSearch** — message archive + user memory indices
- **Mistral AI** — response generation, engagement evaluation, memory extraction, agents + web search
- **OpenBao** — secure token storage for user impersonation PATs (K8s auth, KV v2)
- **Gitea** — git hosting API for devtools agent (repos, issues, PRs)

key crates: `matrix-sdk` 0.9 (E2EE + sqlite), `mistralai-client` 1.1.0 (private registry), `opensearch` 2, `deno_core` 0.393, `rusqlite` 0.32 (bundled), `ruma` 0.12.

## building

```sh
cargo build --release
```

docker (cross-compile to x86_64 linux, vendored deps):

```sh
docker build -t sol .
```

production build + deploy:

```sh
sunbeam build sol --push --deploy
```

the Dockerfile uses a two-stage build: deps layer (cached until Cargo.toml/vendor change) → source layer (only sol code recompiles). final image is `gcr.io/distroless/cc-debian12:nonroot`.

## testing

```sh
cargo test
```

unit tests covering:

- config parsing (minimal, full, missing sections/fields, services, vault)
- conversation windowing, context management, reset_all, delete_all
- engagement rules (mention, DM, name invocation, case sensitivity, false positives)
- personality template substitution (date, room, members, memory notes, timestamps, room context rules)
- memory document serialization, extraction parsing, category normalization
- archive search query building (filters, date ranges, wildcards, room_name keyword field)
- TypeScript transpilation (basic, arrow, interface, invalid)
- sandbox path isolation (traversal, symlink escape, nested dirs)
- deno_core script execution (basic math, output capture)
- SQLite CRUD (conversations, agents, service_users, load_all, bulk delete)
- conversation message merging (DM, group, empty, single)
- context derivation (`@user:server` → `user@server`, localpart extraction)
- tool bridge registration and execution
- agent UX formatting (tool calls, result truncation)
- agent definitions (orchestrator instructions, dynamic delegation, deterministic hash)
- token expiry validation (PAT, future, past, malformed, null)
- Gitea API type deserialization (repos, issues, PRs, files)
- PAT conflict status codes and scope constants
- username mapping (OIDC → service identity)

## license

[MIT](LICENSE)