docs: update CLAUDE.md, deployment, and conversations docs

CLAUDE.md: updated source layout with orchestrator, grpc, code_index, breadcrumbs modules. Deployment: added gRPC service, startup flowchart, new secrets and troubleshooting. Conversations: updated lifecycle to show orchestrator path and gRPC session keys.
2026-03-24 12:58:35 +00:00
parent 2949ea354f
commit b3a38767e0
3 changed files with 94 additions and 45 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -4,10 +4,16 @@

 ```sh
 cargo build --release                              # debug: cargo build
-cargo test                                         # 102 unit tests, no external services needed
+cargo test                                         # unit tests, no external services needed
 cargo build --release --target x86_64-unknown-linux-gnu  # cross-compile for production
 ```

+Integration tests (require `.env` with `SOL_MISTRAL_API_KEY`):
+
+```sh
+cargo test --test integration_test
+```
+
 Docker (multi-stage, vendored deps):

 ```sh
@@ -37,10 +43,12 @@ This updates the vendored sources. Commit `vendor/` changes alongside `Cargo.loc
 ## Key Architecture Notes

 - **`chat_blocking()` workaround**: The Mistral client's `chat_async` holds a `std::sync::MutexGuard` across `.await`, making the future `!Send`. All chat calls use `chat_blocking()` which runs `client.chat()` via `tokio::task::spawn_blocking`.
- **Two response paths**: Controlled by `agents.use_conversations_api` config toggle.
-  - Legacy: manual `Vec<ChatMessage>` assembly, chat completions, tool iteration loop.
-  - Conversations API: `ConversationRegistry` with persistent state (SQLite-backed), agents, function call loop.
+- **Transport-agnostic orchestrator**: The `orchestrator` module emits `OrchestratorEvent`s via broadcast channel. It has no knowledge of Matrix or gRPC. Transport bridges subscribe to events and translate to their protocol.
+- **Two input paths**: Matrix sync loop and gRPC `CodeAgent` service. Both feed `GenerateRequest` into the orchestrator.
+- **Tool dispatch routing**: `ToolSide::Server` tools execute locally in Sol. `ToolSide::Client` tools are relayed to the gRPC client (sunbeam code TUI) via oneshot channels.
+- **Conversations API**: `ConversationRegistry` with persistent state (SQLite-backed), agents, function call loop. Enabled via `agents.use_conversations_api`.
 - **deno_core sandbox**: `run_script` tool spins up a fresh V8 isolate per invocation with `sol.*` host API bindings. Timeout via V8 isolate termination. Output truncated to 4096 chars.
+- **Code indexing**: Tree-sitter symbol extraction (Rust, Python, TypeScript, Go, Java) from Gitea repos and `sunbeam code` IndexSymbols. Stored in OpenSearch `sol_code` index.

 ## K8s Context

@@ -55,17 +63,23 @@ This updates the vendored sources. Commit `vendor/` changes alongside `Cargo.loc

 ## Source Layout

- `src/main.rs` — startup, component wiring, backfill, agent recreation + sneeze
+- `src/main.rs` — startup, component wiring, gRPC server, backfill, agent recreation + sneeze
 - `src/sync.rs` — Matrix event handlers, context hint injection for new conversations
- `src/config.rs` — TOML config with serde defaults (6 sections: matrix, opensearch, mistral, behavior, agents, services, vault)
+- `src/config.rs` — TOML config with serde defaults (8 sections: matrix, opensearch, mistral, behavior, agents, services, vault, grpc)
 - `src/context.rs` — `ResponseContext`, `derive_user_id`, `localpart`
 - `src/conversations.rs` — `ConversationRegistry` (room→conversation mapping, SQLite-backed, reset_all)
 - `src/persistence.rs` — SQLite store (WAL mode, 3 tables: `conversations`, `agents`, `service_users`)
- `src/agent_ux.rs` — `AgentProgress` (reaction lifecycle + thread posting)
- `src/matrix_utils.rs` — message extraction, image download, reactions
- `src/brain/` — evaluator (full system prompt context), responder (per-message context headers + memory), personality, conversation manager
- `src/agents/` — registry (instructions hash + automatic recreation), definitions (dynamic delegation)
- `src/sdk/` — vault client (K8s auth), token store (Vault-backed), gitea client (PAT auto-provisioning)
+- `src/matrix_utils.rs` — message extraction, reply/edit/thread detection, image download
+- `src/time_context.rs` — time utilities
+- `src/tokenizer.rs` — token counting
+- `src/orchestrator/` — transport-agnostic event-driven pipeline (engine, events, tool dispatch)
+- `src/grpc/` — gRPC CodeAgent service (server, session, auth, bridge to orchestrator events)
+- `src/brain/` — evaluator (engagement decision), responder (response generation), personality, conversation manager
+- `src/agents/` — registry (instructions hash + automatic recreation), definitions (orchestrator + domain agents)
+- `src/sdk/` — vault client (K8s auth), token store (Vault-backed), gitea client (PAT auto-provisioning), kratos client
 - `src/memory/` — schema, store, extractor
- `src/tools/` — registry (12 tools), search, room_history, room_info, script, devtools (gitea), bridge
+- `src/tools/` — registry + dispatch, search, code_search, room_history, room_info, script, devtools (gitea), identity (kratos), web_search (searxng), research (parallel micro-agents), bridge
+- `src/code_index/` — schema, gitea repo walker, tree-sitter symbol extraction, OpenSearch indexer
+- `src/breadcrumbs/` — adaptive code context injection (project outline + hybrid search)
 - `src/archive/` — schema, indexer
+- `proto/code.proto` — gRPC service definition (CodeAgent: Session + ReindexCode)
--- a/docs/conversations.md
+++ b/docs/conversations.md
@@ -8,7 +8,7 @@ The Conversations API path provides persistent, server-side conversation state p
 sequenceDiagram
    participant M as Matrix Sync
    participant E as Evaluator
-    participant R as Responder
+    participant O as Orchestrator
    participant CR as ConversationRegistry
    participant API as Mistral Conversations API
    participant T as ToolRegistry
@@ -17,14 +17,14 @@ sequenceDiagram
    M->>E: message event
    E-->>M: MustRespond/MaybeRespond

-    M->>R: generate_response_conversations()
-    R->>CR: send_message(room_id, input, is_dm)
+    M->>O: GenerateRequest
+    O->>CR: send_message(conversation_key, input, is_dm)

-    alt new room (no conversation)
+    alt new conversation
        CR->>API: create_conversation(agent_id?, model, input)
        API-->>CR: ConversationResponse + conversation_id
        CR->>DB: upsert_conversation(room_id, conv_id, tokens)
-    else existing room
+    else existing conversation
        CR->>API: append_conversation(conv_id, input)
        API-->>CR: ConversationResponse
        CR->>DB: update_tokens(room_id, new_total)
@@ -32,20 +32,20 @@ sequenceDiagram

    alt response contains function_calls
        loop up to max_tool_iterations (5)
-            R->>T: execute(name, args)
-            T-->>R: result string
-            R->>CR: send_function_result(room_id, entries)
+            O->>T: execute(name, args)
+            T-->>O: result string
+            O->>CR: send_function_result(room_id, entries)
            CR->>API: append_conversation(conv_id, FunctionResult entries)
            API-->>CR: ConversationResponse
            alt more function_calls
-                Note over R: continue loop
+                Note over O: continue loop
            else text response
-                Note over R: break
+                Note over O: break
            end
        end
    end

-    R-->>M: response text (or None)
+    O-->>M: response text (or None)
    M->>M: send to Matrix room
    M->>M: fire-and-forget memory extraction
 ```
@@ -59,6 +59,8 @@ Each Matrix room maps to exactly one Mistral conversation:

 The mapping is stored in `ConversationRegistry.mapping` (HashMap in-memory, backed by SQLite `conversations` table).

+For gRPC coding sessions, the conversation key is the project path + branch, creating a dedicated conversation per coding context.
+
 ## ConversationState

 ```rust
@@ -101,7 +103,7 @@ This means conversation history is lost on compaction. The archive still has the

 ### startup recovery

-On initialization, `ConversationRegistry::new()` calls `store.load_all_conversations()` to restore all room→conversation mappings from SQLite. This means conversations survive pod restarts.
+On initialization, `ConversationRegistry::new()` calls `store.load_all_conversations()` to restore all room-to-conversation mappings from SQLite. This means conversations survive pod restarts.

 ### SQLite schema

--- a/docs/deployment.md
+++ b/docs/deployment.md
@@ -7,7 +7,7 @@ Sol runs as a single-replica Deployment in the `matrix` namespace. SQLite is the
 ```mermaid
 flowchart TD
    subgraph OpenBao
-        vault[("secret/sol<br/>matrix-access-token<br/>matrix-device-id<br/>mistral-api-key")]
+        vault[("secret/sol<br/>matrix-access-token<br/>matrix-device-id<br/>mistral-api-key<br/>gitea-admin-username<br/>gitea-admin-password")]
    end

    subgraph "matrix namespace"
@@ -18,6 +18,7 @@ flowchart TD
        deploy[Deployment<br/>sol]
        init[initContainer<br/>fix-permissions]
        pod[Container<br/>sol]
+        svc[Service<br/>sol-grpc<br/>port 50051]
    end

    vault --> |VSO sync| vss
@@ -29,6 +30,7 @@ flowchart TD
    cm --> |subPath mounts| pod
    pvc --> |/data| init
    pvc --> |/data| pod
+    svc --> |gRPC| pod
 ```

 ## manifests
@@ -49,6 +51,7 @@ replicas: 1

 - Resources: 256Mi request / 512Mi limit memory, 100m CPU request
 - `enableServiceLinks: false` — avoids injecting service env vars that could conflict
+- Ports: 50051 (gRPC)

 **Environment variables** (from Secret `sol-secrets`):

@@ -57,6 +60,8 @@ replicas: 1
 | `SOL_MATRIX_ACCESS_TOKEN` | `matrix-access-token` |
 | `SOL_MATRIX_DEVICE_ID` | `matrix-device-id` |
 | `SOL_MISTRAL_API_KEY` | `mistral-api-key` |
+| `SOL_GITEA_ADMIN_USERNAME` | `gitea-admin-username` |
+| `SOL_GITEA_ADMIN_PASSWORD` | `gitea-admin-password` |

 Fixed env vars:

@@ -115,17 +120,19 @@ spec:

 The `rolloutRestartTargets` field means VSO will automatically restart the Sol deployment when secrets change in OpenBao.

-Three keys synced from OpenBao `secret/sol`:
+Five keys synced from OpenBao `secret/sol`:

 - `matrix-access-token`
 - `matrix-device-id`
 - `mistral-api-key`
+- `gitea-admin-username`
+- `gitea-admin-password`

 ## `/data` mount layout

 ```
 /data/
-├── sol.db           SQLite database (conversations + agents tables, WAL mode)
+├── sol.db           SQLite database (conversations, agents, service_users — WAL mode)
 └── matrix-state/    Matrix SDK sqlite state store (E2EE keys, sync tokens)
 ```

@@ -140,7 +147,9 @@ Store secrets at `secret/sol` in OpenBao KV v2:
 openbao kv put secret/sol \
  matrix-access-token="syt_..." \
  matrix-device-id="DEVICE_ID" \
-  mistral-api-key="..."
+  mistral-api-key="..." \
+  gitea-admin-username="..." \
+  gitea-admin-password="..."
 ```

 These are synced to K8s Secret `sol-secrets` by the Vault Secrets Operator.
@@ -162,23 +171,36 @@ The Docker build cross-compiles to `x86_64-unknown-linux-gnu` on macOS. The fina

 ## startup sequence

-1. Initialize `tracing_subscriber` with `RUST_LOG` env filter (default: `sol=info`)
-2. Load config from `SOL_CONFIG` path
-3. Load system prompt from `SOL_SYSTEM_PROMPT` path
-4. Read 3 secret env vars (`SOL_MATRIX_ACCESS_TOKEN`, `SOL_MATRIX_DEVICE_ID`, `SOL_MISTRAL_API_KEY`)
-5. Build Matrix client with E2EE sqlite store, restore session
-6. Connect to OpenSearch, ensure archive + memory indices exist
-7. Initialize Mistral client
-8. Build components: Personality, ConversationManager, ToolRegistry, Indexer, Evaluator, Responder
-9. Backfill conversation context from archive (if `backfill_on_join` enabled)
-10. Open SQLite database (fallback to in-memory on failure)
-11. Initialize AgentRegistry + ConversationRegistry (load persisted state from SQLite)
-12. If `use_conversations_api` enabled: ensure orchestrator agent exists on Mistral server
-13. Backfill reactions from Matrix room timelines
-14. Start background index flush task
-15. Start Matrix sync loop
-16. If SQLite failed: send `*sneezes*` to all joined rooms
-17. Log "Sol is running", wait for SIGINT
+```mermaid
+flowchart TD
+    start[Start] --> tracing[Init tracing<br/>RUST_LOG env filter]
+    tracing --> config[Load config + system prompt]
+    config --> secrets[Read env vars<br/>access token, device ID, API key]
+    secrets --> matrix[Build Matrix client<br/>E2EE sqlite store, restore session]
+    matrix --> opensearch[Connect OpenSearch<br/>ensure archive + memory + code indices]
+    opensearch --> mistral[Init Mistral client]
+    mistral --> components[Build components<br/>Personality, ConversationManager,<br/>ToolRegistry, Indexer, Evaluator]
+    components --> backfill[Backfill conversation context<br/>from archive]
+    backfill --> sqlite{Open SQLite}
+    sqlite --> |success| agents[Init AgentRegistry +<br/>ConversationRegistry]
+    sqlite --> |failure| inmemory[In-memory fallback]
+    inmemory --> agents
+    agents --> orchestrator{use_conversations_api?}
+    orchestrator --> |yes| ensure_agent[Ensure orchestrator agent<br/>exists on Mistral]
+    orchestrator --> |no| skip[Skip]
+    ensure_agent --> grpc{grpc config?}
+    skip --> grpc
+    grpc --> |yes| grpc_server[Start gRPC server<br/>on listen_addr]
+    grpc --> |no| skip_grpc[Skip]
+    grpc_server --> reactions[Backfill reactions<br/>from Matrix timelines]
+    skip_grpc --> reactions
+    reactions --> flush[Start background<br/>index flush task]
+    flush --> sync[Start Matrix sync loop]
+    sync --> sneeze{SQLite failed?}
+    sneeze --> |yes| sneeze_rooms[Send *sneezes*<br/>to all rooms]
+    sneeze --> |no| running[Sol is running]
+    sneeze_rooms --> running
+```

 ## monitoring

@@ -195,6 +217,8 @@ Key log events:
 | Conversation created | info | `room`, `conversation_id` |
 | Agent restored/created | info | `agent_id`, `name` |
 | Backfill complete | info | `rooms`, `messages` / `reactions` |
+| gRPC session started | info | `session_id`, `project` |
+| Code reindex complete | info | `repos_indexed`, `symbols_indexed` |

 Set `RUST_LOG=sol=debug` for verbose output including tool results, evaluation prompts, and memory details.

@@ -226,3 +250,12 @@ Sol auto-joins rooms on invite (3 retries with exponential backoff). If it can't
 **Agent creation failure:**

 If the orchestrator agent can't be created, Sol falls back to model-only conversations (no agent). Check Mistral API key and quota.
+
+**gRPC connection refused:**
+
+If `sunbeam code` can't connect, verify the gRPC server is configured and listening:
+
+```sh
+sunbeam k8s get svc sol-grpc -n matrix
+sunbeam logs matrix/sol | grep grpc
+```