feat(orchestrator): Phase 2 engine + tokenizer + tool dispatch

Orchestrator engine:
- engine.rs: unified Mistral Conversations API tool loop that emits
  OrchestratorEvent instead of calling Matrix/gRPC directly
- tool_dispatch.rs: ToolSide routing (client vs server tools)
- Memory loading stubbed (migrates in Phase 4)

Server-side tokenizer:
- tokenizer.rs: HuggingFace tokenizers-rs with Mistral's BPE tokenizer
- count_tokens() for accurate usage metrics
- Loads from local tokenizer.json or falls back to bundled vocab
- Config: mistral.tokenizer_path (optional)

No behavior change — engine is wired but not yet called from
sync.rs or session.rs (Phase 2 continuation).
This commit is contained in:
2026-03-23 17:40:25 +00:00
parent ec4fde7b97
commit 9e5f7e61be
9 changed files with 1065 additions and 31 deletions

View File

@@ -43,6 +43,7 @@ tonic-prost = "0.14"
prost = "0.14"
tokio-stream = "0.1"
jsonwebtoken = "9"
tokenizers = { version = "0.22", default-features = false, features = ["onig", "http"] }
[build-dependencies]
tonic-build = "0.14"