Sienna Meridian Satterwhite
2f7785774b
Scaleway setup, k3s, kustomize structure, deployment phases, DNS, cert-manager, backup strategy, image registry.
259 lines
7.1 KiB
Markdown
259 lines
7.1 KiB
Markdown
# Serving Looks in Production 👠
|
||
|
||
The Super Boujee Business Box ✨ runs on a single Scaleway Elastic Metal server in Paris. One box, one bill, European data sovereignty by default. No multi-cloud, no multi-region, no per-seat fees — just a server that belongs to you.
|
||
|
||
---
|
||
|
||
## The Server
|
||
|
||
| Property | Value |
|
||
|----------|-------|
|
||
| **Provider** | Scaleway Elastic Metal |
|
||
| **Region** | Paris (PAR1/PAR2) |
|
||
| **RAM** | 64GB minimum |
|
||
| **Storage** | Local NVMe |
|
||
| **Network** | Public IPv4, configurable reverse DNS |
|
||
| **Monthly cost** | ~€50–80 (server) + ~€7–16 (services) |
|
||
|
||
### External Scaleway Services
|
||
|
||
| Service | Purpose | Cost |
|
||
|---------|---------|------|
|
||
| Object Storage | PostgreSQL backups (barman), cold asset overflow | ~€5–10/mo |
|
||
| Transactional Email (TEM) | Outbound SMTP relay | ~€1/mo |
|
||
| Generative APIs | AI inference for all components | ~€1–5/mo |
|
||
|
||
Total: under €100/month for the entire collaboration suite. That's less than three Slack seats. 💅
|
||
|
||
---
|
||
|
||
## k3s Installation
|
||
|
||
```bash
|
||
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -
|
||
```
|
||
|
||
Traefik is disabled because we use Pingora. k3s gives us single-binary Kubernetes without the overhead of a full cluster setup.
|
||
|
||
---
|
||
|
||
## Deployment Flow
|
||
|
||
### First-Time Setup
|
||
|
||
```bash
|
||
# 1. Install k3s (above)
|
||
|
||
# 2. Configure sunbeam
|
||
sunbeam config set \
|
||
--domain sunbeam.pt \
|
||
--host admin@{SERVER_IP} \
|
||
--infra-dir /path/to/sbbb \
|
||
--acme-email ops@sunbeam.pt
|
||
|
||
# 3. Bring up core infrastructure
|
||
sunbeam up # cert-manager, Linkerd, TLS
|
||
|
||
# 4. Deploy everything
|
||
sunbeam apply # all namespaces
|
||
|
||
# 5. Seed credentials
|
||
sunbeam seed # all secrets into OpenBao
|
||
|
||
# 6. Bootstrap services
|
||
sunbeam bootstrap # Gitea orgs, repos, service accounts
|
||
|
||
# 7. Verify
|
||
sunbeam verify # OpenBao + VSO sync test
|
||
sunbeam check # functional health probes
|
||
```
|
||
|
||
### Ongoing Deployments
|
||
|
||
The daily driver. Most deploys are just one command.
|
||
|
||
```bash
|
||
# Deploy all namespaces
|
||
sunbeam apply
|
||
|
||
# Deploy specific namespace
|
||
sunbeam apply lasuite
|
||
sunbeam apply monitoring
|
||
sunbeam apply matrix
|
||
|
||
# Build, push, and deploy a service
|
||
sunbeam build sol --push --deploy
|
||
```
|
||
|
||
---
|
||
|
||
## Kustomize Structure
|
||
|
||
```
|
||
sbbb/
|
||
├── base/ # Canonical manifests (environment-agnostic)
|
||
│ ├── data/ # PostgreSQL, Valkey, OpenSearch, OpenBao, SearXNG
|
||
│ ├── devtools/ # Gitea
|
||
│ ├── ingress/ # Pingora proxy
|
||
│ ├── lasuite/ # All La Suite apps + Hive + Postfix
|
||
│ ├── matrix/ # Tuwunel + Sol☀️
|
||
│ ├── media/ # LiveKit
|
||
│ ├── mesh/ # Linkerd
|
||
│ ├── monitoring/ # Prometheus, Grafana, Loki, Tempo, Alloy
|
||
│ ├── ory/ # Kratos + Hydra
|
||
│ ├── storage/ # SeaweedFS
|
||
│ ├── cert-manager/ # Certificate management
|
||
│ ├── longhorn/ # Volume management
|
||
│ └── vso/ # Vault Secrets Operator
|
||
├── overlays/
|
||
│ ├── local/ # Local dev patches (resource limits, mkcert)
|
||
│ └── production/ # Production patches (SSL, full resources)
|
||
├── scripts/ # Bash automation
|
||
└── docs/ # You are here ✨
|
||
```
|
||
|
||
### The Pattern
|
||
|
||
- **Base** holds canonical config — deployments, services, configmaps, values files, vault secrets, alert rules, service monitors
|
||
- **Overlays** hold only patches — resource overrides, domain-specific values, environment differences
|
||
- **DOMAIN_SUFFIX** placeholder gets replaced by `sed` at deploy time
|
||
|
||
```bash
|
||
# What sunbeam apply does under the hood
|
||
kustomize build overlays/production \
|
||
| sed 's/DOMAIN_SUFFIX/sunbeam.pt/g' \
|
||
| kubectl apply --server-side -f -
|
||
```
|
||
|
||
---
|
||
|
||
## Deployment Phases
|
||
|
||
The stack deploys in order of dependency — foundation first, glamour later:
|
||
|
||
### Phase 1 — Core Infrastructure
|
||
1. Kubernetes & networking (k3s, Linkerd)
|
||
2. Secrets (OpenBao, Vault Secrets Operator)
|
||
3. Database (CloudNativePG)
|
||
4. Cache (Valkey)
|
||
5. Storage (SeaweedFS)
|
||
|
||
### Phase 2 — Identity & Edge
|
||
6. Ory Kratos & Hydra
|
||
7. Pingora edge proxy
|
||
8. cert-manager (Let's Encrypt)
|
||
|
||
### Phase 3 — Applications
|
||
9. Docs (Collabora)
|
||
10. Meet (LiveKit)
|
||
11. Drive
|
||
12. Messages (full email stack)
|
||
13. Calendars
|
||
14. People
|
||
15. Projects
|
||
16. Find
|
||
17. Integration navbar
|
||
|
||
### Phase 4 — Connectivity & Sync
|
||
18. Hive (Drive ↔ S3 sync)
|
||
19. Postfix (outbound SMTP)
|
||
|
||
### Phase 5 — Matrix & AI
|
||
20. Tuwunel (Matrix homeserver)
|
||
21. Sol☀️ (AI agent)
|
||
|
||
### Phase 6 — Development Tools
|
||
22. Gitea
|
||
23. SearXNG
|
||
|
||
### Phase 7 — Observability
|
||
24. Prometheus + Grafana
|
||
25. Loki + Alloy
|
||
26. Tempo
|
||
27. AlertManager
|
||
|
||
---
|
||
|
||
## DNS Setup
|
||
|
||
All subdomains point to the server's public IP via A records:
|
||
|
||
```
|
||
docs.sunbeam.pt → A → {SERVER_IP}
|
||
meet.sunbeam.pt → A → {SERVER_IP}
|
||
drive.sunbeam.pt → A → {SERVER_IP}
|
||
mail.sunbeam.pt → A → {SERVER_IP}
|
||
messages.sunbeam.pt → A → {SERVER_IP}
|
||
people.sunbeam.pt → A → {SERVER_IP}
|
||
src.sunbeam.pt → A → {SERVER_IP}
|
||
auth.sunbeam.pt → A → {SERVER_IP}
|
||
cal.sunbeam.pt → A → {SERVER_IP}
|
||
projects.sunbeam.pt → A → {SERVER_IP}
|
||
metrics.sunbeam.pt → A → {SERVER_IP}
|
||
livekit.sunbeam.pt → A → {SERVER_IP}
|
||
s3.sunbeam.pt → A → {SERVER_IP}
|
||
# ... and the rest
|
||
```
|
||
|
||
### Email DNS
|
||
|
||
Email requires additional records:
|
||
|
||
| Record | Value |
|
||
|--------|-------|
|
||
| **MX** | Points to server IP (messages-mta-in listens on port 25) |
|
||
| **SPF** | `v=spf1 ip4:{SERVER_IP} include:tem.scaleway.com ~all` |
|
||
| **DKIM** | Generated by Postfix/Rspamd, published as TXT record |
|
||
| **DMARC** | `v=DMARC1; p=quarantine; rua=mailto:dmarc@sunbeam.pt` |
|
||
| **PTR** | Reverse DNS configured in Scaleway console |
|
||
|
||
---
|
||
|
||
## TLS Certificates
|
||
|
||
cert-manager handles Let's Encrypt certificates in production:
|
||
|
||
- ACME HTTP-01 challenges routed by Pingora itself (watches Ingress objects for solver pods)
|
||
- Certificates stored in K8s Secret `pingora-tls`
|
||
- Pingora watches the Secret and hot-reloads on renewal (zero downtime)
|
||
|
||
---
|
||
|
||
## Backups
|
||
|
||
### PostgreSQL
|
||
|
||
- **Tool:** barman (continuous archiving)
|
||
- **Destination:** Scaleway Object Storage
|
||
- **Retention:** 30 days (see [ops.md](ops.md) for the 7-year cold storage gap)
|
||
- **Recovery:** CloudNativePG handles point-in-time recovery
|
||
|
||
### Secrets
|
||
|
||
- OpenBao (Vault) data backed up separately
|
||
- Root token and unseal keys stored securely offline
|
||
|
||
### Object Storage
|
||
|
||
- SeaweedFS volumes on local NVMe
|
||
- Game assets also exist in Drive (Hive keeps them synced — bless that fairy)
|
||
- Consider periodic snapshots for critical buckets
|
||
|
||
---
|
||
|
||
## Image Registry
|
||
|
||
Container images are built and stored in the Gitea registry at `src.sunbeam.pt`:
|
||
|
||
```bash
|
||
# Build and push
|
||
sunbeam build proxy --push
|
||
sunbeam build sol --push
|
||
|
||
# Images tagged as
|
||
src.sunbeam.pt/studio/proxy:latest
|
||
src.sunbeam.pt/studio/sol:latest
|
||
```
|
||
|
||
For arm64 targets, `sunbeam mirror` rebuilds amd64-only La Suite images for multi-arch support. We don't discriminate against architectures. ✨
|