Full tour of the SBBB✨ stack: Pingora proxy, Ory identity, La Suite
apps, Linkerd mesh, OpenBao secrets, data layer, monitoring, Matrix,
Sol☀️, and the platform itself.
Full rewrite with boujee tone — app inventory, architecture diagram,
custom components (Sol☀️, Pingora, Sunbeam CLI), team bios, and links
to the new documentation suite.
Root token and unseal key were lost when a placeholder manifest
overwrote the openbao-keys Secret. Documents root cause, timeline,
5 whys, remediation actions, and monitoring requirements.
- SearXNG deployment in data namespace (free, no-tracking web search)
- sol-config: SearXNG URL, research config, identity agent, updated
system prompt (DM search rules, research mode, silence, hard rules)
- sol-deployment: debug logging (RUST_LOG=sol=debug), full image path
- opensearch: tolerate missing prometheus-exporter plugin on OS 3
Adds pingora routes for id, hydra, search, vault subdomains.
Each gated by auth_request to Hydra userinfo — only valid SSO
bearer tokens pass through. Adds new SANs to the TLS certificate.
sol-config: added [vault] and [services.gitea] sections, fetch
allowlist (wttr.in, open-meteo, github), bumped context windows
to 200, updated system prompt with run_script docs and tool rules.
sol-deployment: added gitea admin credential env vars from
sol-secrets, automountServiceAccountToken for vault k8s auth.
vault-secrets: added gitea-admin-username and gitea-admin-password
templates to sol-secrets VSS.
- Prometheus: discover ServiceMonitors/PodMonitors in all namespaces,
enable remote write receiver for Tempo metrics generator
- Tempo: enable metrics generator (service-graphs + span-metrics)
with remote write to Prometheus
- Loki: add Grafana Alloy DaemonSet to ship container logs
- Grafana: enable dashboard sidecar, add Pingora/Loki/Tempo/OpenBao
dashboards, add stable UIDs and cross-linking between datasources
(Loki↔Tempo derived fields, traces→logs, traces→metrics, service map)
- Linkerd: enable proxy tracing to Alloy OTLP collector, point
linkerd-viz at existing Prometheus instead of deploying its own
- Pingora: add OTLP rollout plan (endpoint commented out until proxy
telemetry panic fix is deployed and Alloy is verified healthy)
Sol is a Matrix bot with E2EE that archives conversations to OpenSearch
and responds via Mistral AI function calling. Adds deployment, PVC,
ConfigMap (sol.toml + system prompt), VaultStaticSecret for credentials,
and production overlay image entry.
Deploy Planka-based project management at projects.DOMAIN_SUFFIX:
- ConfigMap with OIDC, S3, SMTP, La Gaufre widget config
- Deployment + Service (init container for DB migrations, Sails on 1337)
- OAuth2Client (client_secret_basic, redirect to /oidc-callback)
- VaultDynamicSecret for DATABASE_URL, VaultStaticSecret for SECRET_KEY
- Pingora route with websocket support (Socket.io)
- Image overrides in both local and production overlays
- TLS cert dnsNames updated for projects subdomain
- Integration service.json updated with Projects entry
- seaweedfs-s3-credentials rolloutRestartTargets includes projects
Enable TOTP, WebAuthn, and lookup secret MFA methods in Kratos config.
Fix Monaspace Neon font CDN URL in Gitea theme ConfigMap. Remove
redundant Google Fonts preconnect from people-frontend nginx config.
Delete unused login-ui-deployment.yaml (login-ui is part of the Ory
Helm chart, not a standalone deployment).
Add K8s manifests for calendars backend, frontend (Caddy), CalDAV
server, and Celery worker. Wire Pingora routing for cal.sunbeam.pt
with path-based backend/caldav/static splits. Add OAuth2Client for
OIDC, VaultDynamicSecret for DB credentials, VaultStaticSecret for
Django/CalDAV keys, and TLS cert coverage for the cal subdomain.
Register calendars in the integration service gaufre widget.
- Add matrix to hydra-maester enabledNamespaces for OAuth2Client CRD
- Update allowed_return_urls and selfservice URLs: chat→messages
- Add Kratos verification flow, employee/external identity schemas
- Extend session lifespan to 30 days with persistent cookies
- Route messages.* to tuwunel via Pingora with WebSocket support
- Replace login-ui with kratos-admin-ui as unified auth frontend
- Update TLS certificate SANs: chat→messages, add monitoring subdomains
- Add tuwunel + La Suite images to production overlay
- Switch DDoS/scanner detection to compiled-in ensemble models (observe_only)
- Upgrade from OpenSearch 2 to 3 (required for ML Commons pre-trained models)
- Rename PLUGINS_SECURITY_DISABLED → DISABLE_SECURITY_PLUGIN (OS3 change)
- Enable ML Commons plugin settings for on-data-node inference
- Increase memory limits (2Gi) and JVM heap for neural model inference
- Add fsGroup security context for volume permissions
Kubernetes manifests for tuwunel — a Rust Matrix homeserver using RocksDB
for storage. Includes deployment, service, PVC, ConfigMap (tuwunel.toml),
Hydra OAuth2Client for SSO, and Vault secrets for credentials injection.
Key design decisions:
- enableServiceLinks: false to prevent K8s TUWUNEL_* env var conflicts
- strategy: Recreate for RocksDB exclusive lock (no rolling updates)
- Identity provider configured entirely via env vars (client_id/secret
from hydra-maester Secret, not hardcoded)
- OpenSearch model_id injected via ConfigMap from CLI post-apply hook
- SSO-only auth (login_with_password=false, single_sso=true)
- OpenSearch hybrid neural+BM25 search (768-dim, all-mpnet-base-v2)
- Set otlp_endpoint to Tempo HTTP receiver (port 4318) for request tracing
- Add hostNetwork to prometheusSpec so it can reach kubelet/node-exporter on node public IP
- Add ServiceMonitor for proxy metrics scrape on port 9090
- Add CORS origin and Grafana datasource config for monitoring stack
- Add DDoS, scanner, and rate limiter configuration to pingora-config
- Add kubernetes config section with configurable namespace/resource names
- Expose metrics port 9090 on deployment and service
- meet-config: rename ALLOWED_HOSTS → DJANGO_ALLOWED_HOSTS (django-configurations
ListValue uses DJANGO_ prefix by default; without it the list was empty and
every browser request got 400 DisallowedHost)
- meet-config: set LIVEKIT_API_URL to public https://livekit.DOMAIN_SUFFIX so
the meet frontend can reach LiveKit for WebSocket signaling
- pingora-config: add livekit.DOMAIN_SUFFIX → livekit-server:80 WebSocket route
- cert-manager: add livekit.DOMAIN_SUFFIX to TLS cert dnsNames
- oidc-clients: fix meet redirect URI /oidc/callback/ → /api/v1.0/callback/
(meet embeds mozilla-django-oidc inside the api/v1.0/ prefix); add
postLogoutRedirectUri for clean logout
- livekit-values: replace hardcoded devkey:secret-placeholder with key_file
loaded from a VSO-managed K8s Secret (secret/livekit in OpenBao)
- media/vault-secrets: add VaultAuth + VaultStaticSecret for media namespace
to sync livekit API credentials from OpenBao
Meet: add backend/frontend/celery deployments and services, meet-config
ConfigMap, nginx SPA config, VSO secrets (meet-db-credentials VDS,
meet-django-secret and meet-livekit VSS). Wire oidc-meet OAuth2Client.
La Suite overlay discipline: move people/docs frontend nginx ConfigMaps
and patches from overlays/local to base so both environments share them.
Remove values-ory.yaml (folded into base). Add docs-frontend nginx config
with sub_filter theming. Add local gitea mkcert CA patch.
Pingora: add [ssh] TCP passthrough block (port 22 → Gitea SSH pod) and
split meet route into frontend default + backend paths for /api/, /admin/,
/oidc/, /static/, /__. Remove now-unused values-pingora.yaml from production
overlay (host ports moved to patch-pingora-hostport.yaml).
Update both overlay kustomizations to reference all new resources and
add meet-backend/meet-frontend image entries.
Add new bases for cert-manager (Let's Encrypt + wildcard cert), Longhorn
distributed storage, and monitoring (kube-prometheus-stack + Loki + Tempo
+ Grafana OIDC). Add cloud-init for Scaleway Elastic Metal provisioning.
Production overlay: add patches for postgres sizing, SeaweedFS volume,
OpenSearch storage, LiveKit service, Pingora host ports, resource limits,
and CNPG daily barman backups. Update cert-manager.yaml with full dnsNames
for all *.sunbeam.pt subdomains.
The impress chart renders this Job unconditionally (no if-enabled guard),
then auto-deletes it after 30s (ttlSecondsAfterFinished). Each sunbeam
apply recreated it and it failed because no superuser credentials are set
(users authenticate via OIDC). Override the command to true so the Job
exits 0 immediately and disappears cleanly.
- Fix Hydra postLogoutRedirectUris for docs and people to match the
actual URI sent by mozilla_django_oidc v5 (/api/v1.0/logout-callback/)
instead of the root URL, resolving 599 logout errors.
- Fix docs y-provider WebSocket backend port: use Service port 443
(not pod port 4444 which has no DNAT rule) in Pingora config.
- Tighten VSO VaultDynamicSecret rotation sync: add allowStaticCreds:true
and reduce refreshAfter from 1h to 5m across all static-creds paths
(kratos, hydra, gitea, hive, people, docs) so credential rotation is
reflected within 5 minutes instead of up to 1 hour.
- Set Hydra token TTLs: access_token and id_token to 5m; refresh_token
to 720h (30 days). Kratos session carries silent re-auth so the short
access token TTL does not require users to log in manually.
- Set SESSION_COOKIE_AGE=3600 (1h) in docs and people backends. After
1h, apps silently re-auth via the active Kratos session. Disabled
identities (sunbeam user disable) cannot re-auth on next expiry.
Replace the inline gaufre.js/nginx.conf ConfigMap approach with a
purpose-built custom image (sunbeam/integration-service) that builds
the lagaufre.js v2 widget from the suitenumerique/integration source
and serves it via nginx.
Changes:
- Rewrite integration-deployment.yaml: custom image, v2 services.json
format, only actually-deployed services (docs, meet, people)
- Add people-frontend nginx sub_filter overlay to rewrite the hardcoded
production integration URL baked into the Next.js bundle at build time
- Register integration image in local overlay kustomization
Django backends call the OIDC token, userinfo, and JWKS endpoints
server-side. Pointing these at the public auth.DOMAIN_SUFFIX URL caused
an SSLError in pods because mkcert CA certificates are not trusted inside
containers.
Split the configmap entries:
- OIDC_OP_AUTHORIZATION_ENDPOINT and OIDC_OP_LOGOUT_ENDPOINT remain as
public HTTPS URLs -- the browser navigates to these.
- OIDC_OP_TOKEN_ENDPOINT, OIDC_OP_USER_ENDPOINT, OIDC_OP_JWKS_ENDPOINT
now point to http://hydra-public.ory.svc.cluster.local:4444 -- Django
calls these directly, bypassing the proxy and its TLS certificate.
Affects all La Suite apps (docs, people) that use lasuite-oidc-provider.
Deploys the suitenumerique/lasuite-integration app that serves the La
Gaufre app launcher (gaufre.js) and acts as the federation hub for the
La Suite Numérique app switching menu.
The service runs at integration.DOMAIN_SUFFIX and exposes
/api/v1/gaufre.js — referenced by docs, people, and other La Suite
apps via GAUFREJS_URL to render the unified app switcher.
Local Lima VM (12 GB) doesn't need HA replicas. Each extra pod with a
Linkerd sidecar wastes ~64 MB. Scale people-backend, people-celery-worker,
and people-frontend to 1 replica each.
- Switch all user-facing app OAuth2 clients to client_secret_post
(mozilla-django-oidc sends credentials in POST body by default)
- Set LOGIN_REDIRECT_URL=/ so Django redirects to frontend after login
- Add local overlay patch to disable OIDC SSL verification
(mkcert CA not trusted inside pods; production uses real certs)
- Add oauth2_provider.url pointing to hydra-admin so login_challenge
params are accepted (fixes People OIDC login flow)
- Scope session cookie to parent DOMAIN_SUFFIX so admin.* subdomains
share the session (fixes redirect loop on kratos-admin-ui)
- Add allowed_return_urls for admin.*, enable recovery flow, add error
and recovery ui_url entries
- Fix KRATOS_PUBLIC_URL port in login-ui deployment (4433 → 80)
Deploy the custom Kratos admin UI (Deno/Hono + Cunningham React):
- K8s Deployment + Service in ory namespace
- VSO VaultStaticSecret for cookie/csrf/admin-identity-ids secrets
- Pingora route for admin.DOMAIN_SUFFIX
