The Prometheus operator uses snake_case (inhibit_rules) not camelCase
(inhibitRules), causing alertmanager reconciliation to fail. Also route
InfoInhibitor alerts to null to stop flooding the Matrix alerts room.
v1.11.0 had a critical proxy connection leak in the instance-manager
(longhorn/longhorn#12575) that consumed 38.8GB on apollo, pushing the
server to 92% memory with swap exhausted.
v1.11.1 fixes the leak. Also adds a 2Gi per-container LimitRange in
longhorn-system as a safety net against future regressions.
Add sunbeam-sccache bucket and a dedicated sccache S3 identity scoped
to Read/Write/List/Tagging on that bucket only. Bump volume server
max from 50 to 100 (was full, blocking all new writes).
MCP server:
- Replace vite build --watch + livePreview with static vite preview
(watch mode was reloading the plugin iframe, killing WebSocket)
- Bake WS_URI at Docker build time for production WebSocket URL
- Add server-side application-level keepalive messages every 25s
- Add client-side auto-reconnect with exponential backoff
- Set Pingora route timeout to 86400s for WebSocket idle tolerance
Penpot:
- Add AWS_ACCESS_KEY_ID/SECRET env vars for S3 SDK compatibility
- Set S3 region to satisfy AWS SDK credential chain
- Enable OIDC registration (disable-registration blocks OIDC signup)
- Fix frontend port (8080 not 80)
- Add penpot bucket to seaweedfs-buckets init job
Penpot (designer.sunbeam.pt):
- Frontend/backend/exporter deployments with OIDC-only auth via Hydra
- VSO-managed DB, S3, and app secrets from OpenBao
- PostgreSQL user/db in CNPG postInitSQL
- Hydra Maester enabledNamespaces extended to devtools
Penpot MCP server (mcp-designer.sunbeam.pt):
- Pre-built Node.js image pushed to Gitea registry
- Auth-gated via Pingora auth_request → Hydra /userinfo
- WebSocket path for browser plugin connection
Wildcard TLS:
- Switched cert-manager from HTTP-01 (per-SAN) to DNS-01 via Scaleway webhook
- Certificate collapsed to *.sunbeam.pt + sunbeam.pt
- Added scaleway-certmanager-webhook Helm chart
- VSO secret for Scaleway DNS API credentials in cert-manager namespace
- Added cert-manager to OpenBao VSO auth role
Identity permissions flow from Kratos metadata_admin.groups through
Hydra ID token claims to Gitea's OIDC group-to-team mapping:
- super-admin → site admin + Owners + Employees teams
- employee → Owners + Employees teams
- community → Contributors team (social sign-up users)
Kratos: Discord + GitHub social login providers, community identity
schema, OIDC method enabled with env-var credential injection via VSO.
Gitea: OIDC-only login (no local registration, no password form),
APP_NAME, favicon, auto-registration with account linking.
Also: messages-mta-in recreate strategy + liveness probe for milter.
cert-manager self-signed CA issues server and client certs for BuildKit
mTLS. Buildkitd serves TLS on its ClusterIP (hostNetwork removed) and
is publicly reachable at build.DOMAIN_SUFFIX:443 through Pingora's new
SNI-based TLS passthrough router. Clients authenticate with the client
certificate from the buildkitd-client-tls secret.
Documents the missing lk-jwt-service, well-known URL fix, bare domain
routing, DNS apex records, and IPv6 cert-manager self-check failure.
Includes dual-stack K3s migration plan.
Element Call expects livekit_service_url to be an HTTPS endpoint
(lk-jwt-service), not a WebSocket URL. The client connects to LiveKit
via WSS separately after getting a JWT.
Split livekit.* requests: /sfu/get, /healthz, /get_token → lk-jwt-service,
everything else → livekit-server (WebSocket). Add sunbeam.pt bare domain
route so Element X can discover RTC foci from the server_name.
Bridges Element Call to LiveKit by exchanging Matrix OpenID tokens for
LiveKit JWTs. Shares API credentials with livekit-server via the
existing VSO secret (removed excludeRaw so raw fields are available).
Prometheus, Loki, and Tempo external endpoints were publicly accessible
with no authentication. Add auth_request to all three routes using
Hydra's userinfo endpoint (same pattern as admin APIs).
- Add readiness/liveness probes to Collabora (GET /hosting/discovery)
- Add init container to Drive backend that waits for Collabora and runs
trigger_wopi_configuration on every pod start — fixes WOPI silently
breaking after server restarts (chart Job only ran on sunbeam apply)
- Add OIDC_RESPONSE_MODE=query to Projects config
28 alert rules across 9 PrometheusRule files covering infrastructure
(Longhorn, cert-manager), data (PostgreSQL, OpenBao, OpenSearch),
storage (SeaweedFS), devtools (Gitea), identity (Hydra, Kratos),
media (LiveKit), and mesh (Linkerd golden signals for all services).
Severity routing: critical alerts fire to Matrix + email, warnings
to Matrix only (AlertManager config updated in separate commit).
Replace monolithic dashboards-configmap.yaml with 10 dedicated files,
one per Grafana folder: Ingress, Observability, Infrastructure, Storage,
Identity, DevTools, Search, Media, La Suite, Communications.
New dashboards for Longhorn, PostgreSQL/CNPG, Cert-Manager, SeaweedFS,
Hydra, Kratos, Gitea, OpenSearch, LiveKit, La Suite golden signals
(Linkerd metrics), Matrix, and Email Pipeline.
Full tour of the SBBB✨ stack: Pingora proxy, Ory identity, La Suite
apps, Linkerd mesh, OpenBao secrets, data layer, monitoring, Matrix,
Sol☀️, and the platform itself.
Full rewrite with boujee tone — app inventory, architecture diagram,
custom components (Sol☀️, Pingora, Sunbeam CLI), team bios, and links
to the new documentation suite.
Root token and unseal key were lost when a placeholder manifest
overwrote the openbao-keys Secret. Documents root cause, timeline,
5 whys, remediation actions, and monitoring requirements.
- SearXNG deployment in data namespace (free, no-tracking web search)
- sol-config: SearXNG URL, research config, identity agent, updated
system prompt (DM search rules, research mode, silence, hard rules)
- sol-deployment: debug logging (RUST_LOG=sol=debug), full image path
- opensearch: tolerate missing prometheus-exporter plugin on OS 3
Adds pingora routes for id, hydra, search, vault subdomains.
Each gated by auth_request to Hydra userinfo — only valid SSO
bearer tokens pass through. Adds new SANs to the TLS certificate.
