feat: replace nginx placeholder with custom Pingora proxy; add Postfix MTA

Ingress:
- Deploy custom sunbeam-proxy (Pingora/Rust) replacing nginx placeholder
- HTTPS termination with mkcert (local) / rustls-acme (production)
- Host-prefix routing with path-based sub-routing for auth virtual host:
  /oauth2 + /.well-known + /userinfo → Hydra, /kratos → Kratos (prefix stripped), default → login-ui
- HTTP→HTTPS redirect, WebSocket passthrough, JSON audit logging, OTEL stub
- cert-manager HTTP-01 ACME challenge routing via Ingress watcher
- RBAC for Ingress watcher (pingora-watcher ClusterRole)
- local overlay: hostPorts 80/443, LiveKit TURN demoted to ClusterIP to avoid klipper conflict

Infrastructure:
- socket_vmnet shared network for host↔VM reachability (192.168.105.2)
- local-up.sh: cert-manager installation, eth1-based LIMA_IP detection, correct DOMAIN_SUFFIX sed substitution
- Postfix MTA in lasuite namespace: outbound relay via Scaleway TEM, accepts SMTP from cluster pods
- Kratos SMTP courier pointed at postfix.lasuite.svc.cluster.local:25
- Production overlay: cert-manager ClusterIssuer, ACME-enabled Pingora values
This commit is contained in:
2026-03-01 16:25:11 +00:00
parent a589e6280d
commit cdddc334ff
15 changed files with 391 additions and 64 deletions

View File

@@ -20,12 +20,26 @@ resources:
- ../../base/media
- ../../base/devtools
images:
# Local dev: image is built and imported directly into k3s containerd.
# imagePullPolicy: Never is set in values-pingora.yaml so k3s never tries to pull.
# Production overlay points this at src.DOMAIN_SUFFIX/sunbeam/sunbeam-proxy:latest.
- name: sunbeam-proxy
newName: sunbeam-proxy
newTag: dev
patches:
# Disable rustls-acme; add hostPort for TURN relay range on Lima VM
# Add hostPort for TURN relay range on Lima VM
- path: values-pingora.yaml
target:
kind: Deployment
name: pingora
# Downgrade LiveKit TURN service from LoadBalancer → ClusterIP (klipper would take hostPort 443)
- path: values-livekit.yaml
target:
kind: Service
name: livekit-server-turn
# Apply §10.7 memory limits to all Deployments
- path: values-resources.yaml

View File

@@ -0,0 +1,10 @@
# Local override: change LiveKit TURN service type from LoadBalancer to ClusterIP.
# k3s klipper-lb would otherwise bind hostPort 443, conflicting with Pingora.
# External TURN on port 443 is not needed in local dev (no NAT traversal required).
apiVersion: v1
kind: Service
metadata:
name: livekit-server-turn
namespace: media
spec:
type: ClusterIP

View File

@@ -1,7 +1,6 @@
# Patch: local Pingora overrides
# - Disables rustls-acme (ACME negotiation not needed locally)
# - Mounts mkcert wildcard cert from the pingora-tls Secret
# - Exposes TURN relay range as hostPort on the Lima VM
# - ACME disabled (mkcert wildcard cert from pingora-tls Secret)
# - hostPort for TURN relay range on the Lima VM
apiVersion: apps/v1
kind: Deployment
@@ -13,10 +12,17 @@ spec:
spec:
containers:
- name: pingora
env:
- name: ACME_ENABLED
value: "false"
imagePullPolicy: Never
ports:
# Bind HTTP/HTTPS directly to the Lima VM's host network
- name: http
containerPort: 80
hostPort: 80
protocol: TCP
- name: https
containerPort: 443
hostPort: 443
protocol: TCP
# Expose full TURN relay range as hostPort so the Lima VM forwards UDP
- name: turn-start
containerPort: 49152
@@ -26,5 +32,6 @@ spec:
containerPort: 49252
hostPort: 49252
protocol: UDP
# TLS cert comes from mkcert Secret created by scripts/local-certs.sh
# Secret name: pingora-tls, keys: tls.crt / tls.key
# acme.enabled = false is the default in pingora-config.yaml.
# The mkcert cert Secret (pingora-tls) is created by scripts/local-certs.sh
# before kustomize runs, so it is always present on first apply.

View File

@@ -46,7 +46,7 @@ spec:
- name: pingora
resources:
limits:
memory: 64Mi
memory: 128Mi
---
apiVersion: apps/v1

View File

@@ -0,0 +1,58 @@
# cert-manager resources for production TLS.
#
# Prerequisites:
# cert-manager must be installed in the cluster before applying this overlay:
# kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
#
# DOMAIN_SUFFIX and ACME_EMAIL are substituted by sed at deploy time.
# See overlays/production/kustomization.yaml for the deploy command.
---
# ClusterIssuer: Let's Encrypt production via HTTP-01 challenge.
#
# cert-manager creates one Ingress per challenged domain. The pingora proxy
# watches these Ingresses and routes /.well-known/acme-challenge/<token>
# requests to the per-domain solver Service, so multi-SAN certificates are
# issued correctly even when all domain challenges run in parallel.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: ACME_EMAIL
privateKeySecretRef:
name: letsencrypt-production-account-key
solvers:
- http01:
ingress:
# ingressClassName is intentionally blank: cert-manager still creates
# the Ingress object (which the proxy watches), but no ingress
# controller needs to act on it — the proxy handles routing itself.
ingressClassName: ""
---
# Certificate: single multi-SAN cert covering all proxy subdomains.
# cert-manager issues it via HTTP-01, stores it in pingora-tls Secret, and
# renews it automatically ~30 days before expiry. The watcher in sunbeam-proxy
# detects the Secret update and triggers a graceful upgrade so the new cert is
# loaded without dropping any connections.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: pingora-tls
namespace: ingress
spec:
secretName: pingora-tls
issuerRef:
name: letsencrypt-production
kind: ClusterIssuer
dnsNames:
- docs.DOMAIN_SUFFIX
- meet.DOMAIN_SUFFIX
- drive.DOMAIN_SUFFIX
- mail.DOMAIN_SUFFIX
- chat.DOMAIN_SUFFIX
- people.DOMAIN_SUFFIX
- src.DOMAIN_SUFFIX
- auth.DOMAIN_SUFFIX
- s3.DOMAIN_SUFFIX

View File

@@ -2,8 +2,12 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Production overlay — targets Scaleway Elastic Metal (Paris)
# Deploy with: kubectl apply -k overlays/production/
# TODO: fill in all production values before first production deploy
#
# Deploy (DOMAIN_SUFFIX and ACME_EMAIL are substituted by sed):
# DOMAIN="yourdomain.com" EMAIL="ops@yourdomain.com"
# kustomize build overlays/production/ \
# | sed -e "s/DOMAIN_SUFFIX/${DOMAIN}/g" -e "s/ACME_EMAIL/${EMAIL}/g" \
# | kubectl apply --server-side --force-conflicts -f -
resources:
- ../../base/mesh
@@ -14,13 +18,17 @@ resources:
- ../../base/lasuite
- ../../base/media
- ../../base/devtools
# cert-manager ClusterIssuer + Certificate (requires cert-manager to be installed)
- cert-manager.yaml
images:
# Set to your container registry. DOMAIN_SUFFIX is substituted by sed.
- name: sunbeam-proxy
newName: src.DOMAIN_SUFFIX/sunbeam/sunbeam-proxy
newTag: latest
patches:
# TODO: set domain to sunbeam.pt
# - path: values-domain.yaml
# TODO: enable rustls-acme + Let's Encrypt, bind to public IP
# - path: values-pingora.yaml
- path: values-pingora.yaml
# TODO: set OIDC redirect URIs to https://*.sunbeam.pt/...
# - path: values-ory.yaml

View File

@@ -0,0 +1,66 @@
# Patch: production Pingora overrides
#
# DOMAIN_SUFFIX and ACME_EMAIL are substituted by sed at deploy time.
# See overlays/production/kustomization.yaml for the deploy command.
# Production config: routes only (TLS and telemetry are the same as base).
# The cert is issued by cert-manager via the ClusterIssuer defined in
# cert-manager.yaml and stored in the pingora-tls Secret. The proxy fetches
# it from the K8s API on startup and on renewal — no acme-cache PVC needed.
apiVersion: v1
kind: ConfigMap
metadata:
name: pingora-config
namespace: ingress
data:
config.toml: |
[listen]
http = "0.0.0.0:80"
https = "0.0.0.0:443"
[tls]
cert_path = "/etc/tls/tls.crt"
key_path = "/etc/tls/tls.key"
[telemetry]
otlp_endpoint = ""
[[routes]]
host_prefix = "docs"
backend = "http://docs.lasuite.svc.cluster.local:8000"
websocket = true
[[routes]]
host_prefix = "meet"
backend = "http://meet.lasuite.svc.cluster.local:8000"
websocket = true
[[routes]]
host_prefix = "drive"
backend = "http://drive.lasuite.svc.cluster.local:8000"
[[routes]]
host_prefix = "mail"
backend = "http://messages.lasuite.svc.cluster.local:8000"
[[routes]]
host_prefix = "chat"
backend = "http://conversations.lasuite.svc.cluster.local:8000"
websocket = true
[[routes]]
host_prefix = "people"
backend = "http://people.lasuite.svc.cluster.local:8000"
[[routes]]
host_prefix = "src"
backend = "http://gitea.devtools.svc.cluster.local:3000"
websocket = true
[[routes]]
host_prefix = "auth"
backend = "http://hydra-public.ory.svc.cluster.local:4444"
[[routes]]
host_prefix = "s3"
backend = "http://seaweedfs-filer.storage.svc.cluster.local:8333"