Commit Graph

67 Commits

Author SHA1 Message Date
lebaudantoine
51270571bc 🔧(helm) update transcription microservice in staging
Updated the webhook URL to the definitive version in docs.

Jacques also updated the webhook secret for authentication
against Impress API. Not tested locally.
2024-12-16 23:08:04 +01:00
lebaudantoine
e4f30f926c 🩹(summary) fix configuration in staging
@rouja forgot to add the right environment variable for the celery
workers. Fixed it.
2024-12-09 18:15:47 +01:00
lebaudantoine
fc61f58596 🔧(summary) update LLM and whisper endpoints
@rouja fixed the issue from OutScale load balancer.
He found the right parameter in OutScale documentation,
that was causing a request timeout.
2024-12-09 18:15:47 +01:00
lebaudantoine
459bbf65a8 🔧(summary) use our self-deployed models
Configure dev and staging environment to use our self-deployed
models (Whisper and LLM). Secrets need to be updated btw.

Because of outscale LB bug, which timeout after 60s, we need to
connect directly to the svc.
2024-12-04 10:49:17 +01:00
lebaudantoine
4b4f16e93f 🔖(patch) bump release to 0.1.10
I made a mistake, and forgot to set summary and celery
deployment to 0 replicas.
2024-12-03 02:31:10 +01:00
lebaudantoine
3e3f964ac5 🚑️(helm) set celery and summary replicas to 0
These two services are not yet needed in production. Disable them.
2024-12-03 02:25:52 +01:00
lebaudantoine
bcb111defb 🔖(patch) bump release to 0.1.9
Release the new ui, the logo, the footer, etc.
2024-12-03 02:12:36 +01:00
lebaudantoine
b74d6e72f9 🔧(helm) configure staging to notify summary service
Add the relevant configurations, to be tested when deployed.
2024-12-02 14:33:54 +01:00
lebaudantoine
4fe01ae2bf 💩(backend) notify the summary service when a new recording is available
Draft a piece of code to try the feature in staging. I'll consolidate this
implementation ASAP, as soon we have a first implementation functional.

What's missing?
- when owners are multiple
- retry when the backend cannot reach the summary service
- factorize the key oneliner, duplicated from the egress service
- optimize SQL query
- unit tests
2024-12-02 14:33:54 +01:00
lebaudantoine
300756b323 🔧(backend) configure sentry in staging and production
Share the same project as the microservice one. Will see in the future
if we need a separate sentry account for the backend. Good enough
at this project stage.
2024-12-02 14:33:54 +01:00
lebaudantoine
0af79912c5 📝(helm) generate documentation for summary microservices
Run the generate-readme script to add the documentation related to
the newly introduced microservice.
2024-12-02 14:33:54 +01:00
lebaudantoine
446270c153 🩹(summary) fix Helm Readme
Oopsie, I forgot to update the Helm readme.
Fixed it.
2024-12-02 14:33:54 +01:00
lebaudantoine
e6377ce182 (summary) enable sentry monitoring
Necessary to monitor how the micro service acts.
Will tweak sentry configurations later on.
2024-12-02 14:33:54 +01:00
lebaudantoine
50146e95f4 🩹(summary) use the right s3 env variable
I use 'url' instead of the 'endpoint' as desired.
Leading the Minio client to fail connecting to the bucket.
Fixed it!
2024-12-02 14:33:54 +01:00
lebaudantoine
35741a1bc1 ♻️(summary) reorganize API code
I totally reorganized the API code, to gain in clarity.
2024-12-02 14:33:54 +01:00
lebaudantoine
33c8a3a9e4 💩(summary) declare microservice in staging
Add the minimal extra templates and env values to deploy the micro
service.
2024-11-29 18:39:40 +01:00
lebaudantoine
dd0cb61ebc 💩(summary) kubernitize the micro service
Add the micro service to the helm chart.
2024-11-29 18:39:40 +01:00
lebaudantoine
6b03ebb393 🔧(backend) enable recording and storage hook
Update values for dev and staging environment to enable
recording-related endpoints. A new secret need to be created.

Production values will be added in an upcoming commit.
2024-11-29 17:05:29 +01:00
Jacques ROUSSEL
0627510f10 ♻️(tilt) simplify the local stack
This commit solves few issues:
- sharing the relevant certificates with minio so when triggering the webhook
notification, the minio pod can verify our backend domain certificates.
- making sure everything spawn in the right namespace (LiveKit and the Egress)
without relying on a dirty fix in the make start-tilt.

all these fixes were made by @rouja, I don't fully understand them yet.
He simplified the stack, removing two Kind nodes to make it lightweight.

thx @rouja.
2024-11-29 15:54:24 +01:00
lebaudantoine
c4206b4b24 🔖(patch) bump release to 0.1.8 2024-11-19 00:24:29 +01:00
lebaudantoine
8a96b6939a 🐛(helm) rename bucket
Wrong copy paste. My bad, bucket's name wasn't matching our project.
2024-11-18 10:04:21 +01:00
lebaudantoine
dcba3330f7 🛂(backend) request given and usual name scopes from ProConnect
Request the necessary scopes from ProConnect service.
Update configurations in every environments.

Note: ask given_name and usual_name scopes to get users' info.

(these scopes should be granted by default by ProConnect when
requesting a client id client secret)
2024-11-16 00:29:58 +01:00
lebaudantoine
60f5c8486b 🩹(backend) add missing Django settings for cold storage access
After configuring the cold storage infrastructure, I forgot to add Django
settings needed to access the configuration.
2024-11-08 10:45:58 +01:00
lebaudantoine
15e922f9df 🔥(backend) vendor analytics code
Analytics code is now useless, we mostly use
frontend tracking.
2024-11-04 17:49:15 +01:00
lebaudantoine
b69f777e5a 🔧(helm) expose MinIO console via ingress
Add ingress configuration for MinIO web console access. Include Nginx proxy
buffer annotations to support large file uploads through web interface.
2024-11-04 15:24:58 +01:00
lebaudantoine
f7ed70dc9c (livekit) add Livekit Egress
Egress is already deployed in staging. But, while
working locally on feature relying on Egress, it's not
suitable to test your development or iterate.

Especially I'll need to test the connection between the Egress
and the minio bucket in my next PR.

We faced quite a few issue while starting the whole stack.
Egress didn't want to start. Its connection with the livekit server
while the egress participant was joining the room was not successful.

The Turn part of the livekit server helm chart was activated. We needed
to update few values to in the helm configuration to enabled this turn.

Updated CoreDNS to expose Egress pod. Egress tries connecting to MinIO at
127.0.0.1, where no instance exists. Using minio.127.0.0.1.nip.io resolves
to 127.0.0.1, causing Egress to connect to itself for uploads. The CoreDNS
rewrite directs this to the Ingress IP, correctly routing to MinIO.
2024-10-28 10:26:51 +01:00
lebaudantoine
427b23ca80 (backend) add S3 objects
Inspired by Impress and @sampaccoud's work.

We use Indie Hoster’s Kubernetes objects in staging and production.
In the "dev" environment, we install the `bitnami/minio` chart to mimic
Indie Hoster’s MinIO setup.

To access the MinIO admin interface in dev, use port forwarding;
the interface runs on port 9001.
2024-10-28 10:26:51 +01:00
Jacques ROUSSEL
fb9bf6b08e 🔐(tilt) add Samuel's key
Add Samuel's key to handle secret.
2024-10-09 11:14:11 +02:00
lebaudantoine
0dadd472ff 🔖(patch) bump release to 0.1.7
Bump release to 0.1.7
2024-09-25 20:05:29 +02:00
lebaudantoine
d50d167d0a 🐛(helm) fix certificates configuration
Newly introduced ingress for PostHog were misconfigured, oopsie.
Here is the fix.
2024-09-25 11:58:34 +02:00
lebaudantoine
e4c7bc0826 ♻️(helm) separate PostHog ingress
Based on feedback from @rouja, I've updated the Helm configuration for PostHog
to use separate ingress resources for each service. Although the documentation
suggests sharing the same ingress, the services have different externalName
values, which conflicts with the use of a vhost in the ingress annotations.
This change ensures proper service redirection by aligning each service with
its own ingress.
2024-09-25 11:40:44 +02:00
lebaudantoine
b5244a5ec0 🔧(helm) configure support and analytics
Declare the expected support and analytics env variables
expected by the frontend for each environment. To avoid consuming
too much credits from our PostHog free tier plan.
2024-09-25 11:40:44 +02:00
Jacques ROUSSEL
eeb4dae12d 💚(ci) fix argocd job which handles sync
Fixed by @rouja. ArgoCD should be more robust than ever, while
syncing with our code.
2024-09-25 11:40:44 +02:00
lebaudantoine
271389d459 🚨(helm) fix linter errors
Initial set-up errors. Fixed them. Removed unused Preprod environment,
as secrets are not configured yet, it raises an error.
2024-09-25 11:40:44 +02:00
Jacques ROUSSEL
fe6eefa1f0 👷(ci) lint helmfile
Introduced by @rouja. Added a new linter to ensure helm and yaml
files can be properly parsed into templates.
ArgoCD can not break anymore.
2024-09-25 11:40:44 +02:00
lebaudantoine
a276517278 📈(frontend) setup a reverse proxy for analytics
Proxy analytics requests through our backend to minimize
ad-blockers impact. I configured the Helm Charts following
PostHog official documentation.
2024-09-25 11:40:44 +02:00
Jacques ROUSSEL
90c88a8bd3 🔒️(helm) change domainon production
Add ingress in order to migrate from meet.numerique.gouv.fr to
visio.numerique.gouv.fr
2024-09-23 20:20:56 +02:00
lebaudantoine
1103902c12 🔖(minor) bump release to 0.1.6
Release to 0.1.6
2024-09-23 20:10:16 +02:00
Jacques ROUSSEL
3391165e4b 🔒️(helm) change domain
Change the domain to visio-staging.beta.numerique.gouv.fr
2024-09-23 12:07:13 +02:00
Jacques ROUSSEL
0be94aa572 🔒️(helm) setup temporary redirect
Add a specific certificate to prepare redirect
2024-09-23 12:07:13 +02:00
Jacques ROUSSEL
a1a56402d1 🔒️(helm) fix secret sync precedence
When new secret is added to backend secret, it's not sync at the
beginning of argocd synchronisation and jobs are blocked. Theses new
annotations fix this issue.
2024-09-20 14:55:15 +02:00
lebaudantoine
f76e7f0e51 🔖(minor) bump release to 0.1.5
Minor release pre-hackathon.
2024-09-12 01:34:44 +02:00
Jacques ROUSSEL
f6bc57ba91 🔒️(helm) configure staging to use livekit-staging
Reconfigure staging environment to use
livekit-staging.beta.numerique.gouv.fr
2024-08-21 10:54:49 +02:00
lebaudantoine
223cee3828 🔖(minor) bump release to 0.1.4 2024-08-12 23:32:19 +02:00
lebaudantoine
d965ee7060 🔒️(helm) allow server host and whitelist pod IP for health checks
Updated Django's ALLOWED_HOSTS setting from '*' to the specific host of the
server. Setting ALLOWED_HOSTS to '*' is a security risk as it allows any host
to access the application, potentially exposing it to malicious attacks.
Restricting ALLOWED_HOSTS to the server's host ensures only legitimate
requests are processed.

In a Kubernetes environment, we also needed to whitelist the pod's IP address
to allow health checks to pass. This ensures that Kubernetes liveness and
readiness probes can access the application to verify its health.
2024-08-09 17:25:09 +02:00
lebaudantoine
8f59b08088 🩺(helm) update liveness and readiness probes from 10s to 30s
Updated the liveness and readiness probes interval from every 10 seconds to
every 30 seconds. This change reduces the load on the server by decreasing
the frequency of health checks.

Given the current stability of the application, a 30-second interval is
sufficient to ensure that the application remains responsive and healthy.
2024-08-09 17:25:09 +02:00
lebaudantoine
366998b3d6 🔇(backend) silence security warnings for reverse proxy setup
Silenced certain Django security warnings because the application is served
behind a reverse proxy. These warnings are not applicable in our deployment
context, where the reverse proxy handles these security concerns.

This change ensures relevant security measures are appropriately managed
while avoiding unnecessary warnings. Any question? asked @rouja.

/!\ actually, this commit is not working, and should be fixed.
2024-08-09 17:25:09 +02:00
lebaudantoine
e7dc54d6c5 🔖(minor) bump release to 0.1.3 2024-08-05 23:01:26 +02:00
lebaudantoine
b261f2ee5b 🛂(backend) disallow unregistered rooms
Require users to create a room in the database
before requesting a LiveKit token.

If user request an access token for a room that doesn't
exist in our db, its request would end in a 404 error.

Ensure that rooms must be registered by a user before they can be accessed.
By default, all created rooms remain public, allowing anonymous users to join
any room created by a logged-in user.

However, anonymous users cannot create rooms themselves.
2024-08-05 22:28:45 +02:00
lebaudantoine
aa54075e6b 📈(helm) add separate namespaces for each environment
I have set up distinct namespaces for each environment. You can now push
events to the development namespace without affecting production data.

Please note that these keys are not 'secret'. They will also be configured
in the browser SDK, which is inherently insecure. The documentation does not
specify a secure storage method for these keys.
2024-08-05 17:30:12 +02:00