In development, sessions are saved in local memory. It's working well,
however it doesn't adapt to a kubernetized setup. Several pods need
to access the current sessions, which need to be stored in a single
source of truth.
With a local memory cache, pods cannot read session saved in other pods.
We end up returning 401 errors, because we cannot authenticate the user.
I preferred setting up a proper cache than storing sessions in database,
because in the long run it would be a performance bottleneck. Cache will
decrease data access latency when reading current sessions.
I added a Redis cache backend to the production settings. Sessions would
be persisted to Redis. In K8s, a Redis operator will make sure the cached
data are not lost.
Two new dependencies were added, redis and django-redis.
I followed the installation guide of django-redis dependency. These
setting were tested deploying the app to a local K8s cluster.
To facilitate deployment on Kubernetes, we've introduced a Docker image for the
frontend. The Next.js project is built, and its static output is served using an
Nginx reverse proxy.
Since DevOps lacks a certified cold storage solution (e.g., S3) for serving
static files, we've decided to containerize the frontend as a quick workaround
for deploying staging environments.
Please note this Docker Image is WIP. One of the main issue still not resolved
concerns environment variables, which are only available when building the
Docker Image. Thus, having different environment variables values between
environment (dev, pre-prod, prod) will require us to build several frontend
images, and tag them with the appropriate target environment.
The `.env.production` values are not the final ones. For now, they were set to
dev values. It allows us to test the frontend image with the development setup.
Important: The frontend image is built-on top of an unprivileged Nginx image,
which exposes by default port 8080 instead of 80 for classic Nginx image.
You can find more info https://github.com/nginxinc/docker-nginx-unprivileged.
The Docker Compose Nginx service is used to proxy OIDC requests to keycloak,
in order to share the same host when initiating an OIDC flow, from outside and
inside docker virtual network.
All Nginx configurations related to serve frontend static build were moved to a
newly created conf file under src/frontend/apps/desk. When starting the frontend
image, we desire to start the minimum Nignx config required to serve frontend
statics.
The current implementation of our product demo via the make command lacks
user identity for a significant portion of generated users, limiting the
realism of the showcased scenarios. As it stands, users created by the make
command lack complete information, such as full names and email addresses,
because they don't have any identity.
I tried to come up with the simplest solution:
We now generate a very small portion of our users with 0 identities. The
probability for users to have only 1 identity is the highest but they
can have up to 4 with decreasing probabilities. I removed the possibility
to set a maximum number of identities as it doesn't bring any value.
3% percent of the identities created will have no email and 3% no name.
Fixes https://github.com/numerique-gouv/people/issues/90
Compute Trigram similarity on user's name, and sum it up
with existing one based on user's email.
This approach is inspired by Contact search feature, which
computes a Trigram similarity score on first name and last
name, to sum up their scores.
With a similarity score influenced by both email and name,
API results would reflect both email and name user's attributes.
As we sum up similarities, I increased the similarity threshold.
Its value is empirical, and was finetuned to avoid breaking
existing tests. Please note, the updated value is closer to the
threshold used to search contacts.
Email or Name can be None. Summing two similarity scores with
one of them None, results in a None total score. To mitigate
this issue, I added a default empty string value, to replace
None values. Thus, the similarity score on this default empty
string value is equal to 0 and not to None anymore.
When testing user search, we generated few identities
with mocked emails.
Name attribute was introduced on Identity model. Currently
names are freely and randomly generated by the factory.
To make this mocked data more realist, mock also identities'
names to match their email.
It should not break existing tests, and will make them more
predictable when introducing advanced search features.
Nest invitation router below team router and add create endpoints for
authenticated administrators/owners to invite new members to their team,
list valid and expired invitations or delete invite altogether.
Update will not be handled for now. Delete and recreate if needed.
Break copy/pasted comment from Joanie in several inline
comments, that are more specific and easy to read.
Hopefully, it will help future myself understanding this
queryset and explaining it.
To compute accesses's abilities, we need to determine
which is the user's role in the team.
We opted for a subquery, which retrieves the user's role
within the team and annotate queryset's results.
The current subquery was broken, and retrieved other
users than the request's user. It led to compute accesses'
abilities based on a randomly picked user.
Abilities on team accesses are computed based on request user role.
Thus, members' roles in relation with user's role matters a lot, to
ensure the abilities were correctly computed.
Complexified the test that lists team accesses while being authenticated.
More members are added to the team with privileged roles. The user
is added last to the less with the less privileged role, "member".
Order matters, because when computing the sub query to determine
user's role within the team, code use the first result value to set the
role to compute abilities.
When running make ruff-check, a warning informs the user that
some config are deprecated, and gives her the step to migrate.
This warning appears after Ruff released its v0.2.0.
Fix it, by keeping our pyproject.toml up to date.
We recently updated Ruff from 0.2.2 to v0.3, which introduced
Ruff 2024.2 style. This new style updated Ruff formatter's behavior,
making our make lint command fails.
Ruff 2024.2 style add a blank line after the module docstring.
Please take a look at Ruff ChangeLog to get more info.
Add serializers to return basic user info when listing /team/<id>/accesses/
endpoint. This will allow front-end to retrieve members info without having
to query API for each user.id.
When we run e2e tests with the CI, we are doing lot of
calls to the backend in a short amount of time. This can
lead to a rate limit particulary on the "user/me" endpoint.
To avoid this, we will use different backend settings
for the e2e tests.
Secret settings should not contain any default value as we risk shipping
them to production. The default value can be set via an environment variable
in the `env.d/development/common` file: OIDC_RP_CLIENT_SECRET
We need a name for the user when we display the members in the
frontend. This commit adds the name column to the identity model.
We sync the Keycloak user with the identity model when the user
logs in to fill and udpate the name automatically.
Integrate 'mozilla-django-oidc' dependency, to support
Authorization Code flow, which is required by Agent Connect.
Thus, we provide a secure back channel OIDC flow, and return
to the client only a session cookie.
Done:
- Replace JWT authentication by Session based authentication in DRF
- Update Django settings to make OIDC configurations easily editable
- Add 'mozilla-django-oidc' routes to our router
- Implement a custom Django Authentication class to adapt
'mozilla-django-oidc' to our needs
'mozilla-django-oidc' routes added are:
- /authenticate
- /callback (the redirect_uri called back by the Idp)
- /logout
Enabled Dockerflow Django app by activating liveness probes. The previously
unavailable routes such as `__heartbeat__` and `__lbheartbeat__` are now
accessible. New endpoints include:
* GET /__version__
* GET /__heartbeat__
* GET /__lbheartbeat__
The local deployment of the Production image through docker-compose was
failing due to issues in the Django configurations, influenced by Joanie.
The bug stemmed from a dependency on a development-specific package
(drf-spectacular-sidecar) while attempting to run the application in
production mode.
Changes Made:
- Introduced new Django settings for local demo environments.
- Uncommented the nginx configuration to address the production image
deployment issues.
Create invitation model, factory and related tests to prepare back-end
for invitation endpoints. We chose to use a separate dedicated model
for separation of concerns, see
https://github.com/numerique-gouv/people/issues/25
*Broken Identity string representation
Resolving a format error in the Identity string representation caused by
potential None values in the email field. This issue was discovered when
attempting to access the User details page in the Django Admin
*Broken User creation form
The replacement of the User's username with an email led to errors in
the UserAdmin class. The base class used the 'username' field in the
'add_fieldsets' attribute. This problem was discovered while attempting to
create a new user in the Django Admin.
Make Team's Slug field non-editable in the Django admin. It avoid
UX issues by preventing accidental slug overwrites during updates.
The Slug is now displayed in the teams list view.
* ✨(api) search users by email
The front end should be able to search users by email.
To that goal, we added a list method to the users viewset
thus creating the /users/ endpoint.
Results are filtered based on similarity with the query,
based on what preexisted for the /contacts/ endpoint.
* ✅(api) test list users by email
Test search when complete, partial query,
accentuated and capital.
Also, lower similarity threshold for user search by email
as it was too high for some tests to pass.
* 💡(api) improve documentation and test comments
Improve user viewset documentation
and comments describing tests sections
Co-authored-by: aleb_the_flash <45729124+lebaudantoine@users.noreply.github.com>
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
* 🛂(api) set isAuthenticated as base requirements
Instead of checking permissions or adding decorators
to every viewset, isAuthenticated is set as base requirement.
* 🛂(api) define throttle limits in settings
Use of Djando Rest Framework's throttle options, now set globally
to avoid duplicate code.
* 🩹(api) add email to user serializer
email field added to serializer. Tests modified accordingly.
I added the email field as "read only" to pass tests, but we need to discuss
that point in review.
* 🧱(api) move search logic to queryset
User viewset "list" method was overridden to allow search by email.
This removed the pagination. Instead of manually re-adding pagination at
the end of this method, I moved the search/filter logic to get_queryset,
to leave DRF handle pagination.
* ✅(api) test throttle protection
Test that throttle protection succesfully blocks too many requests.
* 📝(tests) improve tests comment
Fix typos on comments and clarify which setting are tested on test_throttle test
(setting import required disabling pylint false positive error)
Co-authored-by: aleb_the_flash <45729124+lebaudantoine@users.noreply.github.com>
---------
Co-authored-by: aleb_the_flash <45729124+lebaudantoine@users.noreply.github.com>
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
