Important ordering fields for TeamAccess depend on user's
identities data. User and identities has a one-to-many relationship,
which forced us to prefetch the user-related data when listing
team's accesses.
Prefetch get data from the database using two SQL queries, and
join data in Python. User's data were not available in the first
SQL query.
Without annotating the query set with user main identities data,
we could not use default OrderingFilter backend code, which order_by()
the queryset.
Enhance list capabilities, by adding the OrderingFilter as filter backend,
to the TeamAccess viewset.
API response can be ordered by TeamAccess role. More supported ordering
fields will be supported later on.
We created a custom Pagination class, were max_page_size is overriden.
I decided to add bare minimum tests to make sure we can set the maximum
page size using the 'page_size' query parameter.
Our Pagination class inherits from the PageNumberPagination Django class.
However, this base class as not ordering attribute. Thus, setting a
default value wont have any effect on the code.
Why did we end up passing a value to this non-existing attribute? Becasue
we copy/pasted some code sources from Joanie, and Joanie also has this
attribute set to a default value.
If you take a look at DRF pagination style documentation, the only three
attributes they set on the child class are 'page_size', 'max_page_size'
'page_size_query_param'. 'ordering' is not mentionned in the attributes
you may override. However, the CursorPagination class offers the latter
attribute, which may explain why we did end up setting this non-existing
attribute in Joanie.
djangorestframework released version 3.15.0, which includes modifications of
details upon returning 404 errors (see related issue
https://github.com/encode/django-rest-framework/pull/8051).
This commit changes the expected details of 404 responses in our tests,
to match DRF 3.15.0.
In development, sessions are saved in local memory. It's working well,
however it doesn't adapt to a kubernetized setup. Several pods need
to access the current sessions, which need to be stored in a single
source of truth.
With a local memory cache, pods cannot read session saved in other pods.
We end up returning 401 errors, because we cannot authenticate the user.
I preferred setting up a proper cache than storing sessions in database,
because in the long run it would be a performance bottleneck. Cache will
decrease data access latency when reading current sessions.
I added a Redis cache backend to the production settings. Sessions would
be persisted to Redis. In K8s, a Redis operator will make sure the cached
data are not lost.
Two new dependencies were added, redis and django-redis.
I followed the installation guide of django-redis dependency. These
setting were tested deploying the app to a local K8s cluster.
To facilitate deployment on Kubernetes, we've introduced a Docker image for the
frontend. The Next.js project is built, and its static output is served using an
Nginx reverse proxy.
Since DevOps lacks a certified cold storage solution (e.g., S3) for serving
static files, we've decided to containerize the frontend as a quick workaround
for deploying staging environments.
Please note this Docker Image is WIP. One of the main issue still not resolved
concerns environment variables, which are only available when building the
Docker Image. Thus, having different environment variables values between
environment (dev, pre-prod, prod) will require us to build several frontend
images, and tag them with the appropriate target environment.
The `.env.production` values are not the final ones. For now, they were set to
dev values. It allows us to test the frontend image with the development setup.
Important: The frontend image is built-on top of an unprivileged Nginx image,
which exposes by default port 8080 instead of 80 for classic Nginx image.
You can find more info https://github.com/nginxinc/docker-nginx-unprivileged.
The Docker Compose Nginx service is used to proxy OIDC requests to keycloak,
in order to share the same host when initiating an OIDC flow, from outside and
inside docker virtual network.
All Nginx configurations related to serve frontend static build were moved to a
newly created conf file under src/frontend/apps/desk. When starting the frontend
image, we desire to start the minimum Nignx config required to serve frontend
statics.
The current implementation of our product demo via the make command lacks
user identity for a significant portion of generated users, limiting the
realism of the showcased scenarios. As it stands, users created by the make
command lack complete information, such as full names and email addresses,
because they don't have any identity.
I tried to come up with the simplest solution:
We now generate a very small portion of our users with 0 identities. The
probability for users to have only 1 identity is the highest but they
can have up to 4 with decreasing probabilities. I removed the possibility
to set a maximum number of identities as it doesn't bring any value.
3% percent of the identities created will have no email and 3% no name.
Fixes https://github.com/numerique-gouv/people/issues/90
Compute Trigram similarity on user's name, and sum it up
with existing one based on user's email.
This approach is inspired by Contact search feature, which
computes a Trigram similarity score on first name and last
name, to sum up their scores.
With a similarity score influenced by both email and name,
API results would reflect both email and name user's attributes.
As we sum up similarities, I increased the similarity threshold.
Its value is empirical, and was finetuned to avoid breaking
existing tests. Please note, the updated value is closer to the
threshold used to search contacts.
Email or Name can be None. Summing two similarity scores with
one of them None, results in a None total score. To mitigate
this issue, I added a default empty string value, to replace
None values. Thus, the similarity score on this default empty
string value is equal to 0 and not to None anymore.
When testing user search, we generated few identities
with mocked emails.
Name attribute was introduced on Identity model. Currently
names are freely and randomly generated by the factory.
To make this mocked data more realist, mock also identities'
names to match their email.
It should not break existing tests, and will make them more
predictable when introducing advanced search features.
Nest invitation router below team router and add create endpoints for
authenticated administrators/owners to invite new members to their team,
list valid and expired invitations or delete invite altogether.
Update will not be handled for now. Delete and recreate if needed.
Break copy/pasted comment from Joanie in several inline
comments, that are more specific and easy to read.
Hopefully, it will help future myself understanding this
queryset and explaining it.
To compute accesses's abilities, we need to determine
which is the user's role in the team.
We opted for a subquery, which retrieves the user's role
within the team and annotate queryset's results.
The current subquery was broken, and retrieved other
users than the request's user. It led to compute accesses'
abilities based on a randomly picked user.
Abilities on team accesses are computed based on request user role.
Thus, members' roles in relation with user's role matters a lot, to
ensure the abilities were correctly computed.
Complexified the test that lists team accesses while being authenticated.
More members are added to the team with privileged roles. The user
is added last to the less with the less privileged role, "member".
Order matters, because when computing the sub query to determine
user's role within the team, code use the first result value to set the
role to compute abilities.
When running make ruff-check, a warning informs the user that
some config are deprecated, and gives her the step to migrate.
This warning appears after Ruff released its v0.2.0.
Fix it, by keeping our pyproject.toml up to date.
We recently updated Ruff from 0.2.2 to v0.3, which introduced
Ruff 2024.2 style. This new style updated Ruff formatter's behavior,
making our make lint command fails.
Ruff 2024.2 style add a blank line after the module docstring.
Please take a look at Ruff ChangeLog to get more info.
Add serializers to return basic user info when listing /team/<id>/accesses/
endpoint. This will allow front-end to retrieve members info without having
to query API for each user.id.
When we run e2e tests with the CI, we are doing lot of
calls to the backend in a short amount of time. This can
lead to a rate limit particulary on the "user/me" endpoint.
To avoid this, we will use different backend settings
for the e2e tests.
Secret settings should not contain any default value as we risk shipping
them to production. The default value can be set via an environment variable
in the `env.d/development/common` file: OIDC_RP_CLIENT_SECRET
We need a name for the user when we display the members in the
frontend. This commit adds the name column to the identity model.
We sync the Keycloak user with the identity model when the user
logs in to fill and udpate the name automatically.
Integrate 'mozilla-django-oidc' dependency, to support
Authorization Code flow, which is required by Agent Connect.
Thus, we provide a secure back channel OIDC flow, and return
to the client only a session cookie.
Done:
- Replace JWT authentication by Session based authentication in DRF
- Update Django settings to make OIDC configurations easily editable
- Add 'mozilla-django-oidc' routes to our router
- Implement a custom Django Authentication class to adapt
'mozilla-django-oidc' to our needs
'mozilla-django-oidc' routes added are:
- /authenticate
- /callback (the redirect_uri called back by the Idp)
- /logout
Enabled Dockerflow Django app by activating liveness probes. The previously
unavailable routes such as `__heartbeat__` and `__lbheartbeat__` are now
accessible. New endpoints include:
* GET /__version__
* GET /__heartbeat__
* GET /__lbheartbeat__
The local deployment of the Production image through docker-compose was
failing due to issues in the Django configurations, influenced by Joanie.
The bug stemmed from a dependency on a development-specific package
(drf-spectacular-sidecar) while attempting to run the application in
production mode.
Changes Made:
- Introduced new Django settings for local demo environments.
- Uncommented the nginx configuration to address the production image
deployment issues.
Create invitation model, factory and related tests to prepare back-end
for invitation endpoints. We chose to use a separate dedicated model
for separation of concerns, see
https://github.com/numerique-gouv/people/issues/25
*Broken Identity string representation
Resolving a format error in the Identity string representation caused by
potential None values in the email field. This issue was discovered when
attempting to access the User details page in the Django Admin
*Broken User creation form
The replacement of the User's username with an email led to errors in
the UserAdmin class. The base class used the 'username' field in the
'add_fieldsets' attribute. This problem was discovered while attempting to
create a new user in the Django Admin.
Make Team's Slug field non-editable in the Django admin. It avoid
UX issues by preventing accidental slug overwrites during updates.
The Slug is now displayed in the teams list view.
