For now, the reconciliation requests are imported through CSV in the
Django admin, which sends confirmation email to both addresses. When
both are checked, the actual reconciliation is processed, and all
user-related content is updated.
## Purpose
Fix#1616 // Replaces #1708
For now, the reconciliation requests are imported through CSV in the
Django admin, which sends confirmation email to both addresses. When
both are checked, the actual reconciliation is processed, and all
user-related content is updated.
## Proposal
- [x] New `UserReconciliationCsvImport` model to manage the import of
reconciliation requests through a task
(`user_reconciliation_csv_import_job`)
- [x] New `UserReconciliation` model to store the user reconciliation
requests themselves (a row = a `active_user`/`inactive_user` pair)
- [x] On save, a confirmation email is sent to the users
- [x] A `process_reconciliation` admin action process the action on the
requested entries, if both emails have been checked.
- [x] Bulk update the `DocumentAccess` items, while managing the case
where both users have access to the document (keeping the higher role)
- [x] Bulk update the `LinkTrace` items, while managing the case where
both users have link traces to the document
- [x] Bulk update the `DocumentFavorite` items, while managing the case
where both users have put the document in their favorites
- [x] Bulk update the comment system items (`Thread`, `Comment` and
`Reaction` items)
- [x] Bulk update the `is_active` status on both users
- [x] New `USER_RECONCILIATION_FORM_URL` env variable for the "make a
new request" URL in an email.
- [x] Write unit tests
- [x] Remove the unused `email_user()` method on `User`, replaced with
`send_email()` similar to the one on the `Document` model
## Demo page reconciliation success
<img width="1149" height="746" alt="image"
src="https://github.com/user-attachments/assets/09ba2b38-7af3-41fa-a64f-ce3c4fd8548d"
/>
---------
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
## Purpose
Allows a user to find more easily the other users they search, with the
following order of priority:
- users they already share documents with (more recent first)
- users that share the same full email domain
- ~~users that share the same partial email domain (last two parts)~~
- ~~other users~~
Edit: We need to ilter out other users in order to not reveal email
addresses from members of other organisations. It's still possible to
invite them by email.
Solves #1521
## Proposal
- [x] Add a new function in `core/utils.py`:
`users_sharing_documents_with()`
- [x] Use it as a key to sort the results of a basic user search
- [x] Filter user results to avoid reveal of users (and email addresses)
of other orgs or that have not been interacted with.
- [x] User research through "full" email address (contains the '@') is
left unaffected.
---------
Co-authored-by: Anthony LC <anthony.le-courric@mail.numerique.gouv.fr>
Not every project requires silent login.
This commit adds a new feature flag
FRONTEND_SILENT_LOGIN_ENABLED to enable or
disable silent login functionality.
Most of Docs app is configured thanks to environment
variables, except the url in the email that
was from the django site table.
Now we can set it with DJANGO_EMAIL_URL_APP
environment variable to have a better consistency.
We keep the previous way to avoid breaking
changes.
The max size and allowed extensions for document
import are now fetched from the application
configuration.
This ensures consistency across the app and
allows for easier updates to these
settings in the future.
Added Helm templates for docspec deployment and service to enable
document specification conversion in the Kubernetes environment.
Updated Tiltfile, compose.yml, and Helm values to
configure docspec integration alongside the
backend converter service for document import functionality.
Refactored converter services based on PR #1609 review comments:
- Renamed parameter to `data` across all convert methods for consistency
- Replaced recursive call with explicit sequential calls for readability
- Hardcoded CONVERSION_API_SECURE=True in Production class for security
- Removed unused YdocConverter import from viewsets.py
- Updated tests to match new error message wording
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
Added comprehensive tests covering DocSpec converter service,
converter orchestration, and document creation with file uploads.
Tests validate DOCX and Markdown conversion workflows, error
handling, service availability, and edge cases including empty
files and Unicode filenames.
Signed-off-by: Stephan Meijer <me@stephanmeijer.com>
We can now import documents in formats .docx and .md.
To do so we added a new container "docspec", which
uses the docspec service to convert
these formats to Blocknote format.
More here: #1567#1569.
The template feature is removed.
Migration created to drop related tables.
Files modified:
- viewsets
- serializers
- models
- admin
- factories
- urls
- tests
- demo data
## Purpose
The email template is made with the idea that they link to a document.
This change allows to customize the label of the button (currently,
"Open") to allow for a different action verb. Additionally, the
'document_title' parameter is renamed to 'link_label' to reflect that it
can link to other things than documents.
## Proposal
- [x] Email template `template.mjml` updated as proposed
- [x] Method `send_email()` updated
- [x] Translations updated
We want to monitor AI actions. For this we choose to use langfuse. As
this usage is optional, we load langfuse sdk only if settings are
configured. Also, the openai client from langfuse is a dropin
replacement of openai client, so we only have to change how openai is
imported.
The cors-proxy endpoint was allowing redirect when fetching the target
url. This can be usefull if an image url has changed but also dangerous
if an attacker wants to hide a SSRF behind a redirect.
When the content-type return by the targeted url is not an image, the
endpoint was returning a 415 status code. We don't want to provide this
info anymore avoid disclosing information an attacker can use.
The cors-proxy endpoint allow to download images host externally without
being blocked by cors headers. The response is filter on the return
content-type to avoid disclosure and the usage of this endpoint as the
proxy used by attacker. We want to restrict the usage of this endpoint
by filtering on non legit ips used. This filter avoid exploitation of
Server Side Request Forgery (SSRF).
When all the backend authentication has been moved in the django-lasuite
library, we kept the tests to ensure that the mirgration was successful
and we didn't miss something during the transition. Now this tests are
managed in the django-lasuite library and should be maintained in it,
not in docs.
In one test related to the Document::restore function, one more query is
made. Probably a cache issue fixed in django-treebeard 4.8.0. When
updating the numchild parent, one more query is made to fetch in
database the parent document, this was not made before.
Make sure the full is never John for the first user in order to make
sure we always have only 2 users (as the search is performed on both the
email and the full name).
Fixes#1765
Signed-off-by: Anto59290 <antonin59290@hotmail.com>
External dashboards need to find the latest updated documents across
the entire hierarchy. Currently this requires many API calls to
/documents/ and /documents/{id}/children for each level.
This endpoint allows retrieving all accessible documents in a single
request, enabling dashboards to efficiently display recently changed
documents regardless of their position in the hierarchy.
Signed-off-by: ChristopherSpelt <christopherspelt@icloud.com>
As we filter the empty documents from the batch during indexing some batches
can be empty and cause an error. Now they are ignored.
Add --batch-size argument to the index command.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Reduce the number of Find API calls by grouping all the latest changes
for indexation : send all the documents updated or deleted since the
triggering of the task.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Keep ordering by score from Find API on search/ results and
fallback search still uses "-update_at" ordering as default
Refactor pagination to work with a list instead of a queryset
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Replace indexer_debounce_lock|release functions by indexer_throttle_acquire()
Instead of mutex-like mechanism, simply set a flag in cache for an amount of
time that prevents any other task creation.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Rename FindDocumentIndexer as SearchIndexer
Rename FindDocumentSerializer as SearchDocumentSerializer
Rename package core.tasks.find as core.task.search
Remove logs on http errors in SearchIndexer
Factorise some code in search API view.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
Filter deleted documents from visited ones.
Set default ordering to the Find API search call (-updated_at)
BaseDocumentIndexer.search now returns a list of document ids instead of models.
Do not call the indexer in signals when SEARCH_INDEXER_CLASS is not defined
or properly configured.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
New SEARCH_INDEXER_CLASS setting to define the indexer service class.
Raise ImpoperlyConfigured errors instead of RuntimeError in index service.
Signed-off-by: Fabre Florian <ffabre@hybird.org>
On document content or permission changes, start a celery job that will call the
indexation API of the app "Find".
Signed-off-by: Fabre Florian <ffabre@hybird.org>
We observe some throttling pick here and there.
We observed that when the collaboration has a
problem, it is retrying to connect, leading to more
requests to the django backend. At one point, the
throttling is reached and the user would not
be able to use the application anymore.
Now when the request comes from a collaboration
server, we do not throttle it anymore.
We have the user full name through OIDC in the database, but the search only
used the email field.
This change allows to search for a user by their first and/or
last name (fix#929).
Given that user names are more likely than emails to include diacritics, it
unaccents both the query and the database entry for search (fix#1091).
It also unaccents for email so that internationalized domain names are
managed whether or not the accent is included in the search.
An unaccented gin index is added on users full_name an email fields.
Using a manual migration because a wrapper around unaccent is necessary
to make it IMMUTABLE (cf.
https://stackoverflow.com/questions/9063402/ )
We want to create a new user in a marketing system to create a dedicated
onboarding for each of them. The marketing service is implemented in the
django-lasuite library and it is possible to pick the backend we want
or implement a new one following the documentation on this library.
In the UserLightSerializer we were fallbacking on a strategy to never
have a full_name or short_name empty. We use the part of the email
befire the @. We are doing the same thing now in the main
UserSerializer.
This commit add the CRUD part to manage comment lifeycle. Permissions
are relying on the Document and Comment abilities. Comment viewset
depends on the Document route and is added to the
document_related_router. Dedicated serializer and permission are
created.
In order to store the comments on a document, we created a new model
Comment. User is nullable because anonymous users can comment a Document
is this one is public with a link_role commentator.
A complete API was able to manage templates lifecycle, from the creation
to the deletion and managing accesses on them. This API is not used by
the frontend application, is not finished. A connected user can interact
with this API and lead to unwanted behavior in the interface. Refering
ot issue #1222 templates can maybe totaly remove in the future. While
it's here and used, we only keep list and retrive endpoints. The
template management can still be done in the admin interface.
Like in other abilities, we compute a set_role_to property on the
abilities. This set_role_to contains all the roles lower or equal than
the current user role. We rely on this propoerty to validate the accept
endpoint and it will be used by the front allpication to built the role
select list.
