🚸(backend) on user search match emails by Levenstein distance

When the query looks like an email (includes @) we search by
Levenstein distance because we are just trying to prevent typing
errors, not searching anymore.

It is important to still propose results with a short Levenstein
distance because it is frequent to forget a double letter in
someone's name for example "Pacoud" or even "pacou" instead of
"Paccoud" and we want to prevent duplicates or failing on
invitation.

We consider the query string to be an email as soon as it contains
a "@" character. Trying harder to identify a string that is really
an email would lead to weird behaviors like toto@example.gouv looking
like and email but if we continue typing toto@example.gouv.f not
looking like an email... before toto@example.gouv.fr finally looking
like an email. The result would be jumping from one type of search
to the other. As soon as there is a "@" in the query, we can be
sure that the user is not looking for a name anymore and we can
switch to matching by Levenstein distance.
This commit is contained in:
Samuel Paccoud - DINUM
2025-01-25 10:51:30 +01:00
committed by Samuel Paccoud
parent 265a24fe7e
commit 609ff91894
4 changed files with 78 additions and 61 deletions

View File

@@ -20,6 +20,7 @@ from django.db.models import (
Subquery,
Value,
)
from django.db.models.expressions import RawSQL
from django.http import Http404
import rest_framework as drf
@@ -150,29 +151,35 @@ class UserViewSet(
"""
queryset = self.queryset
if self.action == "list":
# Exclude all users already in the given document
if document_id := self.request.GET.get("document_id", ""):
queryset = queryset.exclude(documentaccess__document_id=document_id)
if self.action != "list":
return queryset
# Filter users by email similarity
if query := self.request.GET.get("q", ""):
# For performance reasons we filter first by similarity, which relies on an index,
# then only calculate precise similarity scores for sorting purposes
queryset = queryset.filter(email__trigram_word_similar=query)
# Exclude all users already in the given document
if document_id := self.request.GET.get("document_id", ""):
queryset = queryset.exclude(documentaccess__document_id=document_id)
queryset = queryset.annotate(
similarity=TrigramSimilarity("email", query)
if not (query := self.request.GET.get("q", "")):
return queryset
# For emails, match emails by Levenstein distance to prevent typing errors
if "@" in query:
return (
queryset.annotate(
distance=RawSQL("levenshtein(email::text, %s::text)", (query,))
)
# When the query only is on the name part, we should try to make many proposals
# But when the query looks like an email we should only propose serious matches
threshold = 0.6 if "@" in query else 0.1
.filter(distance__lte=3)
.order_by("distance", "email")
)
queryset = queryset.filter(similarity__gt=threshold).order_by(
"-similarity", "email"
)
return queryset
# Use trigram similarity for non-email-like queries
# For performance reasons we filter first by similarity, which relies on an
# index, then only calculate precise similarity scores for sorting purposes
return (
queryset.filter(email__trigram_word_similar=query)
.annotate(similarity=TrigramSimilarity("email", query))
.filter(similarity__gt=0.2)
.order_by("-similarity", "email")
)
@drf.decorators.action(
detail=False,