Skip to content

[AB#301335] Add Latin name fields for beneficiaries - #5913

Draft
mmaciekk wants to merge 88 commits into
developfrom
301335/handle-local-names
Draft

[AB#301335] Add Latin name fields for beneficiaries#5913
mmaciekk wants to merge 88 commits into
developfrom
301335/handle-local-names

Conversation

@mmaciekk

@mmaciekk mmaciekk commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

AB#301335

Beneficiary names can be written in a local script (e.g. Cyrillic). This PR adds *_latin fields that store the same names in the Latin alphabet. The existing name fields are not changed.

Changes

  • Model: new optional fields given/middle/family/full_name_latin on Individual, only Latin letters allowed
  • Import: RDI (XLSX/Kobo/People), the RDI push API and Aurora fill the Latin names automatically (transliteration) when they are not provided
  • Search: searching by name also checks the Latin fields — individual & household lists, office search, grievance list, payment collector search. ES part is behind the ES_USE_LATIN_NAMES constance flag (needs fleet reindex first), plain DB search works right away
  • Grievance: changing a name through a data-change ticket refreshes the Latin fields; adding an individual fills them too
  • API: *_latin fields returned by individual/household/grievance/payment endpoints
  • UI: Latin name shown in small font under the regular name
  • Backfill: one-time script fills Latin names for existing individuals — goes business area by business area, program by program, in small batches (safe for ~30M rows), can be stopped and re-run (never overwrites filled values), bad records are skipped and written to a JSONL report

Rollout order

  1. deploy (flag OFF) → 2. run backfill script → 3. es_reindex --all → 4. turn on ES_USE_LATIN_NAMES
image image image

@mmaciekk mmaciekk changed the title [AB#310816]-local names [AB#301335]-local names Apr 16, 2026
@pavlo-mk pavlo-mk added the work in progress Change doesn't need to be reviewed label Apr 16, 2026
@codecov

codecov Bot commented Apr 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.44961% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.09%. Comparing base (8c1b639) to head (29ce1a9).

Files with missing lines Patch % Lines
src/hope/apps/household/utils.py 85.71% 1 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (98.44%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff            @@
##           develop    #5913    +/-   ##
=========================================
  Coverage    97.08%   97.09%            
=========================================
  Files          500      501     +1     
  Lines        34566    34687   +121     
  Branches      4104     4123    +19     
=========================================
+ Hits         33559    33678   +119     
- Misses         559      560     +1     
- Partials       448      449     +1     
Flag Coverage Δ
e2e 55.14% <45.73%> (-0.07%) ⬇️
unit 96.81% <98.44%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mmaciekk mmaciekk changed the title [AB#301335]-local names [AB#301335]-local names/ latin names Apr 21, 2026
@Jeeziorny
Jeeziorny marked this pull request as draft June 22, 2026 13:29
@Jeeziorny

Copy link
Copy Markdown
Contributor

If it's blocked it's blocked (not opened)

pavlo-mk and others added 7 commits July 1, 2026 12:43
# Conflicts:
#	tests/unit/api_contract/_api_checker/test_payments/_api_rest_business-areas_business-area-0_programs_c6r4_payment-plans_911e9fb2-a41a-4d07-97c2-29722c9d749a_payments_f533f1fa-fbb4-4719-a515-6c99d493fe57_/get/dc937b59892604f5a86ac96936cd7ff09e25f18ae6b758e8014a24c7fa039e91.response.json
#	tests/unit/api_contract/_api_checker/test_verification_records/_api_rest_business-areas_business-area-0_programs_43mc_payment-verifications_b546a8d8-a5c1-4161-b3fb-6186f0ca192f_verifications_480e9f86-3a14-4620-b6eb-0cb5eea687fa_/get/dc937b59892604f5a86ac96936cd7ff09e25f18ae6b758e8014a24c7fa039e91.response.json
johniak and others added 5 commits August 19, 2026 10:16
full_name_latin only exists in indexes rebuilt after the mapping change,
so the clause silently matches nothing on programs that are not reindexed
yet. Keep it out of the query until the fleet reindex is done and the flag
is flipped in constance.

Serializers, the API response and the DB search fallback are unaffected -
latin names stay visible everywhere regardless of the flag.
@johniak johniak changed the title [AB#301335]-local names/ latin names // BLOCKED BY ES BLUE-GREEN REINDEX REFINEMENT ? [AB#301335] Add Latin name fields for beneficiaries Sep 1, 2026
johniak and others added 8 commits September 1, 2026 12:00
…l-names

# Conflicts:
#	tests/unit/api_contract/_api_checker/test_payments/_api_rest_business-areas_business-area-0_programs_pd2p_payment-plans_ad89a016-ec17-43c6-81ab-729de501c8d8_payments_/get/dc937b59892604f5a86ac96936cd7ff09e25f18ae6b758e8014a24c7fa039e91.response.json
#	tests/unit/api_contract/_api_checker/test_payments/_api_rest_business-areas_business-area-0_programs_pd2p_payment-plans_ad89a016-ec17-43c6-81ab-729de501c8d8_payments_8a21fe13-2603-4ac3-9878-0acbf714cf73_/get/dc937b59892604f5a86ac96936cd7ff09e25f18ae6b758e8014a24c7fa039e91.response.json
#	tests/unit/api_contract/_api_checker/test_payments_global/_api_rest_business-areas_business-area-0_payments_/get/dc937b59892604f5a86ac96936cd7ff09e25f18ae6b758e8014a24c7fa039e91.response.json
#	tests/unit/api_contract/_api_checker/test_verification_records/_api_rest_business-areas_business-area-0_programs_1cpl_payment-verifications_cb35a199-c561-4206-a989-065b41e8f319_verifications_/get/dc937b59892604f5a86ac96936cd7ff09e25f18ae6b758e8014a24c7fa039e91.response.json
#	tests/unit/api_contract/_api_checker/test_verification_records/_api_rest_business-areas_business-area-0_programs_1cpl_payment-verifications_cb35a199-c561-4206-a989-065b41e8f319_verifications_f6730d02-ba17-4532-8d32-a3d456eb2768_/get/dc937b59892604f5a86ac96936cd7ff09e25f18ae6b758e8014a24c7fa039e91.response.json
…orms

Add a "Transliterate Latin names automatically" checkbox (default on) to the
Individual Data Update, People Data Update and Add Individual forms, and send
individual_data.transliterate_latin_names on both the create and update paths.

Without it the backend rejects any name change that carries no *_latin twin,
and the frontend surfaced that as an unexplained nested DRF 400. The rule is
now mirrored client-side so the form fails fast, latin inputs are hidden while
the checkbox is on, and stale latin values are dropped from the payload since
explicit latin values win over transliteration server side.

The flag is also skipped in the approval screens - it is a UX flag, not a
field change, and it would otherwise have counted as an unapproved change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants