Skip to content

[i18n] Add support for 日本語 #9747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Conversation

driusan
Copy link
Collaborator

@driusan driusan commented Apr 11, 2025

This puts the infrastructure in place to translate strings in loris on the PHP side using the gettext library. Japanese is added as an example (mostly using machine translation which could probably use proof reading by a native speaker.)

Each module has its own textdomain for strings that come from that module, as well as a "loris" textdomain for general LORIS terms (menu categories, common terms, etc). The translations for a module are in $MODULEDIR/locale and the loris textdomain is in $LORISROOT/locale.

The module names, menu categories, and breadcrumb text are currently marked up as translateable strings. This is enough to ensure that the menus and breadcrumbs in LORIS get translated.
Screenshot of LORIS with menus translated

A new LORIS middleware detects the preferred language by looking for:

  1. A lang=? passed in the query parameter (this can be used for testing, but does not currently have any way in the GUI to select)
  2. The User's preference under "my_preferences"
  3. The HTTP request's Accept-Language header

in that order and ensuring that it's a language supported by the project by looking for the language_code in the LORIS "language" table.

The pot files at the root of the locale directories are translation templates to give to translators. The locale/$lang/LC_MESSAGES/$textdomain.po contain the translation mapping. The .mo files are the compiled version used by the software and can be generated by "make locales". (We may want to rethink the structure of the Makefile in the future to have make $modulename do both the locale and javascript compilation.)

Projects may override strings by putting the (compiled) gettext .mo file in the $project/locale/ directory (which would also be a good place to store the .po files). These are generated with msgfmt -o filename.mo inputfile.po (msgfmt is included with gettext.)

Note that the format of the locale directories must be: locale/$lang/LC_MESSAGES/$textdomain.mo. We do not have any control over that. (Other then the name of the first "locale" directory, but that seems pretty standard.)

It's also worth noting that, from what I've read, the server must have the locale installed for gettext to support the translation, not just LORIS. I don't know why. It seems silly.

Testing instructions

  1. Ensure the languages table in loris includes the language that you want to test (ie. ja_JP, unless you want to try making the .po file for another language for testing.)
  2. You may also need to install the language pack on the server you're testing on. You can run locale -a to see which locales are available on your system.
  3. Change your language preference in the my_preference table (note that it won't take effect until you go to another page, since the page is rendered before the preference is updated.)

@driusan driusan added Release: Add to release notes PR whose changes should be highlighted in the release notes Category: Feature PR or issue that aims to introduce a new feature Language: PHP PR or issue that update PHP code Module: statistics PR or issue related to statistics module Module: issue_tracker PR or issue related to issue tracker module Module: dqt PR or issue related to (old) dqt module Module: acknowledgements PR or issue related to acknowledgements module Module: api_docs PR or issue related to api_docs module Module: battery_manager PR or issue related to battery_manager module Module: behavioural_qc PR or issue related behavioural_qc module Module: brainbrowser PR or issue related to brainbrowser module Module: bvl_feedback PR or issue related to bvl_feedback module Module: candidate_list PR or issue related to candidate_list module Module: candidate_parameters PR or issue related to candidate_parameters module Module: candidate_profile PR or issue related to candidate_profile module Module: configuration PR or issue related to configuration module Module: conflict_resolver PR or issue related to conflict_resolver module Module: create_timepoint PR or issue related to create_timepoint module Module: dashboard PR or issue related to dashboard module Module: datadict PR or issue related to (old) datadict module Module: dictionary PR or issue related to (new) dictionary module Module: document_repository PR or issue related to documen_repository module Module: electrophysiology_browser PR or issue related to electrophysiology_browser module Module: electrophysiology_uploader PR or issue related to electrophysiology_uploader Module: examiner PR or issue related to examiner module Module: genomic_browser PR or issue related to genomic_browser module Module: help_editor PR or issue related to help_editor module Module: imaging_browser PR or issue related to imaging_browser module Module: imaging_qc PR or issue related to imaging_qc module Module: imaging_uploader PR or issue related to imaging_uploader module labels Apr 11, 2025
@driusan driusan added Module: instrument_builder PR or issue related to instrument_builder module Module: instrument_list PR or issue related to instrument_list module Module: instrument_manager PR or issue related to instrument_manager module Module: instruments PR or issue related to instruments module Module: login PR or issue related to login module Module: media PR or issue related to media module Module: module_manager PR or issue related to module_manager module Module: mri_violations PR or issue related to mri_violations module Module: my_preferences PR or issue related to my_preferences module Module: new_profile PR or issue related to new_profile module Module: next_stage PR or issue related to next_stage module Module: oidc PR or issue related to the oidc (OpenID Connect) module Module: publication PR or issue related to the publication module Module: schedule_module PR or issue related to the schedule_manager module Module: server_processes_manager PR or issue related to the server_processes_manger Module: survey_accounts PR or issue related to the survey_accounts module Module: timepoint_list PR or issue related to the timepoint_list module Module: user_accounts PR or issue related to the user_accounts module Module: dicom_archive PR or issue related to dicom_archive module Module: data_release PR or issue related to data_release module Module: dataquery PR or issue related to (new) dataquery module Multilingual Any tasks related to making LORIS multilingual labels Apr 11, 2025
@driusan driusan mentioned this pull request Apr 11, 2025
This puts the infrastructure in place to translate strings in loris
on the PHP side using the gettext library. Japanese is added as an
example (mostly using machine translation which could probably use
proof reading by a native speaker.)

Each module has its own textdomain for strings that come from that
module, as well as a "loris" textdomain for general LORIS terms
(menu categories, common terms, etc). The translations for
a module are in $MODULEDIR/locale and the loris textdomain is
in $LORISROOT/locale.

The module names, menu categories, and breadcrumb text are currently
marked up as translateable strings. This is enough to ensure that
the menus and breadcrumbs in LORIS get translated.

A new LORIS middleware detects the preferred language by looking for:
1. A lang=? passed in the query parameter (this can be used for testing,
   but does not currently have any way in the GUI to select)
2. The User's preference under "my_preferences"
3. The HTTP request's Accept-Language header

in that order and ensuring that it's a language supported by
the project by looking for the language_code in the LORIS
"language" table.

The pot files at the root of the locale directories are translation
templates to give to translators. The locale/$lang/LC_MESSAGES/$textdomain.po
contain the translation mapping. The .mo files are the compiled version
used by the software and can be generated by "make locales". (We may
want to rethink the structure of the Makefile in the future to have
`make $modulename` do both the locale and javascript compilation.)

Projects may override strings by putting the (compiled) gettext
.mo file in the $project/locale/ directory (which would also be
a good place to store the .po files). These are generated with
`msgfmt -o filename.mo inputfile.po` (msgfmt is included with
gettext.)

Note that the format of the locale directories must be:
locale/$lang/LC_MESSAGES/$textdomain.mo. We do not have
any control over that. (Other then the name of the first
"locale" directory, but that seems pretty standard.)

It's also worth noting that, from what I've read, the *server* must
have the locale installed for gettext to support the translation, not
just LORIS. I don't know why. It seems silly.
@ridz1208
Copy link
Collaborator

ridz1208 commented Apr 13, 2025

Couple questions to make sure I understand.

  1. you seem to be using the full function name and not _(''), is there a reason?
  2. .mo files are only compile never committed? does that also apply to projects? What workflow are you suggesting for project overrides? Projects create a .po in their project/locale/ directory and commit it, then they compile on production to generate .mo files?
  3. can we have the make command find and compile the .po files without listing them explicitely? it could catch project ones as well (depending on the answer in 2 above)
  4. When upgrading projects would be responsible to update their translations if they made overrides I presume
  5. since module .po files are in the module, what happens to module overrides? does the .mo file in the module overrride take precedence orthe one in /project/locale/
  6. Does this leave us the option to translate english to english? I presume you can by populating the english .po files with values? or would that be the .pot files?

@driusan
Copy link
Collaborator Author

driusan commented Apr 14, 2025

  1. I'm not sure what you mean. I'm using dgettext so that I can specify the text domain (ie. if the string comes from the core LORIS terminology or the module's.) I think it's probably a good practice to be explicit. If there's somewhere that I missed that I'm using the _ alias comment on the PR and I'll fix it.
  2. Yes. .mo are compiled by msgfmt and there's no reason to commit them. They're the compiled version of the .po files that PHP uses. They're output to the -o argument of msgfmt which needs to be in the right location. Technically, the location of the .po src doesn't matter as long as the .mo is in the right place, but I would recommend structuring it the same way and committing them to make life easier.
  3. I don't know. If you can figure out how I'm not opposed to changing the build process. This was just the path of least resistance for getting started. You might be able to script something with xargs.. but I don't think it makes sense for LORIS's Makefile to pick up project locales since the workflow is different and they're output to a different location. Updating any project translated strings is part of the project deployment (sort of like instruments), not the LORIS deployment.
    I was thinking it would make sense to have make $modulename make a given module (both js and locales) and make all have every $modulename as a build dependency, but we are currently inconsistent in whether make $modulename even exists for any given module, so I think that would be a different PR. I am not planning on doing it right now.
  4. I'm not doing it for them.
  5. I think the way I've set it up in this PR is project/locale has precedence. LORIS doesn't know whether a module is an override or not.
  6. There should have the option of doing English->English but I haven't tested it. The .pot are po template files, they're the "empty" untranslated strings to base a new language on. You would copy them to a .po for the locale and put the "translations" in there, then compile that with msgfmt

@driusan
Copy link
Collaborator Author

driusan commented Apr 14, 2025

Just reread and realized I phrased 2. poorly. The .mo files are the compiled version of the .po files. PHP uses the .mo files.

@regisoc regisoc added the State: Discussion required PR or issue that requires the resolution of a discussion with the relevant parties to proceed label Apr 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Feature PR or issue that aims to introduce a new feature Language: PHP PR or issue that update PHP code Module: acknowledgements PR or issue related to acknowledgements module Module: api_docs PR or issue related to api_docs module Module: battery_manager PR or issue related to battery_manager module Module: behavioural_qc PR or issue related behavioural_qc module Module: brainbrowser PR or issue related to brainbrowser module Module: bvl_feedback PR or issue related to bvl_feedback module Module: candidate_list PR or issue related to candidate_list module Module: candidate_parameters PR or issue related to candidate_parameters module Module: candidate_profile PR or issue related to candidate_profile module Module: configuration PR or issue related to configuration module Module: conflict_resolver PR or issue related to conflict_resolver module Module: create_timepoint PR or issue related to create_timepoint module Module: dashboard PR or issue related to dashboard module Module: data_release PR or issue related to data_release module Module: datadict PR or issue related to (old) datadict module Module: dataquery PR or issue related to (new) dataquery module Module: dicom_archive PR or issue related to dicom_archive module Module: dictionary PR or issue related to (new) dictionary module Module: document_repository PR or issue related to documen_repository module Module: dqt PR or issue related to (old) dqt module Module: electrophysiology_browser PR or issue related to electrophysiology_browser module Module: electrophysiology_uploader PR or issue related to electrophysiology_uploader Module: examiner PR or issue related to examiner module Module: genomic_browser PR or issue related to genomic_browser module Module: help_editor PR or issue related to help_editor module Module: imaging_browser PR or issue related to imaging_browser module Module: imaging_qc PR or issue related to imaging_qc module Module: imaging_uploader PR or issue related to imaging_uploader module Module: instrument_builder PR or issue related to instrument_builder module Module: instrument_list PR or issue related to instrument_list module Module: instrument_manager PR or issue related to instrument_manager module Module: instruments PR or issue related to instruments module Module: issue_tracker PR or issue related to issue tracker module Module: login PR or issue related to login module Module: media PR or issue related to media module Module: module_manager PR or issue related to module_manager module Module: mri_violations PR or issue related to mri_violations module Module: my_preferences PR or issue related to my_preferences module Module: new_profile PR or issue related to new_profile module Module: next_stage PR or issue related to next_stage module Module: oidc PR or issue related to the oidc (OpenID Connect) module Module: publication PR or issue related to the publication module Module: schedule_module PR or issue related to the schedule_manager module Module: server_processes_manager PR or issue related to the server_processes_manger Module: statistics PR or issue related to statistics module Module: survey_accounts PR or issue related to the survey_accounts module Module: timepoint_list PR or issue related to the timepoint_list module Module: user_accounts PR or issue related to the user_accounts module Multilingual Any tasks related to making LORIS multilingual Release: Add to release notes PR whose changes should be highlighted in the release notes State: Discussion required PR or issue that requires the resolution of a discussion with the relevant parties to proceed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants