UnicodeDecodeError while logging

[zainab-amir]

Running the i18n_tool validate command kept failing because of some UnicodeDecodeError that occurred when logging the problem with the .po file

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/bin/i18n_tool", line 11, in <module>
    sys.exit(main())
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/main.py", line 60, in main
    return module.main()
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/__init__.py", line 51, in __call__
    return self.run(args)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 250, in run
    if validate_po_files(self.configuration, locale_dir, report_empty=args.empty, check_all=args.check_all):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 43, in validate_po_files
    if msgfmt_check_po_file(locale_dir, filename):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/i18n/validate.py", line 80, in msgfmt_check_po_file
    log.warning(u'\n' + err.decode('utf8'))
  File "/edx/app/edxapp/venvs/edxapp/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 328: invalid continuation byte

This was the error(err) that it was trying to log in log.warning(u'\n' + err.decode('utf8'))

../../Desktop/Project/i18n-tools/i18n/django-partial.po:6509: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, there is an unterminated format directive.\n../../Desktop/Project/i18n-tools/i18n/django-partial.po:9602: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '\xd0' cannot start a field name.\n../../Desktop/Project/i18n-tools/i18n/django-partial.po:12407: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '\xd0' cannot start a field name.\nmsgfmt: found 3 fatal errors\n"
663
../../Desktop/Project/i18n-tools/i18n/django-partial.po:6509: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, there is an unterminated format directive.
../../Desktop/Project/i18n-tools/i18n/django-partial.po:9602: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '�' cannot start a field name.
../../Desktop/Project/i18n-tools/i18n/django-partial.po:12407: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '�' cannot start a field name.
msgfmt: found 3 fatal errors

Since the filename or language code was not getting logged for which the validation was failing on jenkins pull_translations job it was particularly difficult to find the file and issue.

[mrtndwrd]

I just ran into this same problem, which was super frustrating to debug, because the python error doesn't mention where in the actual .po file the problem exists.

Is there a reason at all to explicitly utf-8-decode the output of msgfmt?

For me the output was this:

INFO:i18n.execute:msgfmt -c -o /dev/null fa_IR/LC_MESSAGES/django-partial.po
INFO:i18n.validate:

Traceback (most recent call last):
  File "/openedx/venv/bin/i18n_tool", line 11, in <module>
    load_entry_point('edx-i18n-tools', 'console_scripts', 'i18n_tool')()
  File "/openedx/venv/src/edx-i18n-tools/i18n/main.py", line 60, in main
    return module.main()
  File "/openedx/venv/src/edx-i18n-tools/i18n/__init__.py", line 50, in __call__
    return self.run(args)
  File "/openedx/venv/src/edx-i18n-tools/i18n/validate.py", line 252, in run
    if validate_po_files(self.configuration, locale_dir, report_empty=args.empty, check_all=args.check_all):
  File "/openedx/venv/src/edx-i18n-tools/i18n/validate.py", line 42, in validate_po_files
    if msgfmt_check_po_file(locale_dir, filename):
  File "/openedx/venv/src/edx-i18n-tools/i18n/validate.py", line 79, in msgfmt_check_po_file
    log.warning('\n%s', err.decode('utf8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 1150: invalid continuation byte
make: *** [Makefile:49: validate] Error 1

I ran msgfmt inside the openedx-i18n container to get the failing output, it's this:

openedx@e4a301097015:~/edx-platform/conf/locale/fa_IR/LC_MESSAGES$ msgfmt -c -o /dev/null django-partial.po
django-partial.po:1642: a format specification for argument 'platform_name' doesn't exist in 'msgstr'
django-partial.po:1652: 'msgstr' is not a valid Python format string, unlike 'msgid'. Reason: The character that terminates the directive number 1 is not a valid conversion specifier.
django-partial.po:1710: a format specification for argument 'platform_name' doesn't exist in 'msgstr'
django-partial.po:1886: 'msgstr' is not a valid Python format string, unlike 'msgid'. Reason: The character that terminates the directive number 1 is not a valid conversion specifier.
django-partial.po:1908: 'msgstr' is not a valid Python format string, unlike 'msgid'. Reason: The string ends in the middle of a directive.
django-partial.po:3459: 'msgstr' is not a valid Python format string, unlike 'msgid'. Reason: The character that terminates the directive number 1 is not a valid conversion specifier.
django-partial.po:4890: 'msgstr' is not a valid Python brace format string, unlike 'msgid'. Reason: In the directive number 0, '�' cannot start a field name.
django-partial.po:8047: 'msgstr' is not a valid Python format string, unlike 'msgid'. Reason: The string ends in the middle of a directive.
django-partial.po:8130: a format specification for argument 'Register_label_strong' doesn't exist in 'msgid'
django-partial.po:8185: a format specification for argument 'allow_domain' doesn't exist in 'msgid'
django-partial.po:8516: a format specification for argument 'usage_key' doesn't exist in 'msgstr'
django-partial.po:8654: a format specification for argument 'expiration_date' doesn't exist in 'msgstr'
django-partial.po:8661: a format specification for argument 'expiration_date' doesn't exist in 'msgstr'
django-partial.po:8671: a format specification for argument 'expiration_date' doesn't exist in 'msgstr'
django-partial.po:8790: a format specification for argument 'Enterpr__Customer_name' doesn't exist in 'msgid'

This is the offending line in the django-partial.po:

msgstr "ﺲﻓﺍﺮﺷ {ﺲﻓﺍﺮﺷ} ﭗﺸﺘﯿﺑﺎﻨﯾ ﻦﻤﯾ<200c>ﺷﻭﺩ"