Using .gitattributes to encode file as IBM-1047 results in some characters being converted incorrectly

[zsw007]

Description

The way I have it set up is that the file is stored as UTF-8 in git, and I'm using .gitattributes to set the working tree encoding to ibm-1047. When cloning the repo, git converts and tags the file as IBM-1047. The issue is that for some characters such as ® (the registered symbol), it appears as ▒ after the conversion.

Reproduce

0. Create a new repository.
1. Create file.txt containing the ® character.
2. Create a .gitattributes file containing either file.txt zos-working-tree-encoding=ibm-1047 git-encoding=iso8859-1 or file.txt working-tree-encoding=ibm-1047
3. Clone the repository.
4. Read file.txt using vim or cat. It will display ▒ instead of ®

Additional info

Upon examination of the file with a hex editor, it appears that it's converting the ® character from C2 AE to AF which results in it being unreadable. Whereas for it to be readable, it would have to be 62 AF.

This is consistent with the behavior when using iconv to convert from UTF-8 to IBM-1047.

Whereas converting from ISO8859-1 to IBM-1047 seems to result in the correct conversion.

[MikeFultonDev]

Thanks for opening this... So the character is a UTF-8 character and is not being properly mapped to the IBM-1047 character...
Can you try the z/OS OpenTools libiconv utility using the //TRANSLIT option and see if it maps the character properly? We may want to consider using it instead (or optionally)...

[MikeFultonDev]

So - reading up in https://en.wikipedia.org/wiki/Code_page_37#1047 it seems AF is what it should be. EBCDIC is single-byte so the ® symbol will need to get mapped to AF.
Having said that, I wasn't able to display the ® symbol properly but it is AF.

As you pointed out:

/bin/iconv -fISO8859-1 -tIBM-1047 </tmp/txt >/tmp/ebcdic

does produce a file that has 0x62AF which does show up right in vim, but I think that's actually wrong... It should just be 0xAF?

[MikeFultonDev]

here's a better site: http://www.ascii.ca/ebc1047.htm

[MikeFultonDev]

so when I switched my terminal display to 'western ascii' from 'utf-8' and I iconv'ed an ASCII ® (code point 0xAE) to EBCDIC (code point 0xAF) it displayed correctly.

I also noted that if I cut and paste ® in the terminal when in western-ASCII, it will actually paste as: (R) (and not ® )

Also, if you use the z/OS Open Tools iconv (in 'sh', not bash - there is a bug around that), then if you try to convert UTF-8 ® to IBM-1047 you get:

printf '\302\256' >utf8 # ® in UTF-8
chtag -tcUTF-8 utf8; # need to either tag it as UTF-8 or remove the tag since it is tagged ISO8859-1 which is wrong
iconv -f UTF-8 -t IBM-1047<utf8 >ebcdic

This has 0xAF in the file ebcdic.
If you set your terminal to 'Western EBCDIC Latin 1' and use vim, you can see the ® properly.

Probably worth trying this out in a 3270 session - that's where I typically see EBCDIC ® but I haven't done that yet.
Is this for COBOL or HL/ASM that you want the ® or some other language?

[MikeFultonDev]

I verified that the 0xAF looks right in 3270

and

[MikeFultonDev]

@zsw007 please clone my repo and see if you have issues...

git clone git@github.com:MikeFultonDev/tester.git

Everything seems to work for me. Note I did use a different .gitattributes setting:

file.txt zos-working-tree-encoding=ibm-1047

so that I could see it on all of my Mac, z/OS and the web... I have updated libiconv which maybe is the difference but I expect git statically binds iconv in which case it's getting the z/OS native iconv (although I might be wrong @IgorTodorovskiIBM ?)

[zsw007]

Hi @MikeFultonDev thanks for looking into this issue. I was able to use a 3270 session and verify that under OMVS, AF correctly displays the ® character.

I also tried your suggestion to switch my terminal encoding on my Windows machine to ISO-8859-1 (Western European), and it has also resulted in AF displaying the ® character.

However, by switching my terminal encoding to ISO-8859-1, UTF-8 files (tagged as 819 / ISO8859-1) that previously would correctly display the ® character now displays it as ® when viewed with cat or vim. Changing the file tag to 1208 / UTF-8 does not fix the issue in vim, and instead results in cat displaying garbled characters.

Is this a known limitation of using a terminal to SSH into a z/OS machine?

Is this for COBOL or HL/ASM that you want the ® or some other language?

At the moment this is just for plain text files. I am mainly using the ® character as an example, but this situation occurs for various symbols (e.g. § ¢ ¬) and isn't just limited to the ® character.

[MikeFultonDev]

Glad to see you poking around. I don’t actually know. I am kind of in the same boat as you

[lbdyck]

The challenge is that EBCDIC and the various flavors of ASCII don't always map to the same display character, and at times a valid EBCDIC character will cause issues during a character set conversion such as new-line, line-feed, and carriage-return. Not to mention that a null can also affect things.

[zsw007]

I saw Igor suggest setting set encoding=utf-8 in .vimrc. With this setting, vim now correctly displays both AF and 62 AF as the ® character for IBM-1047 files, without breaking the display for UTF-8 files tagged as 819 / ISO8859-1.

However, the fix is limited to vim.

I'm not sure if this implies that the issue can be resolved on the application end, rather than requiring users to switch their terminal encoding?