Should we honour the Rocket environment variables in our ports?

[IgorTodorovskiIBM]

For example:

_ENCODE_FILE_NEW

and

_ENCODE_FILE_EXISTING

More details surrounding the environment variables are here:
#66 (comment)

Potential solution:
We could add logic for handling these into the zoslib library - https://github.com/ibmruntimes/zoslib, which is currently linked into several ports including vim, bash, gzip, tar, and potentially more

[IgorTodorovskiIBM]

Here are the instances where some of these envars are used in open source: https://github.com/search?q=%22_ENCODE_FILE_NEW%22&type=code. My guess is that there's more uses in closed source.

[perry-ca]

I get 404 error for the bash link above.

[IgorTodorovskiIBM]

Here's the snippet from their README:

Tool - bash-4.3

Works with the encoding ISO8859-1 and IBM-1047
Note:
  Use the z/OS UNIX environmental variables, _ENCODE_FILE_NEW and _ENCODE_FILE_EXISTING to control bash processing and tagging of new files and existing untagged files.

  _ENCODE_FILE_NEW=ISO8859-1  : New files are created with encoding ISO8859-1 and tagged ISO8859-1
  _ENCODE_FILE_NEW=IBM-1047   : New files are created with encoding IBM-1047 and tagged IBM-1047
  _ENCODE_FILE_NEW=UNTAGGED   : New files are created with encoding ISO8859-1 and untagged
  _ENCODE_FILE_NEW=BINARY     : New files are created without translation and are tagged BINARY

  _ENCODE_FILE_EXISTING=UNTAGGED  : Existing untagged files are assumed to have encoding IBM-1047 and will not be tagged by bash
  _ENCODE_FILE_EXISTING=ISO8859-1 : Existing untagged files are read/written with encoding ISO8859-1 and tagged ISO8859-1
  _ENCODE_FILE_EXISTING=IBM-1047  : Existing untagged files are read/written with encoding IBM-1047 and tagged IBM-1047
  _ENCODE_FILE_EXISTING=BINARY    : Existing untagged files are processed without translation and are tagged BINARY

  Default:
   _ENCODE_FILE_NEW=ISO8859-1
   _ENCODE_FILE_EXISTING=UNTAGGED

  Warning:
   Use of these combinations of environmental variables leads to unpredictable behavior
     _ENCODE_FILE_NEW=UNTAGGED
     _ENCODE_FILE_EXISTING=IBM-1047
   and
     _ENCODE_FILE_NEW=UNTAGGED
     _ENCODE_FILE_EXISTING=UNTAGGED

   Use these environmental variables carefully. Perhaps the behavior or the variables will be changed in the future.

In all cases, for existing tagged files, tags will be respected with respect to encoding.

[MikeFultonDev]

My thought is we wait for examples to come up where the tagging being done is not right and then determine the right approach. I would think the tool in question should be tagging files appropriately not the shell.

[IgorTodorovskiIBM]

I came across a situation with Vim recently where the decision to tag is not so clear. If for example we are editing an existing untagged file, should we save it as untagged or should we tag it as IBM-1047? What if the the contents are in binary? I think we need to let the user decide how untagged files are read and written.

I think this is where the Rocket environment variable came in handy for them. They offload all of the file tagging logic to their libtag, which according to the above environment variables, sets the appropriate tag. We could do something similar in ZOSLIB.

The alternative would be to add option support such as /bin/vi's -W option: -W filecodeset=ISO8859-1,pgmcodeset=IBM-1047. This would require adding in the option specific changes to each project individually.

[MikeFultonDev]

options i can think of:

• provide an env var that we use
• provide an option that we use
• auto-detect then save with a tag
• auto-detect then save without a tag
  I lean towards auto-detect and save without a tag. Not a big fan of env vars - they get out of control. An option would be ok but it would be good to see first if it was needed before we implemented it and tried to upstream it. I don't think I've ever used the vi -W option.

[perry-ca]

The other advantage I see with auto detect & save without a tag is we are preserving the original tagging of the file. Changing the tagging could change the behaviour of some other tool. It would be surprising to see a tool behave differently just because I made an unrelated change with vi. Ideally all editors should behave the same way.

[MikeFultonDev]

opened an issue for vim zopencommunity/vimport#13

opened an issue for bash: zopencommunity/bashport#56

[mfsysprog]

I was looking at the way vimport handles the tagging to solve zopencommunity/nanoport#5
I noticed that vim will always write out untagged or new files as iso8859-1. For existing untagged files this is not correct in my opinion, since it will totally change the contents of the file when looking at it purely binary. The filetag only gives a hint as to how to correctly display the binary information that is in the file and I think at least for existing files we should always only save the changes that were done in the editor and not impact the other bytes.

[mfsysprog]

Are the zopen tools running in ascii mode by default?

[IgorTodorovskiIBM]

Yes, with the exception of CMake, but that'll be ported to ASCII at some point.

The logic in the PR above, which I've ifdef'd out, guesses the codepage of an untagged file.

[IgorTodorovskiIBM]

The logic can also be controlled via these environment variables: https://github.com/ibmruntimes/zoslib/blob/407db3e26c12d744769893468423256c7c4b5640/man/zoslib.1#L35

[AnthonyGiorgio]

This particular problem bit me recently when I was cleaning up my ssh keys. I didn't realize that my authorized_keys files were untagged, and when I edited them to remove stale keys, they were silently converted to ASCII format. At that point, I could no longer use ssh keys for authentication, because the z/OS ssh daemon doesn't support file tags (yet?). I had to manually convert them back to EBCDIC to fix the issue.

[mfsysprog]

Do these exports work at runtime, or do they influence the way zoslib works at compilation?

[mfsysprog]

Would it be possible to override the open call in zoslib so that it will first check if a filetag is present and if not fail the open of the file?

[IgorTodorovskiIBM]

I think at some point we'll need to support files with no tags to cover all use cases (datasets, nfs mounts - although I think you can autotag mounted file systems right?, reading existing files on the OS).

[mfsysprog]

autotag will only set a tag on new or empty files if I remember correctly. I do see your point about having to support untagged files eventually.

[mfsysprog]

One thing that will remain difficult is whenever there are 'non transferable characters' in an ebcdic source. For example x'0a', x'0d', x'15' and some others that cannot losslessly be translated into ascii and back. We have this issue now already sometimes working with git. I can't remember if the enhanced ascii support will give of some warning in such a case.

[mfsysprog]

https://www.ibm.com/docs/en/zos/2.5.0?topic=pages-enhanced-ascii
Restriction: The conversion only applies to the portable subset of characters that are associated with a locale. Only the EBCDIC IBM-1047 encoding of portable characters is supported

But I'm not sure what happens if there are unsupported characters in the file

[mfsysprog]

I've been doing some testing and the ibm-1047 to iso8859-1 conversion is lossless, so you can convert between both codepages round trip without loosing information. The issue with non-transferable characters mainly exists when copying from mvs datasets to omvs files.

[mgrossmann]

I'm currently came across this problem.
In our shop we ares using NFS Shares with automatic code page conversion.
If I open an untaged file from such a NFS mount (original encoding ISO8859-1, mounted on USS as IBM-1141)
VIM can perfectly read this file. After saving the file, it will be saved with some kind of conversion. In the end
the file is non readable on the origin. (linux)

Best, Mike

[mfsysprog]

Can you share the specific mount options you use for the nfs mount?

[mgrossmann]

MOUNT FILESYSTEM(NFS05) TYPE(NFS) MOUNTPOINT('/u/NFS_HOME_1141') +
PARM('xxx.xxx.xxx.xxx:/home,xlat(y),AttrCaching(N),proto(tcp),   +
      cln_ccsid(01141),vers(3),rsize(8192),wsize(8192)')                

[mfsysprog]

Currently VIM will always write out the result for untagged files in ASCII. Thus changing the bytes. When the patch I'm working on is ready VIM (in reality the open en fopen in zoslib) should leave the contents asis

[IgorTodorovskiIBM]

Circling back to this discussion, support for _ENCODE_FILE_NEW is now provided via zoslib. All C/C++ based libraries/tools are built with zoslib as a dependency.
Type man zoslib for details: