Permissions of files migrated to Campaign Storage

[kdraeder]

This has become an issue because files are being migrated
to the RDA from Campaign Storage by people outside our group.
They need global read permission on the files and directories in
/gpfs/csfs1/cisl/dares/Reanalyses/f.e21.FHIST_BGC.f09_025.CAM6assim.011
I've been migrating files to CS using globus
because that was the only way at the beginning,
it has automatic failure recovery,
and it saves some core hours by moving the files
asynchronously after the submission job ends.
But globus generally can't control the permissions
of the files it creates. They are created according
to the umask of the destination environment.
I don't know how to access the umask used for CS.
See https://docs.globus.org/faq/transfer-sharing/
#how_do_i_control_file_permissions_during_transfers
The suggestions on that site didn't look appealing.
So our files on campaign storage end up without global read permission.

The simplest and probably least reliable solution
is to manually change the permissions after each complete year
is copied to CS. Unless we want the migration of data to RDA
to happen more frequently, which would require more frequent
manual changes.

It would be great to include a few lines in the Reanalyis scripts
to change the permissions automatically, but at least some
of the changes need to be made after all of the current scripts
have finished.

So some of the options seem to be:

1. Abandon globus and recode using rsync(?) and any needed error recovery,
   and pay the extra core-hours.
2. Use globus and manually update the permissions as needed.
3. Add an additional script to the workflow that would change the permissions,
   which would be run manually as needed.

Other ideas? Context?

[timhoar]

After reading the globus docs you mention, the documentation mentions

By setting ACLs on the destination such that they are applied to all new files in a directory tree, you can effectively set ACLs on the files created by the GridFTP server

so it sounds like we need to explore access control lists ... which (rather unbelievably) came out today in the CISL Daily Bulletin 'Tip': https://www2.cisl.ucar.edu/resources/storage-and-file-systems/glade/using-access-control-lists
I have not read this, and I don't know what what machine (data-access?) you would use to install the ACL, but it's another idea.

[kdraeder]

Thanks for the tip! I'll look through the Daily B article.
I also don't know what machine to use.
I could ask Chi-fan about that.

[kdraeder]

Following Tim's suggestion; DailyB ACL article:

We want the section dealing with default ACLs:
To enable another user to read and navigate into a directory

setfacl --modify user:bjsmith:r-x my_shared_directory

To set the directory's default ACL so that any new files
or subdirectories automatically have those same permissions,

setfacl --modify default:user:bjsmith:r-x my_shared_directory
ls -ld my_shared_directory
drwxr-x---+ 2 shiquan ncar 4096 Mar 6 10:12 my_shared_directory
The '+' says that there's an ACL for this directory.

We may also want syntax to give access to a group;
replace user:bjsmith:r-x with group:csg:r-x

getfacl shows the ACL, which shows bjsmith as another 'user',
not part of the group, despite the appearance of the permissions.

I'll try to implement this after I find out from Chi-Fan who
needs permission.