Revise file name specification

[RobertPincus]

Two aspects make some file names difficult to discover -

1. The version number is encoded as a data attribute and in the file name. This makes auto-discovery harder than it need be, and means that intake catalogs have to be updated by hand every time the underlying data is revised.
2. the file name specification includes the start time. Many files (e.g. aircraft flight files) cover an entire day but have unpredictable start times.

I propose that the data version number be removed from the naming convention and the _HHMMSS component be made optional.

[d70-t]

For 1, I really think that we need some way of specifying a permanent reference to a file in a specific version. If the file is in some version control system, that system provides such a mechanism. If the file is not in a version control system, the file creators must ensure that old permanent references to a file are not destroyed by updating the file to a newer version. So I would strongly advertise for some well-known way to change the file name / path / url when the file contents change in a way which is not reflected otherwise in the file name (i.e. version update).

For a better auto-discovery, we could still add something like symbolic links which always point to the newest version of the file. People could then be informed that some links are permanent and some are for convenience, but non-permanent. Probably, the intake catalog would be a good way to define symbolic links though? ... and these would need to be updated whenever a file changes. Maybe that is another argument why these catalogues should be auto-generated.

For 2: Yes I stumbled about that issue as well. Maybe we should make a rule that the time in file names should be as unspecific as unambiguously possible?

[RobertPincus]

@d70-t I like your solution to the timestamp problem very much.

Re versions I can see your point about communicating clearly when data versions change, even if this is available in the file metadata. Two concerns: 1) not all users will go through intake, so our solutions should be more general. 2) Symbolic links etc. require action on the part of the data distributors that we may not have control over.

I didn't arrive yet at a clean solution.

[d70-t]

Re versions I can see your point about communicating clearly when data versions change, even if this is available in the file metadata.

My point is less about indicating when the data version changed but more on keeping old versions accessible using a permanent reference. There's Hyrum's Law which I find quite important to keep in mind. So if people start using a dataset remotely in their scripts, there will be quite a few such scripts which will depend on the slightest details of the dataset. So in order not to randomly break scripts of others in interesting fashions, the data for a given reference may not change at all. Otherwise we would force everyone to download all the datasets they are using because they couldn't met any reproducibility guidelines anymore.

On the other hand, if people find errors in the datasets, we want to inform users of the dataset that a new version of that dataset has been created and users should check if their conclusions are still valid. A system which would enable this use case would most likely require even more cooperation of the data distributors though...