Post-processing time-out results in loss of files from history collection

[AgilentGCMS]

I have a run in /discover/nobackup/projects/gmao/geos_carb/sbasu1/runs/GCM/GCM-11.5.2-Azure-c90 where I'm using the post-processing scripts to organize my history collections into YYYYMM folders. I have noticed that occasionally, instead of moving files into those folders, it will create files with YYYYMM names, with junk content. Cases in point:

holding/inst3_2d_carb_Nx/200906
holding/tavg3_2d_carb_Nx/200612
holding/tavg3_2d_carb_Nx/201004

Digging deeper into the logs of those post-processing jobs, I found that in all three cases, the post-processing jobs (post/gcm_post.inst3_2d_carb_Nx.o200906, post/gcm_post.tavg3_2d_carb_Nx.o200612 and post/gcm_post.tavg3_2d_carb_Nx.o201004) had timed out. Worse, the history files that were supposed to be moved to those YYYYMM folder are all gone. This should never happen, especially not on timeouts. Files should only be deleted once they exist in the destination folder.

[briardew]

I don't know how Sourish originally got files instead of folders in holding/inst3_2d_carb_Nx, but it's clear that gcmpost.script has some ... deficiencies ... that fail to identify this as a case and choke. For example, the existence check for those directories merely check that they exist -e instead of that they are directories -d (Line 533). So if it happens to be a file, the following line will just overwrite it until it gets to the last file in the month. That line could also be protected by including a trailing slash to have /bin/mv $local $SOURCE/holding/$collection/$date/ instead of /bin/mv $local $SOURCE/holding/$collection/$date. The former should die instead of just overwriting the file 30 or 31 times.

[AgilentGCMS]

How those files were created is also a mystery to me. The only thing I can be certain of is that they were created by the post-processing script.

[mathomp4]

I'm going to link in @sdrabenh and @michellefrazer as they see these scripts more often than I do.

[AgilentGCMS]

Brad has identified the issue, and I now understand why that file is created. Consider lines 533-534,

if(! -e $SOURCE/holding/$collection/$date ) mkdir -p $SOURCE/holding/$collection/$date
mv $local $SOURCE/holding/$collection/$date

If that first line fails for whatever reason, i.e., the folder is not created, the second line still proceeds, moving a file onto a YYYYMM name. The next iteration of $local (and successive iterations) pass the -e $SOURCE/holding/$collection/$date check (so mkdir is not executed and no error is raised) and the next $local is moved onto the same name. The fix is simple, (a) check for a folder, not a file, and (b) test for destination existence before moving. As added safeguard, make it explicit that the destination is a folder, not a file.

if(! -d $SOURCE/holding/$collection/$date ) mkdir -p $SOURCE/holding/$collection/$date
test -d $SOURCE/holding/$collection/$date && mv $local $SOURCE/holding/$collection/$date/

[AgilentGCMS]

An unrelated comment, why do so many scripts have hardcoded paths for system utilities like mv, cp and ls?

[briardew]

To pick nits, I would suggest something like:

set outfolder = $SOURCE/holding/$collection/$date
mkdir -p $outfolder
if (-d $outfolder) mv $local $outfolder/

The -p already protects for what the if is protecting.

[briardew]

Either way, proper coding should protect against this failure.

[AgilentGCMS]

Here's a one-liner that would work

mkdir -p $SOURCE/holding/$collection/$date && mv $local $SOURCE/holding/$collection/$date/

[mathomp4]

An unrelated comment, why do so many scripts have hardcoded paths for system utilities like mv, cp and ls?

This is sort of a hangover from olden days when at NAS we'd use mcp and mtar and the like which were (at the time) preferred for use on Lustre.

Though, that said, it is always probably a good thing to do in scripting. A user can alias mv to something different and cause hilarity. (A la someone that aliases rm to rm -i)

[briardew]

Oh dear God. But do aliases translate? They shouldn't for non-interactive shells, right? This one always confuses me. It's also starting to remind me of our cmake Python path discussion which still evokes several negative emotions, so maybe let's just ignore it?

…

On Thu, Apr 17, 2025 at 12:46 PM Matt Thompson ***@***.***> wrote: An unrelated comment, why do so many scripts have hardcoded paths for system utilities like mv, cp and ls? This is sort of a hangover from olden days when at NAS we'd use mcp and mtar and the like which were (at the time) preferred for use on Lustre. Though, that said, it is always probably a good thing to do in scripting. A user can alias mv to something different and cause hilarity. (A la someone that aliases rm to rm -i) — Reply to this email directly, view it on GitHub <#134 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKXORYU2HJNEQ32J3SYBYUD2Z7LGXAVCNFSM6AAAAAB3IWTFRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJTGUZDQNRRGU> . You are receiving this because you commented.Message ID: ***@***.***> *mathomp4* left a comment (GEOS-ESM/GEOS_Util#134) <#134 (comment)> An unrelated comment, why do so many scripts have hardcoded paths for system utilities like mv, cp and ls? This is sort of a hangover from olden days when at NAS we'd use mcp and mtar and the like which were (at the time) preferred for use on Lustre. Though, that said, it is always probably a good thing to do in scripting. A user can alias mv to something different and cause hilarity. (A la someone that aliases rm to rm -i) — Reply to this email directly, view it on GitHub <#134 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKXORYU2HJNEQ32J3SYBYUD2Z7LGXAVCNFSM6AAAAAB3IWTFRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJTGUZDQNRRGU> . You are receiving this because you commented.Message ID: ***@***.***>