OpenFOAM Lid driven cavity test case [WIP/Open for testing] #243

satishskamath · 2025-02-25T22:15:14Z

Todos:

Creating a basic ReFrame test case from the slurm batch file.
Since there are many steps involved, need a phased output log to see if any of the steps fail then where it fails.
Current test is a strong scaling test, need to add a weak scaling test as well down the line.

satishskamath · 2025-05-14T11:19:07Z

@casparvl @laraPPr and @smoors this PR is open for testing on the systems, right now. I am still fixing formatting and trying to satisfy the linter but I think more or less the test is done.

satishskamath · 2025-05-15T13:58:25Z

Btw the linter does not recognise reframe pipeline hooks:

./eessi/testsuite/tests/apps/openfoam/openfoam.py:104:6: F821 undefined name 'deferrable'

Have any of you encountered this before? @smoors , @laraPPr , @casparvl ?

"\" in a string leads to extra space between the characters in the string. Correct way of doing it: ``` my_string = "The quick brown dog " \ "jumped over the lazy fox" ```

satishskamath · 2025-05-16T23:09:13Z

[satishk@tcn3 projects]$ reframe -C eessi_reframe/settings_example.py -c test-suite/eessi/testsuite/tests/apps/openfoam/openfoam.py    --system="snellius_eessi:cpu_rome" --run
[ReFrame Setup]
  version:           4.6.1
  command:           '/home/satishk/.local/easybuild/RHEL8/2023/software/ReFrame/4.6.1/bin/reframe -C eessi_reframe/settings_example.py -c test-suite/eessi/testsuite/tests/apps/openfoam/openfoam.py --system=snellius_eessi:cpu_rome --run'
  launched by:       [email protected]
  working directory: '/gpfs/home5/satishk/projects'
  settings files:    '<builtin>', 'eessi_reframe/settings_example.py'
  check search path: '/gpfs/home5/satishk/projects/test-suite/eessi/testsuite/tests/apps/openfoam/openfoam.py'
  stage directory:   '/scratch-shared/satishk/reframe_output/staging'
  output directory:  '/gpfs/home5/satishk/projects/output'
  log files:         '/gpfs/home5/satishk/projects/reframe.log'

[==========] Running 4 check(s)
[==========] Started on Fri May 16 11:24:32 2025+0200

[----------] start processing checks
[ RUN      ] EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=16_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /2f8f29fc @snellius_eessi:cpu_rome+default
[ RUN      ] EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=8_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /0e2a6d3a @snellius_eessi:cpu_rome+default
[ RUN      ] EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=4_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /e79508b8 @snellius_eessi:cpu_rome+default
[ RUN      ] EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=2_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /827f27ca @snellius_eessi:cpu_rome+default
[       OK ] (1/4) EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=2_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /827f27ca @snellius_eessi:cpu_rome+default
P: perf: 1.0 s/timestep (r:0, l:None, u:None)
[       OK ] (2/4) EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=4_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /e79508b8 @snellius_eessi:cpu_rome+default
P: perf: 0.8666666666666667 s/timestep (r:0, l:None, u:None)
[       OK ] (3/4) EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=16_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /2f8f29fc @snellius_eessi:cpu_rome+default
P: perf: 1.0666666666666667 s/timestep (r:0, l:None, u:None)
[       OK ] (4/4) EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=8_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /0e2a6d3a @snellius_eessi:cpu_rome+default
P: perf: 0.4666666666666667 s/timestep (r:0, l:None, u:None)
[----------] all spawned checks have finished

[  PASSED  ] Ran 4/4 test case(s) from 4 check(s) (0 failure(s), 0 skipped, 0 aborted)
[==========] Finished on Fri May 16 12:08:58 2025+0200
Log file(s) saved in '/gpfs/home5/satishk/projects/reframe.log'

Test on rome nodes Snellius.

laraPPr · 2025-05-21T09:05:38Z

I merged with upstream so that I could get easily the changes in the hortense config and I'm getting this error.

For testing should I not merge with main?

ERROR: run session stopped: import error: cannot import name 'COMPUTE_UNIT' from 'eessi.testsuite.constants' (/dodrio/scratch/projects/gadminforever/vsc46128/test-suite/eessi/testsuite/constants.py)

ERROR: Traceback (most recent call last):

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/cli.py", line 1269, in main

    checks_found = loader.load_all(force=True)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/core/logging.py", line 1054, in _fn

    return fn(*args, **kwargs)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 267, in load_all

    checks += self.load_from_dir(d, self._recurse, force)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 239, in load_from_dir

    checks += self.load_from_dir(entry.path, recurse, force)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 246, in load_from_dir

    checks += self.load_from_file(entry.path, force)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 219, in load_from_file

    util.import_module_from_file(filename, force,

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/utility/__init__.py", line 118, in import_module_from_file

    return _do_import_module_from_file(filename, module_name)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/utility/__init__.py", line 70, in _do_import_module_from_file

    spec.loader.exec_module(module)

  File "<frozen importlib._bootstrap_external>", line 850, in exec_module

  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

  File "/dodrio/scratch/projects/gadminforever/vsc46128/test-suite/eessi/testsuite/tests/apps/openfoam/openfoam.py", line 34, in <module>

    from eessi.testsuite.constants import COMPUTE_UNIT, DEVICE_TYPES, SCALES

ImportError: cannot import name 'COMPUTE_UNIT' from 'eessi.testsuite.constants' (/dodrio/scratch/projects/gadminforever/vsc46128/test-suite/eessi/testsuite/constants.py)

satishskamath · 2025-05-21T13:29:38Z

I merged with upstream so that I could get easily the changes in the hortense config and I'm getting this error.

For testing should I not merge with main?

ERROR: run session stopped: import error: cannot import name 'COMPUTE_UNIT' from 'eessi.testsuite.constants' (/dodrio/scratch/projects/gadminforever/vsc46128/test-suite/eessi/testsuite/constants.py)

ERROR: Traceback (most recent call last):

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/cli.py", line 1269, in main

    checks_found = loader.load_all(force=True)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/core/logging.py", line 1054, in _fn

    return fn(*args, **kwargs)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 267, in load_all

    checks += self.load_from_dir(d, self._recurse, force)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 239, in load_from_dir

    checks += self.load_from_dir(entry.path, recurse, force)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 246, in load_from_dir

    checks += self.load_from_file(entry.path, force)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/frontend/loader.py", line 219, in load_from_file

    util.import_module_from_file(filename, force,

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/utility/__init__.py", line 118, in import_module_from_file

    return _do_import_module_from_file(filename, module_name)

  File "/dodrio/scratch/projects/gadminforever/vsc46128/RHEL9_reframe_venv/lib64/python3.9/site-packages/reframe/utility/__init__.py", line 70, in _do_import_module_from_file

    spec.loader.exec_module(module)

  File "<frozen importlib._bootstrap_external>", line 850, in exec_module

  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

  File "/dodrio/scratch/projects/gadminforever/vsc46128/test-suite/eessi/testsuite/tests/apps/openfoam/openfoam.py", line 34, in <module>

    from eessi.testsuite.constants import COMPUTE_UNIT, DEVICE_TYPES, SCALES

ImportError: cannot import name 'COMPUTE_UNIT' from 'eessi.testsuite.constants' (/dodrio/scratch/projects/gadminforever/vsc46128/test-suite/eessi/testsuite/constants.py)

aah wait, I probably have not merged with main for some time. I will merge it and commit again.

satishskamath · 2025-05-21T14:45:25Z

@laraPPr can you try again?

laraPPr · 2025-05-21T16:30:45Z

srun does not work on our system so we have to use a custom launcher. So all the tests are being skipped

[The chosen launcher for this test is different from mpirun or srun which means that the                      test will definitely fail, therefore skipping this test.]

satishskamath · 2025-05-22T08:27:56Z

@smoors do you also use a wrapper for the launcher on your system as well?

satishskamath · 2025-05-22T08:32:48Z

@laraPPr How have you configured your launcher? I will see if I can accommodate a custom launcher and provide it from the command line.

laraPPr · 2025-05-22T08:46:04Z

You mean did I configure it in the test-suite? (I did not) Or did I configure it in general?

laraPPr · 2025-05-22T09:18:40Z

We register the launcher in the config

test-suite/config/vsc_hortense.py

Lines 37 to 40 in b18cdfd

    
           @register_launcher('mympirun') 
        
           class MyMpirunLauncher(JobLauncher): 
        
               def command(self, job): 
        
                   return ['mympirun', '--hybrid', str(job.num_tasks_per_node)]

laraPPr · 2025-05-22T10:15:07Z

I swithched the launcher to test but I'm constantly getting

sbatch: error: QOSGrpSubmitJobsLimit

sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

but when I shedule the job from the staging dir with the same sbatch command it gets submitted. I've also said max_jobs to 1 but it keeps giving me this error.

And I'm not seeing it in the daily run. So this seems to be something that changed in this pr or changes in the test-suite that aren't included in 0.6.0.

I'll see if I also git this when I run the test-suite with the main branch.

Udate:
It does not seem like it has anything to do with the test-suite version. I'll try to figure out what it causing it with testing but is not happening with the CI run

satishskamath · 2025-05-22T13:49:30Z

@laraPPr Too many jobs on your system?

QOSGrpSubmitJobsLimit — The job's QOS has reached the maximum number of jobs that can be running or pending in aggregate at a given time.

laraPPr · 2025-05-22T14:16:04Z

It is specific to the ReFrame version so I don't think so. Just started a run 4.6.3. And those are running.

laraPPr · 2025-05-22T14:43:32Z

Did I do something wrong? This is with the installation in our central stack. I'll try the EESSI stack tomorrow

--> FOAM FATAL ERROR: (openfoam-2406)

cannot find file "/dodrio/scratch/projects/gadminforever/vsc46128/test-suite/stage/hortense/cpu_milan_rhel9/default/EESSI_OPENFOAM_LID_DRIVEN_CAVITY_ada78e7a/system/controlDict"

laraPPr · 2025-05-22T14:49:35Z

And than I'm also seeing this

using ORTE failures because I'm not using our custom launcher 
An ORTE daemon has unexpectedly failed after launch and before

communicating back to mpirun. This could be caused by a number

of factors, including an inability to create a connection back

to mpirun due to a lack of common network interfaces and/or no

route found between them. Please check network connectivity

(including firewalls and network routing requirements).

smoors · 2025-05-23T08:19:44Z

i get the following error after about 5 minutes:

Build  : _1d8f0d55f7-20231221 OPENFOAM=2312 version=v2312
Arch   : "LSB;label=32;scalar=64"
Exec   : blockMesh
--- rfm_job.out ---
--- rfm_job.err (first 10 lines) ---
/var/spool/slurm/slurmd/job10782472/slurm_script: line 17: cd: ./cavity3D/8M/fixedTol: No such file or directory                                                                                                


--> FOAM FATAL ERROR: (openfoam-2312)
Cannot open file "system/decomposeParDict"

    From int main(int, char**)
    in file foamDictionary.C at line 374.

FOAM exiting
--- rfm_job.err ---
Traceback (most recent call last):
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10009/reframe/eessitestsuite/venv_refr4.6.2/lib64/python3.9/site-packages/reframe/frontend/executors/__init__.py", line 317, in _safe_call              
    return fn(*args, **kwargs)
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10009/reframe/eessitestsuite/venv_refr4.6.2/lib64/python3.9/site-packages/reframe/core/hooks.py", line 109, in _fn                                      
    func(obj, *args, **kwargs)
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10009/reframe/eessitestsuite/venv_refr4.6.2/lib64/python3.9/site-packages/reframe/core/pipeline.py", line 2121, in sanity                               
    self.check_sanity()
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10009/reframe/eessitestsuite/venv_refr4.6.2/lib64/python3.9/site-packages/reframe/core/pipeline.py", line 2184, in check_sanity                         
    raise SanityError()
reframe.core.exceptions.SanityError

smoors · 2025-05-23T09:12:36Z

@satishskamath the cavity3D directory is not staged.
you should move it into a src directory (or another one and set sourcesdir).

satishskamath · 2025-05-23T13:53:35Z

@smoors and @laraPPr I moved it now to the src directory. In my local tree I had it, but I think I forgot to move the folder in a commit, that is why it was not reflected here. Can both of you try again? Thanks.

smoors · 2025-05-25T09:06:03Z

it works now in hydra:

[==========] Running 1 check(s)
[==========] Started on Sat May 24 23:00:40 2025+0200

[----------] start processing checks
[ RUN      ] EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=2_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /827f27ca @hydra:skylake+default                       
[       OK ] (1/1) EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=2_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /827f27ca @hydra:skylake+default                 
P: perf: 5.066666666666666 s/timestep (r:0, l:None, u:None)
==> setup: 0.092s compile: 0.010s run: 473.478s sanity: 0.073s performance: 0.003s total: 473.591s                                                                          
[----------] all spawned checks have finished

[  PASSED  ] Ran 1/1 test case(s) from 1 check(s) (0 failure(s), 0 skipped, 0 aborted)                                                                                      
[==========] Finished on Sat May 24 23:08:34 2025+0200

=============================================================================================================================================================================
PERFORMANCE REPORT
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[EESSI_OPENFOAM_LID_DRIVEN_CAVITY %scale=2_nodes %device_type=cpu %module_name=OpenFOAM/v2312-foss-2023a /827f27ca @hydra:skylake:default]                                  
  num_tasks_per_node: 40
  num_tasks: 80
  num_cpus_per_task: 1
  performance:
    - perf: 5.066666666666666 s/timestep (r: 0 s/timestep l: -inf% u: +inf%)

satishskamath · 2025-05-27T11:25:56Z

@laraPPr did it work for you?

laraPPr · 2025-05-27T11:50:01Z

Haven't tried yet but i suspect the launcher will pop-up as a problem. Will try to test it tomorrow.

Satish Kamath and others added 12 commits January 31, 2024 00:04

First commit, creating openfoam from https://github.com/EESSI/eessi-d…

5d74821

…emo/blob/main/OpenFOAM/run.sh .

Merge branch 'main' into openfoam

0c338ce

Merge branch 'main' into openfoam

fac4f70

Merge branch 'main' into openfoam

542fe0b

Merge branch 'main' of github.com:EESSI/test-suite into openfoam

70059ef

First commit for OpenFOAM

3776ad5

Merge branch 'main' into openfoam

59fcc3e

Merge branch 'main' into openfoam

95d0551

Merge branch 'main' into openfoam

c92f2b9

Adding the OpenFOAM testcase [WIP]

d4d58db

Adding the main Lid driven cavity settings file.

530157a

Started adding fixtures. Still WIP.

7a075dd

casparvl marked this pull request as draft March 13, 2025 12:10

Satish Kamath added 3 commits March 13, 2025 14:08

Committing the most recent changes.

0d3a857

Merge branch 'main' into openfoam

1c57bc2

Modified the OpenFOAM test greatly to include only one class.

e248808

satishskamath changed the title ~~OpenFOAM Lid driven cavity test case [WIP]~~ OpenFOAM Lid driven cavity test case [WIP/Open for testing] May 14, 2025

Satish Kamath added 3 commits May 15, 2025 10:58

Minor changes in regex sanity expressions.

c0171e8

Changed required_mem_per_node and some linter suggestions added

e2084c6

Linter woes partially fixed

58b72bd

Satish Kamath added 7 commits May 15, 2025 16:03

Linter woes part 2.

2aed090

Linter woes part 3.

8257300

Linter woes part 4. :D

b012736

Linter based changes in evalLogCalcFixedIter file

4f78771

Hopefully the final linter woe.

5e82a9c

Satisfying the linter introduced an issue

24b4d36

"\" in a string leads to extra space between the characters in the string. Correct way of doing it: ``` my_string = "The quick brown dog " \ "jumped over the lazy fox" ```

Clearing yet another linter issue

ad7ae3d

Satish Kamath added 2 commits May 21, 2025 15:44

Merge branch 'main' into openfoam

b5416b6

Making changes for constants changed from list to tuples

5fe8d6b

Satish Kamath added 4 commits May 23, 2025 15:30

Adding files to the src folder and deleting it from the main folder.

300b80b

For some reason src files didn't get added.

be7dc79

Trying to satisfy the linter

5753c50

Linter should now be satisfied. (hope :))

506b2c3

OpenFOAM Lid driven cavity test case [WIP/Open for testing] #243

Are you sure you want to change the base?

OpenFOAM Lid driven cavity test case [WIP/Open for testing] #243

Uh oh!

Conversation

satishskamath commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Todos:

Uh oh!

satishskamath commented May 14, 2025

Uh oh!

satishskamath commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

satishskamath commented May 16, 2025

Uh oh!

laraPPr commented May 21, 2025

Uh oh!

satishskamath commented May 21, 2025

Uh oh!

satishskamath commented May 21, 2025

Uh oh!

laraPPr commented May 21, 2025

Uh oh!

satishskamath commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

satishskamath commented May 22, 2025

Uh oh!

laraPPr commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laraPPr commented May 22, 2025

Uh oh!

laraPPr commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

satishskamath commented May 22, 2025

Uh oh!

laraPPr commented May 22, 2025

Uh oh!

laraPPr commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laraPPr commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smoors commented May 23, 2025

Uh oh!

smoors commented May 23, 2025

Uh oh!

satishskamath commented May 23, 2025

Uh oh!

smoors commented May 25, 2025

Uh oh!

satishskamath commented May 27, 2025

Uh oh!

laraPPr commented May 27, 2025

Uh oh!

Uh oh!

satishskamath commented Feb 25, 2025 •

edited

Loading

satishskamath commented May 15, 2025 •

edited

Loading

satishskamath commented May 22, 2025 •

edited

Loading

laraPPr commented May 22, 2025 •

edited

Loading

laraPPr commented May 22, 2025 •

edited

Loading

laraPPr commented May 22, 2025 •

edited

Loading

laraPPr commented May 22, 2025 •

edited

Loading