[WIP] multizone CPU/GPU test harness #1869

BenWibking · 2025-10-24T03:59:39Z

This problem is designed to go through (nearly) the full range of densities/temperatures we encounter a hydro sim without doing hydro.

Prints relative L_1, L_2, and L_inf error norms of the GPU solution wrt the CPU solution.
(All error norms are relative to the CPU solution norm.)

BenWibking · 2025-10-24T16:01:55Z

On Frontier with rocm/6.4.3, I get this output:

starting the multi-zone burn...
  CPU multi-zone burn complete; zones failed = 0
  CPU multi-zone temperature range: [3032.437486, 3033.069765]
  CPU multi-zone retries applied: 1
  GPU multi-zone burn complete; zones failed = 0
  GPU multi-zone temperature range: [3029.019663, 3037.23735]
  GPU multi-zone retries applied: 33
  zone 0 T mismatch: CPU = 3032.689294, GPU = 3032.815841

Re-running now to compare with the "known good" rocm/6.3.1...

BenWibking · 2025-10-24T16:05:39Z

rocm/6.3.1:

starting the multi-zone burn...
CPU multi-zone burn complete; zones failed = 0
CPU multi-zone temperature range: [3032.437486, 3033.069765]
CPU multi-zone retries applied: 1
GPU multi-zone burn complete; zones failed = 0
GPU multi-zone temperature range: [3033.179406, 3033.306148]
GPU multi-zone retries applied: 39
zone 0 T mismatch: CPU = 3032.689294, GPU = 3033.210804

I need to compute a better norm to compare them, but it looks like 6.4 is bad for us as well :/

zingale · 2025-10-24T16:06:59Z

to make sure the integrator comparison is the same, run with integrator.use_jacobian_caching=0

BenWibking · 2025-10-24T18:36:27Z

@zingale Jacobian caching is off.

This problem is designed to go through (nearly) the full range of densities/temperatures we encounter a hydro sim without doing hydro. All error norms are relative to the CPU solution norm.

Using rocm/6.3.1 (this is a "minimum acceptable" level of agreement -- I still don't know why the GPU burn fails so much):

starting the multi-zone burn...
CPU multi-zone burn complete; zones failed = 0
CPU multi-zone temperature range: [3032.437486, 3033.069765]
CPU multi-zone retries applied: 1
CPU multi-zone burn walltime (s): 22.2455658
GPU multi-zone burn complete; zones failed = 0
GPU multi-zone temperature range: [3033.179406, 3033.306148]
GPU multi-zone retries applied: 39
GPU multi-zone burn walltime (s): 148.3088752

CPU/GPU multi-zone difference norms:
  temperature: L1 = 0.0001577875179, L2 = 0.0001618865356, L_inf = 0.0002501927092
  number_density(E): L1 = 0.005209218474, L2 = 0.006753860604, L_inf = 0.03300111125
  number_density(Hp): L1 = 0.005209488338, L2 = 0.006754078937, L_inf = 0.03300177472
  number_density(H): L1 = 0.001276108352, L2 = 0.001309240176, L_inf = 0.002021783129
  number_density(Hm): L1 = 0.006037646397, L2 = 0.008004853872, L_inf = 0.03363601704
  number_density(Dp): L1 = 0.9773793951, L2 = 0.9949503469, L_inf = 0.9929392971
  number_density(D): L1 = 0.977414706, L2 = 0.9949523714, L_inf = 0.9929474164
  number_density(H2p): L1 = 0.004850787467, L2 = 0.006439096462, L_inf = 0.03233377457
  number_density(Dm): L1 = 0.9781727897, L2 = 0.9951584351, L_inf = 0.9929340037
  number_density(H2): L1 = 0.0003055037608, L2 = 0.0003134353239, L_inf = 0.0004843801255
  number_density(HDp): L1 = 0.9775253048, L2 = 0.9949493385, L_inf = 0.9929363062
  number_density(HD): L1 = 0.9774503449, L2 = 0.9949603367, L_inf = 0.992959557
  number_density(HEpp): L1 = 4.742830668, L2 = 3.440721513, L_inf = 3.178251298
  number_density(HEp): L1 = 0.02565528324, L2 = 0.1276112673, L_inf = 0.9903962888
  number_density(HE): L1 = 6.146444201e-08, L2 = 8.942926766e-08, L_inf = 3.738828979e-07

CPU/GPU mismatches detected:
  field T mismatched in 128 zone(s); largest difference at zone 7 (CPU = 3032.437486, GPU = 3033.196338)
  field e mismatched in 128 zone(s); largest difference at zone 7 (CPU = 2.721201708e+11, GPU = 2.722070711e+11)
  field xn(D) mismatched in 128 zone(s); largest difference at zone 22 (CPU = 16.75658685, GPU = 0.11817723)
  field xn(Dp) mismatched in 7 zone(s); largest difference at zone 22 (CPU = 3.174138657e-12, GPU = 2.241165001e-14)
  field xn(E) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 19331.7002, GPU = 20000.05813)
  field xn(H) mismatched in 128 zone(s); largest difference at zone 7 (CPU = 1.616812031e+17, GPU = 1.620086389e+17)
  field xn(H2) mismatched in 128 zone(s); largest difference at zone 7 (CPU = 3.379982657e+17, GPU = 3.378345461e+17)
  field xn(H2p) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 15.71035496, GPU = 16.24229929)
  field xn(HD) mismatched in 128 zone(s); largest difference at zone 22 (CPU = 160.5250696, GPU = 1.130167607)
  field xn(HE) mismatched in 128 zone(s); largest difference at zone 99 (CPU = 6.49133784e+16, GPU = 6.491340267e+16)
  field xn(HEp) mismatched in 2 zone(s); largest difference at zone 104 (CPU = 1e-100, GPU = 6.406499605e-12)
  field xn(Hm) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 1.727013205, GPU = 1.789152394)
  field xn(Hp) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 19317.71686, GPU = 19985.60498)

Using rocm/6.4.2 (slightly worse overall, but a lot worse for some species):

starting the multi-zone burn...
CPU multi-zone burn complete; zones failed = 0
CPU multi-zone temperature range: [3032.437486, 3033.069765]
CPU multi-zone retries applied: 1
CPU multi-zone burn walltime (s): 22.2893873
GPU multi-zone burn complete; zones failed = 0
GPU multi-zone temperature range: [3029.019663, 3037.23735]
GPU multi-zone retries applied: 33
GPU multi-zone burn walltime (s): 73.56593662

CPU/GPU multi-zone difference norms:
  temperature: L1 = 9.056564017e-05, L2 = 0.0002005176156, L_inf = 0.001545567878
  number_density(E): L1 = 0.003980398191, L2 = 0.005534918028, L_inf = 0.02641261182
  number_density(Hp): L1 = 0.00398043237, L2 = 0.005534968643, L_inf = 0.02641309697
  number_density(H): L1 = 0.0007323882078, L2 = 0.001622249965, L_inf = 0.01252560215
  number_density(Hm): L1 = 0.004824476122, L2 = 0.007289587544, L_inf = 0.0361106126
  number_density(Dp): L1 = 10.53935966, L2 = 48.17332479, L_inf = 92.24618187
  number_density(D): L1 = 10.68886968, L2 = 48.998352, L_inf = 93.88338289
  number_density(H2p): L1 = 0.003899314924, L2 = 0.005438905368, L_inf = 0.0260830975
  number_density(Dm): L1 = 11.31745067, L2 = 50.84429965, L_inf = 93.06558273
  number_density(H2): L1 = 0.0001753435257, L2 = 0.0003883778746, L_inf = 0.003000808856
  number_density(HDp): L1 = 10.51675715, L2 = 48.34467731, L_inf = 92.57997537
  number_density(HD): L1 = 10.55548406, L2 = 48.29096892, L_inf = 92.51377107
  number_density(HEpp): L1 = 1.014233689, L2 = 0.8154069422, L_inf = 0.7650545114
  number_density(HEp): L1 = 0.1078060008, L2 = 0.3210079961, L_inf = 0.9875212077
  number_density(HE): L1 = 1.565261702e-07, L2 = 3.53309392e-07, L_inf = 2.963830149e-06

CPU/GPU mismatches detected:
  field T mismatched in 128 zone(s); largest difference at zone 124 (CPU = 3032.549534, GPU = 3037.23735)
  field e mismatched in 128 zone(s); largest difference at zone 124 (CPU = 2.721329956e+11, GPU = 2.72670329e+11)
  field xn(D) mismatched in 128 zone(s); largest difference at zone 124 (CPU = 2.397211316, GPU = 1575.562271)
  field xn(Dp) mismatched in 8 zone(s); largest difference at zone 124 (CPU = 4.479440013e-13, GPU = 2.932501159e-10)
  field xn(E) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 19331.7002, GPU = 19866.62396)
  field xn(H) mismatched in 128 zone(s); largest difference at zone 124 (CPU = 1.617295325e+17, GPU = 1.637581034e+17)
  field xn(H2) mismatched in 128 zone(s); largest difference at zone 124 (CPU = 3.379740888e+17, GPU = 3.369598206e+17)
  field xn(H2p) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 15.71035496, GPU = 16.13946526)
  field xn(HD) mismatched in 128 zone(s); largest difference at zone 124 (CPU = 22.9727526, GPU = 14873.75229)
  field xn(HE) mismatched in 128 zone(s); largest difference at zone 98 (CPU = 6.491340265e+16, GPU = 6.491359504e+16)
  field xn(HEp) mismatched in 13 zone(s); largest difference at zone 57 (CPU = 6.387901791e-12, GPU = 1.000000088e-100)
  field xn(Hm) mismatched in 128 zone(s); largest difference at zone 53 (CPU = 1.847400329, GPU = 1.780689571)
  field xn(Hp) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 19317.71686, GPU = 19852.26368)

Using rocm/7.0.2 (actually looks OK for us):

starting the multi-zone burn...
CPU multi-zone burn complete; zones failed = 0
CPU multi-zone temperature range: [3032.437486, 3033.069765]
CPU multi-zone retries applied: 1
CPU multi-zone burn walltime (s): 21.61658048
GPU multi-zone burn complete; zones failed = 0
GPU multi-zone temperature range: [3032.180939, 3033.197728]
GPU multi-zone retries applied: 24
GPU multi-zone burn walltime (s): 54.81239564

CPU/GPU multi-zone difference norms:
  temperature: L1 = 4.403323795e-05, L2 = 5.887483029e-05, L_inf = 0.0002261898486
  number_density(E): L1 = 0.004475270452, L2 = 0.006215328124, L_inf = 0.02840464773
  number_density(Hp): L1 = 0.004475191463, L2 = 0.006215284227, L_inf = 0.02840558136
  number_density(H): L1 = 0.0003560075324, L2 = 0.0004759775774, L_inf = 0.001826624112
  number_density(Hm): L1 = 0.005069360699, L2 = 0.007530198415, L_inf = 0.03478690901
  number_density(Dp): L1 = 1.221532946, L2 = 1.06892868, L_inf = 0.9835793297
  number_density(D): L1 = 1.220648686, L2 = 1.067929547, L_inf = 0.9835910807
  number_density(H2p): L1 = 0.004427627635, L2 = 0.006160110736, L_inf = 0.02803956417
  number_density(Dm): L1 = 1.283818822, L2 = 1.09576158, L_inf = 0.9835746268
  number_density(H2): L1 = 8.523078908e-05, L2 = 0.0001139545031, L_inf = 0.0004377261334
  number_density(HDp): L1 = 1.221549719, L2 = 1.068941756, L_inf = 0.9835783071
  number_density(HD): L1 = 1.220564661, L2 = 1.067872824, L_inf = 0.9835951946
  number_density(HEpp): L1 = 1.013570425, L2 = 0.814533011, L_inf = 0.7650545114
  number_density(HEp): L1 = 0.1711070716, L2 = 0.4084160359, L_inf = 0.9912271414
  number_density(HE): L1 = 1.090274196e-07, L2 = 1.50601551e-07, L_inf = 5.782735312e-07

CPU/GPU mismatches detected:
  field T mismatched in 128 zone(s); largest difference at zone 55 (CPU = 3032.866989, GPU = 3032.180939)
  field e mismatched in 128 zone(s); largest difference at zone 55 (CPU = 2.721693405e+11, GPU = 2.720908055e+11)
  field xn(D) mismatched in 128 zone(s); largest difference at zone 22 (CPU = 16.75658685, GPU = 0.2749574817)
  field xn(Dp) mismatched in 9 zone(s); largest difference at zone 22 (CPU = 3.174138657e-12, GPU = 5.212148444e-14)
  field xn(E) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 19331.7002, GPU = 19906.96784)
  field xn(H) mismatched in 128 zone(s); largest difference at zone 55 (CPU = 1.618664577e+17, GPU = 1.615706287e+17)
  field xn(H2) mismatched in 128 zone(s); largest difference at zone 55 (CPU = 3.379056131e+17, GPU = 3.380535638e+17)
  field xn(H2p) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 15.71035496, GPU = 16.17165239)
  field xn(HD) mismatched in 128 zone(s); largest difference at zone 22 (CPU = 160.5250696, GPU = 2.633382536)
  field xn(HE) mismatched in 128 zone(s); largest difference at zone 119 (CPU = 6.491340533e+16, GPU = 6.491344287e+16)
  field xn(HEp) mismatched in 21 zone(s); largest difference at zone 1 (CPU = 6.411874076e-12, GPU = 1e-100)
  field xn(Hm) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 1.727013205, GPU = 1.791278553)
  field xn(Hp) mismatched in 128 zone(s); largest difference at zone 104 (CPU = 19317.71686, GPU = 19892.58747)

zingale and others added 17 commits October 19, 2025 08:22

start of support for reduced precision Jacobian

78487a4

fix some types

13d0b7b

add control

4c2e7d8

update defines

cb600d6

add docs

f16ee32

Merge branch 'development' into single_precision_jac

678b067

add AGENTS

ceceb51

one-zone ParallelFor

4aa8806

add multi-zone burn

140b4a7

fix multi-zone burn

3bb5898

suppress verbosity

002ff37

suppress more verbosity

0459efd

formatting

1bd729e

remove verbosity

7cb6281

adjust inputs

f73248e

do burn retries on GPU

4e67db2

compare CPU/GPU runs

141e251

summary statistics

e2d0ad0

zingale changed the base branch from main to development October 24, 2025 16:08

BenWibking added 5 commits October 24, 2025 13:54

compute diff norms

77b6d79

remove single-zone burn

0262669

report relative norms

ee66b58

report timings

319610c

turn off debug

d0337fd

BenWibking mentioned this pull request Oct 24, 2025

block ROCm 7.0 or newer in CMakeLists.txt quokka-astro/quokka#1409

Closed

add n_zones parameter

01b4c78

BenWibking and others added 3 commits October 24, 2025 15:01

add OMP on CPU

8d221e6

Merge branch 'development' into multizone-rocm-test

7640e16

Merge branch 'single_precision_jac' into multizone-rocm-test

a708853

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] multizone CPU/GPU test harness #1869

[WIP] multizone CPU/GPU test harness #1869

Uh oh!

BenWibking commented Oct 24, 2025 •

edited

Loading

Uh oh!

BenWibking commented Oct 24, 2025

Uh oh!

BenWibking commented Oct 24, 2025

Uh oh!

zingale commented Oct 24, 2025

Uh oh!

BenWibking commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[WIP] multizone CPU/GPU test harness #1869

Are you sure you want to change the base?

[WIP] multizone CPU/GPU test harness #1869

Uh oh!

Conversation

BenWibking commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BenWibking commented Oct 24, 2025

Uh oh!

BenWibking commented Oct 24, 2025

Uh oh!

zingale commented Oct 24, 2025

Uh oh!

BenWibking commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BenWibking commented Oct 24, 2025 •

edited

Loading