Skip to content

Post-processing: Refactor low-level data representation into a numpy array of size (n_ranks, n_threads) #234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
EdHone wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Post-processing: Refactor low-level data representation into a numpy array of size (n_ranks, n_threads) #234

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0d033af
Convert to fixed allocation of numpy array rather than list
EdHone Mar 9, 2026
9167f57
Switch to numpy array implementation
EdHone Mar 13, 2026
6ce7bd6
Code implementation for handling multithreaded data
EdHone Mar 13, 2026
b5b8b90
Tests passing for numpy implementation
EdHone Mar 16, 2026
dfa8c81
Re-instate tests and fix
EdHone Mar 16, 2026
6bad072
Fix txt output
EdHone Mar 16, 2026
1c6025b
updated array initialisation to be filled with nans rather than zeroes
EdHone Mar 17, 2026
fe38db8
Fix final issues
EdHone Mar 17, 2026
cefb573
Merge branch 'main' into py-vernier-arrays
EdHone Mar 17, 2026
94b4d2d
Sign CLA
EdHone Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CONTRIBUTORS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@
|-----------------|------------------|-------------|------------|
| andrewcoughtrie | Andrew Coughtrie | Met Office | 2026-03-05 |
| mo-marqh | mark Hedley | Met Office | 2026-03-10 |
| oakleybrunt | Oakley Brunt | Met Office | 2026-03-09 |
| oakleybrunt | Oakley Brunt | Met Office | 2026-03-09 |
| EdHone | Ed Hone | Met Office | 2026-03-17 |
8 changes: 4 additions & 4 deletions post-processing/tests/data/vernier-output-test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ Profiling on 1 thread(s).
(Size; Size/sec; Size/call; MinSize; MaxSize)
(self) (sec) (sec) (sec) ms/call ms/call

1 44.130 2.583 2.583 5.854 1 2583.167 5853.527 __test_app__
2 34.563 4.606 2.023 2.077 2 1011.582 1038.697 some_process
1 44.130 2.583 2.583 5.854 1 2583.167 5853.527 __test_app__@0
2 34.563 4.606 2.023 2.077 2 1011.582 1038.697 some_process@0

Task 2 of 2 : MPI rank ID 1
Profiling on 1 thread(s).
Expand All @@ -16,5 +16,5 @@ Profiling on 1 thread(s).
(Size; Size/sec; Size/call; MinSize; MaxSize)
(self) (sec) (sec) (sec) ms/call ms/call

1 43.991 3.009 3.009 3.069 2 1504.362 1534.740 some_process
2 37.835 5.596 2.588 6.839 1 2587.638 6839.336 __test_app__
1 43.991 3.009 3.009 3.069 2 1504.362 1534.740 some_process@0
2 37.835 5.596 2.588 6.839 1 2587.638 6839.336 __test_app__@0
21 changes: 21 additions & 0 deletions post-processing/tests/data/vernier-output-threaded/vernier-output-0
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@

Task 1 of 2 : MPI rank ID 0
Profiling on 4 thread(s).

# % Time Cumul Self Total # of calls Self Total Routine@
(Size; Size/sec; Size/call; MinSize; MaxSize)
(self) (sec) (sec) (sec) ms/call ms/call

1 40.000 2.000 2.000 3.000 2 1000.075 1500.115 MAIN_SUB@0
2 40.000 4.000 2.000 5.000 1 2000.140 5000.373 FULL@0
3 20.000 5.000 1.000 1.000 1 1000.073 1000.073 MAIN_SUB2@3
4 20.000 6.000 1.000 2.000 1 1000.072 2000.152 MAIN_SUB@3
5 20.000 7.001 1.000 1.000 1 1000.072 1000.072 MAIN_SUB2@0
6 20.000 8.001 1.000 1.000 1 1000.071 1000.071 MAIN_SUB2@2
7 20.000 9.001 1.000 1.000 1 1000.069 1000.069 MAIN_SUB2@1
8 20.000 10.001 1.000 2.000 1 1000.068 2000.151 MAIN_SUB@2
9 20.000 11.001 1.000 2.000 1 1000.067 2000.150 MAIN_SUB@1
10 0.000 11.001 0.000 0.000 2 0.008 0.008 __vernier__@2
11 0.000 11.001 0.000 0.000 4 0.004 0.004 __vernier__@0
12 0.000 11.001 0.000 0.000 2 0.008 0.008 __vernier__@1
13 0.000 11.001 0.000 0.000 2 0.007 0.007 __vernier__@3
21 changes: 21 additions & 0 deletions post-processing/tests/data/vernier-output-threaded/vernier-output-1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@

Task 2 of 2 : MPI rank ID 1
Profiling on 4 thread(s).

# % Time Cumul Self Total # of calls Self Total Routine@
(Size; Size/sec; Size/call; MinSize; MaxSize)
(self) (sec) (sec) (sec) ms/call ms/call

1 40.031 2.004 2.004 5.006 1 2003.948 5005.940 FULL@0
2 39.956 4.004 2.000 3.002 2 1000.084 1500.991 MAIN_SUB@0
3 20.056 5.008 1.004 1.004 1 1003.995 1003.995 MAIN_SUB2@1
4 20.017 6.010 1.002 1.002 1 1002.034 1002.034 MAIN_SUB2@3
5 20.012 7.012 1.002 1.002 1 1001.809 1001.809 MAIN_SUB2@0
6 19.979 8.012 1.000 1.000 1 1000.149 1000.149 MAIN_SUB2@2
7 19.978 9.012 1.000 2.004 1 1000.063 2004.060 MAIN_SUB@1
8 19.978 10.012 1.000 2.002 1 1000.063 2002.101 MAIN_SUB@3
9 19.977 11.012 1.000 2.000 1 1000.038 2000.191 MAIN_SUB@2
10 0.002 11.012 0.000 0.000 2 0.052 0.052 __vernier__@2
11 0.002 11.012 0.000 0.000 2 0.052 0.052 __vernier__@1
12 0.002 11.013 0.000 0.000 2 0.052 0.052 __vernier__@3
13 0.001 11.013 0.000 0.000 4 0.007 0.007 __vernier__@0
4 changes: 2 additions & 2 deletions post-processing/tests/data/vernier-output/vernier-output-test-0
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ Profiling on 1 thread(s).
(Size; Size/sec; Size/call; MinSize; MaxSize)
(self) (sec) (sec) (sec) ms/call ms/call

1 44.130 2.583 2.583 5.854 1 2583.167 5853.527 __test_app__
2 34.563 4.606 2.023 2.077 2 1011.582 1038.697 some_process
1 44.130 2.583 2.583 5.854 1 2583.167 5853.527 __test_app__@0
2 34.563 4.606 2.023 2.077 2 1011.582 1038.697 some_process@0
4 changes: 2 additions & 2 deletions post-processing/tests/data/vernier-output/vernier-output-test-1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ Profiling on 1 thread(s).
(Size; Size/sec; Size/call; MinSize; MaxSize)
(self) (sec) (sec) (sec) ms/call ms/call

1 43.991 3.009 3.009 3.069 2 1504.362 1534.740 some_process
2 37.835 5.596 2.588 6.839 1 2587.638 6839.336 __test_app__
1 43.991 3.009 3.009 3.069 2 1504.362 1534.740 some_process@0
2 37.835 5.596 2.588 6.839 1 2587.638 6839.336 __test_app__@0
6 changes: 3 additions & 3 deletions post-processing/tests/test_cli_tools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ def setUp(self):
self.test_data_dir = Path(__file__).parent / 'data'
# pylint: disable=line-too-long
self.test_data_kgo = (
'| Routine | Total time (s) | Self (s) | Cumul time (s) | No. calls | % time | Time per call (s) |\n' +
'| __test_app__ | 6.3465 | 2.5855 | 4.0895 | 1 | 40.9825 | 6.3465 |\n' +
'| some_process | 2.573 | 2.516 | 3.8075 | 2 | 39.277 | 1.2865 |\n'
'| Routine | Total time (s) | Self (s) | Cumul time (s) | Max no. calls | % time | Time per call (s) |\n' +
'| __test_app__ | 6.3465 | 2.5855 | 4.0895 | 1 | 40.9825 | 6.3465 |\n' +
'| some_process | 2.573 | 2.516 | 3.8075 | 2 | 39.277 | 1.2865 |\n'
)
return super().setUp()

Expand Down
47 changes: 23 additions & 24 deletions post-processing/tests/test_vernier_calliper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
"""

import unittest
import numpy as np
from pathlib import Path
import sys

Expand All @@ -24,30 +25,29 @@ def setUp(self):
"""
Initialise useful attributes for testing.
"""
self.calliper_a = VernierCalliper("test_calliper_a")
self.calliper_b = VernierCalliper("test_calliper_b")
self.calliper_a = VernierCalliper("test_calliper_a", 2, 1)
self.calliper_b = VernierCalliper("test_calliper_b", 2, 1)

def test_init(self):
"""
Test that the class initialises as expected (empty lists)
"""
self.assertEqual(self.calliper_a.name, "test_calliper_a")
self.assertEqual(self.calliper_a.time_percent, [])
self.assertEqual(self.calliper_a.cumul_time, [])
self.assertEqual(self.calliper_a.self_time, [])
self.assertEqual(self.calliper_a.total_time, [])
self.assertEqual(self.calliper_a.n_calls, [])
self.assertEqual(self.calliper_a.time_percent.shape, (2, 1))
self.assertEqual(self.calliper_a.cumul_time.shape, (2, 1))
self.assertEqual(self.calliper_a.self_time.shape, (2, 1))
self.assertEqual(self.calliper_a.total_time.shape, (2, 1))
self.assertEqual(self.calliper_a.n_calls.shape, (2, 1))

def test_reduce(self):
"""
Test that the reduce() method returns the expected results.
"""
# pylint: disable=pointless-statement
self.calliper_a.time_percent = [10.0, 20.0]
self.calliper_a.cumul_time = [30.0, 40.0]
self.calliper_a.self_time = [5.0, 15.0]
self.calliper_a.total_time = [25.0, 35.0]
self.calliper_a.n_calls = [2, 2]
self.calliper_a.time_percent = np.array([10.0, 20.0])
self.calliper_a.cumul_time = np.array([30.0, 40.0])
self.calliper_a.self_time = np.array([5.0, 15.0])
self.calliper_a.total_time = np.array([25.0, 35.0])
self.calliper_a.n_calls = np.array([2, 2])

reduced_data = self.calliper_a.reduce()
self.assertEqual(reduced_data[0], "test_calliper_a")
Expand All @@ -63,18 +63,17 @@ def test_compare(self):
Test that the comparison of dataclasses works as expected for the
VernierCalliper class.
"""
# pylint: disable=pointless-statement
self.calliper_a.time_percent = [10.0, 20.0]
self.calliper_a.cumul_time = [30.0, 40.0]
self.calliper_a.self_time = [5.0, 15.0]
self.calliper_a.total_time = [25.0, 35.0]
self.calliper_a.n_calls = [2, 2]
self.calliper_a.time_percent = np.array([10.0, 20.0])
self.calliper_a.cumul_time = np.array([30.0, 40.0])
self.calliper_a.self_time = np.array([5.0, 15.0])
self.calliper_a.total_time = np.array([25.0, 35.0])
self.calliper_a.n_calls = np.array([2, 2])

self.calliper_b.time_percent = [12.0, 25.0]
self.calliper_b.cumul_time = [35.0, 46.0]
self.calliper_b.self_time = [6.0, 19.0]
self.calliper_b.total_time = [28.0, 39.0]
self.calliper_b.n_calls = [2, 2]
self.calliper_b.time_percent = np.array([12.0, 25.0])
self.calliper_b.cumul_time = np.array([35.0, 46.0])
self.calliper_b.self_time = np.array([6.0, 19.0])
self.calliper_b.total_time = np.array([28.0, 39.0])
self.calliper_b.n_calls = np.array([2, 2])

self.assertTrue(self.calliper_a < self.calliper_b)
self.assertFalse(self.calliper_a > self.calliper_b)
Expand Down
Loading
Loading