Add Python bindings to Btwxt via SWIG #44

jmarrec · 2025-01-27T10:00:04Z

I was struggling to understand what EnergyPlus was doing with my Table:Lookup and Table:IndependentVariables, and wanted a way to calculate and plot what was going on (and compare with scipy.interpolate.RegularGridInterpolator).

Instead of using a debugger, or tweaking E+ source code to spit out the values I needed, I figured it would be much better to add Python bindings.

Since I had done that, I figured it could be useful for other people (at least some E+ devs, and yourselves as btwxt devs for inspection/validation purposes), so I'm proposing it for inclusion.

This PR:

Enables building Python bindings via SWIG (this is opt-in)
Adds a Pytest (registered as a Ctest) to exercise the bindings
Adds a demo notebook
Tweaks the existing workflow to build the python wheels on macos x86_64 and arm64, ubuntu x86_64 and windows x86_64
- It would be quite easy to publish them to TestPyPi / PyPi if you want to do so. An example can be found here: https://github.com/NREL/OpenStudio/blob/09472be58fa387840d7e07f93e41f4da97294478/.github/workflows/python_bindings.yml#L329-L375

Find a working workflow with artifacts here: https://github.com/jmarrec/btwxt/actions/runs/12986553478

Note: for MSVC, UseSWIG.cmake already sets it to .pyd

(Tired of windows/MSVC...)

jmarrec · 2025-01-27T10:01:29Z

.github/workflows/build-and-test.yml

          - os: macos
-            os_ver: "11"


macos-11 was removed from GHA a long time ago.

jmarrec · 2025-01-27T10:01:53Z

.github/workflows/build-and-test.yml

          - os: macos
-            os_ver: "12"
+            os_ver: "14"


macos-12 is removed.

actions/runner-images#10721

I uses 13 (x86_64) and 14 (arm64, M1)

jmarrec · 2025-01-27T10:02:23Z

docs/Python_bindings_demo.ipynb

view it rendered at https://github.com/bigladder/btwxt/blob/240d733840c0569ab42386bde8a2da59a2695b99/docs/Python_bindings_demo.ipynb

jmarrec · 2025-01-27T10:02:56Z

CMakeLists.txt

+cmake_policy(SET CMP0078 NEW)
+cmake_policy(SET CMP0086 NEW)


Couple of SWIG-related policies

jmarrec · 2025-01-27T10:03:10Z

CMakeLists.txt

@@ -32,6 +35,7 @@ endif ()
 add_subdirectory("include/${PROJECT_NAME}")
 add_subdirectory(src)
 add_subdirectory(vendor)
+add_subdirectory(python)


Add subdir. Did not change the other directories.

jmarrec · 2025-01-27T10:09:50Z

python/CMakeLists.txt

+  add_custom_target(python_package
+    COMMAND ${CMAKE_COMMAND} -E rm -Rf "build/" "dist/" "btwxt.egg-info/"
+    COMMAND ${CMAKE_COMMAND} -E echo "To create the package, cd 'python/'"
+    COMMAND ${CMAKE_COMMAND} -E echo "${Python_EXECUTABLE} setup.py bdist_wheel"
+    COMMAND ${Python_EXECUTABLE} setup.py bdist_wheel


Build the python wheel (.whl) with this custom target

jmarrec · 2025-01-27T10:10:00Z

python/CMakeLists.txt

+    COMMAND ${CMAKE_COMMAND} -E echo "To upload the package, cd 'python/'"
+    COMMAND ${CMAKE_COMMAND} -E echo "${Python_EXECUTABLE} -m twine upload --verbose --skip-existing --repository ${PYTHON_PIP_REPOSITORY} dist/*"


Do not upload, but tells you how

jmarrec · 2025-01-27T10:10:18Z

python/CMakeLists.txt

+  if (${PROJECT_NAME}_BUILD_TESTING)
+    set(test_case_name btwxt_python)
+    add_test(
+          NAME "${test_case_name}"
+          COMMAND ${Python_EXECUTABLE} -m pytest --verbose test_btwxt.py
+          WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/test
+    )
+    # Pytest Exit code 5 means all tests were deselected
+    set_tests_properties("${test_case_name}" PROPERTIES SKIP_RETURN_CODE 5)
+    set_tests_properties("${test_case_name}" PROPERTIES ENVIRONMENT "PYTHONPATH=${CMAKE_CURRENT_BINARY_DIR}/btwxt")
+  endif()


Register a pytest file as a CTest.

jmarrec · 2025-01-27T10:10:38Z

test/test_btwxt.py

+import btwxt
+import pytest
+from itertools import product
+
+def test_grid_axis():
+    axis_x = btwxt.GridAxis()
+    assert axis_x.get_length() == 0
+
+    xs = [30, 35, 40, 45, 50, 55]
+    axis_x = btwxt.GridAxis(
+        [30, 35, 40, 45, 50, 55], # Values
+        btwxt.InterpolationMethod_cubic, # interpolation method
+        btwxt.ExtrapolationMethod_linear, # extrapolation_method
+        (20.0, 60.0), # extrapolation_limits
+        "LWT", # name
+    )
+    assert axis_x.get_length() == 6
+    assert set(axis_x.get_values()) == set(xs)
+    assert axis_x.get_interpolation_method() == btwxt.InterpolationMethod_cubic
+    assert axis_x.get_extrapolation_method() == btwxt.ExtrapolationMethod_linear
+    assert axis_x.get_extrapolation_limits() == (20.0, 60.0)
+
+
+def F(x, y):
+    return x**2 + y**2
+
+def test_interpolator():
+    x0 = [1.0, 2.0, 3.0]
+    x1 = [10.0, 20.0]
+    # Not using numpy on purpose
+    ys = [F(x,y) for (x, y) in product(x0, x1)]
+
+    axis_x0 = btwxt.GridAxis(
+        x0, # Values
+        btwxt.InterpolationMethod_cubic, # interpolation method
+        btwxt.ExtrapolationMethod_linear, # extrapolation_method
+        (0, 5), # extrapolation_limits
+        "x0", # name
+    )
+    axis_x1 = btwxt.GridAxis(
+        x1, # Values
+        btwxt.InterpolationMethod_cubic, # interpolation method
+        btwxt.ExtrapolationMethod_linear, # extrapolation_method
+        (0, 30), # extrapolation_limits
+        "x1", # name
+    )
+    grid_point_y = btwxt.GridPointDataSet(ys, "y")
+
+    interpolator = btwxt.RegularGridInterpolator([axis_x0, axis_x1], [grid_point_y])
+
+    target_normalization = [2, 10]
+    normalizationDivisor = 1.0
+    normalizationDivisor = interpolator.normalize_grid_point_data_set_at_target(
+        0, target_normalization, normalizationDivisor
+    ) * normalizationDivisor
+    assert normalizationDivisor == 104.0
+
+    interp_x0s = [float(x) for x in range(0, 6, 1)]
+    interp_oat_dbs = [float(x) for x in range(0, 31, 5)]
+
+    test_points = [(x,y) for (x, y) in product(interp_x0s, interp_oat_dbs)]
+    true_values = [F(x, y) for (x, y) in test_points]
+    interp_values = [
+        interpolator.get_value_at_target((x0, x1), 0) * normalizationDivisor
+        for (x0, x1) in test_points
+    ]


pytest file to test out the python bindings

jmarrec · 2025-01-27T10:11:17Z

.github/workflows/build-and-test.yml

+      - name: Set up Python 3.12
+        if: ${{ matrix.python_bindings }}
+        id: setup-python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+
+      - name: Build Python Wheel
+        if: ${{ matrix.python_bindings }}
+        working-directory: ./build/
+        run: |
+          begin_group() { echo -e "::group::\033[93m$1\033[0m"; }
+
+          begin_group "Install other python dependencies"
+          python -m pip install --upgrade pip
+          # Ubuntu 22.04 from apt will get swig that's too old to support attributes like [[ nodiscard ]], so get from PyPi
+          pip install setuptools wheel twine requests packaging pytest swig
+          echo -e "::endgroup::"
+
+          begin_group "Build target"
+          cmake -DBUILD_PYTHON_BINDINGS:BOOL=ON -DPython_REQUIRED_VERSION:STRING=${{ steps.setup-python.outputs.python-version }} -DPython_ROOT_DIR:PATH=$RUNNER_TOOL_CACHE/Python/${{ steps.setup-python.outputs.python-version }}/${{ matrix.arch }}/ .
+          cmake --build . --config ${{ matrix.config }}  --target python_package
+          echo -e "::endgroup::"
+
+      - name: Python CTest
+        if: ${{ matrix.python_bindings }}
+        working-directory: ./build/
+        run: |
+          ctest -C ${{ matrix.config }} -R python -VV
+
+      - name: Zip the wheel to maintain case sensitivy and file permissions
+        if: ${{ matrix.python_bindings }}
+        working-directory: ./build/python
+        shell: bash
+        run: |
+          TGZ_STEM="btwxt-python-${{ matrix.os }}-${{ matrix.os_ver }}-${{ matrix.arch }}"
+          echo "TGZ_STEM=$TGZ_STEM" >> $GITHUB_ENV
+          tar -cvzf "$TGZ_STEM.tar.gz" dist/
+
+      - name: Upload .whl to artifact
+        if: ${{ matrix.python_bindings }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ env.TGZ_STEM }}
+          path: "./build/python/${{ env.TGZ_STEM }}.tar.gz"


New workflow bits to build the python bindings after the regular build is done.

Runs the python ctest, and uploads the wheel as artifacts

jmarrec added 22 commits January 24, 2025 15:57

Add Python bindings via SWIG

0dcde7f

Add a setup.py and cmake commands to create a bdist_wheel

c6f0832

Add a notebook to demo the python bindings

fbab36b

Build wheel on github actions?

e7876bc

Bump runners

6541ef6

Do in two steps

d75b293

bump gcc for 24.04

70ece55

Install setuptools and co

1afbe1e

link to Python::MODULE on macos too?

c9c8742

tweak

9c71d19

Plop

d3b7ca6

tweak

9cbee2b

matrix.config isn't replaced with it's value, try to quote it

5ca75cf

Try installing swig latest from pip

04f18bc

Tweak workflow again

1c6eb0c

Set swig lib suffix to so on Apple (not dylib).

09134a9

Note: for MSVC, UseSWIG.cmake already sets it to .pyd

Set "-C Release" for windows testing (should use Ninja instead...)

56c694a

Maybe I do need a .pyd suffix on MSVC, it's building a .lib instead...

73e984d

Avoid having a Release/ sublevel

ff9ccac

(Tired of windows/MSVC...)

Typo in last commit

497b50d

Rename artifact for clarity

240d733

Remove duplicated find python

4aee6c4

jmarrec commented Jan 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python bindings to Btwxt via SWIG #44

Add Python bindings to Btwxt via SWIG #44

jmarrec commented Jan 27, 2025 •

edited

Loading

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

jmarrec Jan 27, 2025

		COMMAND ${CMAKE_COMMAND} -E echo "To upload the package, cd 'python/'"
		COMMAND ${CMAKE_COMMAND} -E echo "${Python_EXECUTABLE} -m twine upload --verbose --skip-existing --repository ${PYTHON_PIP_REPOSITORY} dist/*"

Add Python bindings to Btwxt via SWIG #44

Are you sure you want to change the base?

Add Python bindings to Btwxt via SWIG #44

Conversation

jmarrec commented Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmarrec commented Jan 27, 2025 •

edited

Loading