Fix pip installation: Migrate from CFFI/f2py to ctypes (SpacePy-style) by sapols · Pull Request #155 · nasa/Kamodo

sapols · 2026-02-20T07:09:36Z

Summary

This PR fixes Kamodo's broken pip installation by replacing CFFI and f2py with direct ctypes loading of shared libraries—the same approach SpacePy uses successfully.

The Problem: pip install kamodo-ccmc from PyPI is broken:

PyPI version (v23.3.1) is nearly 2 years out of date
C/Fortran extensions don't compile correctly
SWMF-GM and GAMERA-GM readers fail to load
Users must use editable installs with manual compiler setup

The Solution: Compile C/Fortran code to standalone shared libraries (.so/.dll/.dylib) at build time, then load them via Python's built-in ctypes at runtime. No CFFI, no f2py, no numpy build dependency.

Status: ✅ CI passing on Ubuntu, macOS, and Windows for Python 3.10 & 3.11 | ⚠️ Python 3.12 & 3.13 fail (Kamodo compatibility — separate issue)

What Changed

Build System (`setup.py`, `pyproject.toml`)

Before	After
CFFI for C extensions	`distutils.ccompiler` abstraction (respects CC env var)
f2py for Fortran extensions	Direct `gfortran` compilation to shared libraries (`.so`/`.dll`/`.dylib`)
Requires `numpy` at build time	No numpy build dependency
4 wheels per platform (cp310-cp313)	1 abi3 wheel per platform (`cp310-abi3-*`, works for Python 3.10+)
Complex build with 2 systems	Single unified build system

New setup.py features (SpacePy-style):

Custom build_ext command that compiles C/Fortran directly
C code compiled via distutils.ccompiler (respects CC env var and platform defaults; no hardcoded gcc)
Uses os.path.abspath(self.build_lib) for correct output paths during compilation
Bundles Fortran runtime libraries (libgfortran, libquadmath) into wheels for portability
macOS rpath set at link time (-Wl,-rpath,@loader_path/../libs); rewrites hardcoded libgcc_s paths in bundled libs
Wheels configured for abi3 (py_limited_api='cp310'); CI verifies produced wheel tags
KAMODO_RELEASE=1 env var for strict mode (fails on any compilation error)
KAMODO_SKIP_NATIVE=1 env var to skip native compilation (for debugging)

New C Wrapper Files

The CFFI build scripts had inline C code for wrapper functions. These have been extracted to standalone files:

kamodo_ccmc/readers/OCTREE_BLOCK_GRID/ctypes_wrappers.c - Contains xyz_ranges() and interpolate_amrdata_multipos()
kamodo_ccmc/readers/Tri2D/ctypes_wrappers.c - Contains interpolate_tri2d_plus_1d_multipos()

C Extension Loaders (OCTREE_BLOCK_GRID, Tri2D)

Before: CFFI with ffi.dlopen() and ffi.new() for memory allocation

After: Pure ctypes with:

Type aliases for clarity (SpacePy pattern): c_int_p, c_float_p
Multi-platform library search (handles .so/.dylib on macOS and MinGW-style .dll.a naming on Windows)
Function signature setup at load time

Fortran Extension Loader (OpenGGCM)

Before: f2py with automatic array handling and Fortran module imports

After: ctypes with SpacePy-style patterns:

Fortran type aliases: int4, real4 (matches Fortran real = 4 bytes)
Dictionary-based signature storage (FORTRAN_FUNCTIONS)
Single name resolution at setup (tries func_, func__, func)
Handles gfortran's hidden string length arguments

Reader Modules (swmfgm_4D, gameragm_4D, swmf_gm_octree)

Updated to use ctypes instead of CFFI:

ffi.new('float[]', size) → numpy.zeros(size, dtype=np.float32)
ffi.cast('float *', arr) → arr.ctypes.data_as(c_float_p)
ffi.addressof() → ctypes.byref()

CI/CD (`.github/workflows/`)

ci.yml - Tests on every push:

Runs on Ubuntu, macOS, Windows × Python 3.10-3.13
Installs gfortran on each platform (with symlink fix for macOS)
Verifies all three extensions load successfully
Inspects Mach-O binaries with otool on macOS to verify rpath
Currently passing for Python 3.10 & 3.11; 3.12 & 3.13 fail (Kamodo compatibility — separate PR)

build-wheels.yml - Builds wheels on version tags:

Uses cibuildwheel for consistent cross-platform builds
Builds for Linux (manylinux2014), macOS (Intel + ARM), Windows
Verifies produced wheels have abi3 tags (fails build if mismatch)
Tests wheel import after build
Publishes to PyPI with OIDC trusted publishing

Comparison with SpacePy

Aspect	SpacePy	Kamodo (After)
C library loading	`ctypes.CDLL()`	`ctypes.CDLL()` ✓
C compilation	`distutils.ccompiler`	`distutils.ccompiler` ✓
Build output paths	`os.path.abspath(self.build_lib)`	`os.path.abspath(self.build_lib)` ✓
Type aliases	`dptr`, `lptr` (C); `int4`, `real8` (Fortran)	`c_int_p`, `c_float_p` (C); `int4`, `real4` (Fortran) ✓
Signature storage	Dictionary	Dictionary ✓
Fortran name mangling	Try `func_` then plain	Try `func_`, `func__`, plain ✓
Runtime lib bundling	macOS/Windows	macOS/Windows ✓
sysconfig fallback	`EXT_SUFFIX` → `SO`	`EXT_SUFFIX` → `SO` ✓
Wheel building	cibuildwheel	cibuildwheel ✓

Key difference: SpacePy's Fortran functions have no string arguments, so all args can be auto-wrapped in POINTER(). Kamodo's OpenGGCM has Fortran strings with hidden length arguments (gfortran ABI), so pointer types are explicit in the signature dict.

Notable Behavioral Fix

This PR also includes a genuine bug fix in kamodo_ccmc/readers/OpenGGCM/openggcm_gm_tocdf.py: removal of a spurious extra 10 argument in read_grid_for_vector calls (for the by/bz/ey/ez grid paths). This corrected the function signature to match the Fortran subroutine interface.

Files Changed

File	Change
`setup.py`	Complete rewrite: custom `build_ext` (`distutils.ccompiler` for C, subprocess `gfortran`/`FC` for Fortran)
`pyproject.toml`	Simplified: only needs setuptools/wheel, no cffi/numpy
`setup.cfg`	Removed cffi from install_requires, added kamodo_ccmc.libs package
`MANIFEST.in`	Updated source/wheel inclusion patterns for native-source packaging
`README.md`	Updated installation/build notes for native extension packaging flow
`QuickStart.md`	Updated install guidance to match current pip/native build behavior
`kamodo_ccmc/__init__.py`	Added Windows runtime DLL discovery (`PATH` and `os.add_dll_directory`)
`kamodo_ccmc/libs/__init__.py`	Added package marker for bundled runtime libraries
`kamodo_ccmc/readers/OCTREE_BLOCK_GRID/__init__.py`	New ctypes loader with type aliases
`kamodo_ccmc/readers/OCTREE_BLOCK_GRID/ctypes_wrappers.c`	New file: C wrapper functions extracted from CFFI build
`kamodo_ccmc/readers/Tri2D/__init__.py`	New ctypes loader with type aliases
`kamodo_ccmc/readers/Tri2D/ctypes_wrappers.c`	New file: C wrapper functions extracted from CFFI build
`kamodo_ccmc/readers/OpenGGCM/__init__.py`	New file: ctypes loader with SpacePy patterns
`kamodo_ccmc/readers/swmfgm_4D.py`	Migrated from CFFI to ctypes
`kamodo_ccmc/readers/gameragm_4D.py`	Migrated from CFFI to ctypes
`kamodo_ccmc/readers/swmf_gm_octree.py`	Migrated from CFFI to ctypes
`kamodo_ccmc/readers/OpenGGCM/openggcm_gm_tocdf.py`	Updated imports for ctypes loader
`.github/workflows/build-wheels.yml`	New: cross-platform wheel building
`.github/workflows/ci.yml`	New: CI testing with gfortran setup
`.gitignore`	Updated: added `.dylib`, `.dll`, `kamodo_ccmc/libs/*` patterns
`docs/PyPI-Release.md`	Updated: reflects ctypes/shared-library packaging

Deleted:

kamodo_ccmc/readers/OCTREE_BLOCK_GRID/interpolate_amrdata_extension_build.py (old CFFI)
kamodo_ccmc/readers/Tri2D/interpolate_tri2d_extension_build.py (old CFFI)

How to Test

Local Build Test

# Clone and checkout
git clone https://github.com/sapols/Kamodo.git
cd Kamodo
git checkout fix-pip-installation-combined-v2

# Clean build
rm -rf build/ dist/ *.egg-info

# Install (requires gcc and gfortran)
pip install .

# Test imports
python -c "
from kamodo_ccmc.readers.OCTREE_BLOCK_GRID import lib, OCTREE_AVAILABLE
print(f'OCTREE: {OCTREE_AVAILABLE}')

from kamodo_ccmc.readers.Tri2D import lib, TRI2D_AVAILABLE
print(f'Tri2D: {TRI2D_AVAILABLE}')

from kamodo_ccmc.readers.OpenGGCM import lib, OPENGGCM_AVAILABLE
print(f'OpenGGCM: {OPENGGCM_AVAILABLE}')

from kamodo_ccmc.readers import swmfgm_4D, gameragm_4D
print('Readers imported successfully!')
"

Expected Outcome

After merging and releasing:

# This should "just work" on any machine with Python 3.10+
pip install kamodo-ccmc

# All readers should be available
python -c "from kamodo_ccmc.readers import swmfgm_4D, gameragm_4D"

No compiler installation required for end users. The wheel contains pre-compiled shared libraries that work on:

Linux (manylinux2014, x86_64)
macOS (Intel and Apple Silicon)
Windows (x86_64)

References

SpacePy ctypes migration PR #749
SpacePy setup.py - Build system pattern
SpacePy irbempy - Fortran library loading pattern
cibuildwheel documentation

This commit fixes the long-standing issue where kamodo-ccmc requires manual compilation of C and Fortran extensions before installation. Following SpacePy's proven approach, the build system now automatically compiles extensions during 'pip install'. Changes: - Created custom build_ext class in kamodo_ccmc/build_tools.py - Refactored CFFI builder scripts to be importable as modules - Updated setup.py to use custom build commands - Updated setup.cfg (added cffi, fixed setuptools constraint) - Added graceful degradation to model readers - Updated README.md and QuickStart.md documentation Outcome: Users can now simply 'pip install kamodo-ccmc' and extensions will be automatically compiled if compilers are available. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

Review fixes for the initial pip installation automation: 1. build_tools.py: Rewrote to override install/develop commands - build_ext alone won't trigger without ext_modules defined - Now extensions compile after package installation completes - Cleaner architecture with compile_extensions() helper function 2. setup.py: Fixed import issue for fresh installs - Added sys.path manipulation to import build_tools before install - Properly registers all three command classes - Graceful fallback if build_tools import fails 3. Model readers: Added __init__ checks for clear error messages - MODEL classes now raise ImportError with actionable instructions - openggcm_gm_tocdf.py: Function-level check added Co-Authored-By: Claude Opus 4.5 <[email protected]>

…ilation Rewrite the build system to use cffi_modules in setup.py instead of custom install/develop commands. This is the proper way to integrate CFFI with setuptools and ensures extensions compile during wheel building. Changes: - setup.py: Use cffi_modules parameter to register CFFI builders - pyproject.toml: Add cffi to build-requires, specify build-backend - interpolate_amrdata_extension_build.py: - Add _rel_path() helper for setuptools-compatible relative paths - Use full module path: kamodo_ccmc.readers.OCTREE_BLOCK_GRID._interpolate_amrdata - Add include_dirs for header file resolution - interpolate_tri2d_extension_build.py: - Same changes as above for Tri2D module Verified working: - pip install . now compiles both CFFI extensions automatically - Extensions install to correct locations in site-packages - Extensions load and work correctly (tested direct import) Note: Full integration testing blocked by upstream kamodo package (v23.3.0) using deprecated numpy.distutils, which fails on Python 3.12. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Complete the pip install fix by adding OpenGGCM Fortran compilation: - setup.py: Add custom build_ext class that compiles Fortran via f2py - Follows SpacePy's pattern: register Extension to trigger build_ext - Runs 'python -m numpy.f2py' via subprocess - Handles build_lib output directory for wheel builds - Graceful degradation if gfortran not available - KAMODO_RELEASE env var for strict release builds - pyproject.toml: Add numpy to build-requires for f2py All three extension types now compile automatically during pip install: 1. OCTREE_BLOCK_GRID (CFFI) - for SWMF-GM reader 2. Tri2D (CFFI) - for GAMER-AM reader 3. readOpenGGCM (f2py) - for OpenGGCM reader Verified working in Python 3.11: - All extensions compile during wheel build - All model readers import and work correctly Co-Authored-By: Claude Opus 4.5 <[email protected]>

Add automated CI/CD pipelines for Kamodo: 1. .github/workflows/ci.yml - CI for PRs and pushes - Tests on Ubuntu and macOS - Python 3.10, 3.11, 3.12 - Verifies all extensions compile and import 2. .github/workflows/build-wheels.yml - Wheel building for releases - Builds wheels for Linux (manylinux2014), macOS (Intel + ARM), Windows - Python 3.10, 3.11, 3.12, 3.13 - Uses cibuildwheel for cross-platform builds - Publishes to PyPI on version tags (requires PyPI trusted publishing setup) - Manual trigger option for testing 3. pyproject.toml - cibuildwheel configuration - Platform-specific settings for Linux, macOS, Windows - Fortran compiler setup for each platform - Wheel repair tools (auditwheel for Linux, delvewheel for Windows) To use: - CI runs automatically on PRs - Wheel builds trigger on 'v*' tags or manual dispatch - PyPI publishing requires setting up trusted publishing on PyPI Co-Authored-By: Claude Opus 4.5 <[email protected]>

This commit combines the best approaches from both AI solutions: From Claude's solution: - Native cffi_modules integration (cleaner, officially recommended) - Comprehensive graceful degradation with try/except and clear error messages - Complete CI/CD workflows (Linux, macOS Intel+ARM, Windows) - cibuildwheel configuration in pyproject.toml From Codex's solution: - Runtime library bundling for portable wheels (SpacePy-style) - KAMODO_SKIP_NATIVE environment variable to skip extensions - Python 3.12+ distutils compatibility workaround - kamodo_ccmc/libs/ package with Windows DLL handling - docs/PyPI-Release.md documentation Changes: - setup.py: Added runtime lib bundling, KAMODO_SKIP_NATIVE, distutils fix - kamodo_ccmc/__init__.py: Added Windows DLL path handling - kamodo_ccmc/libs/__init__.py: New package for bundled runtime libs - .github/workflows/ci.yml: Added Python 3.13 and Windows coverage - pyproject.toml: Fixed TOML syntax error - MANIFEST.in: Added Fortran files, removed build artifacts - docs/PyPI-Release.md: New release checklist documentation Co-Authored-By: Claude Opus 4.5 <[email protected]>

Ensures Tri2D extension and bundled runtime libs are properly included in wheels by design, not by accident. Co-Authored-By: Claude Opus 4.5 <[email protected]>

This commit adds key improvements from the Codex solution: 1. Add _clean_source_artifacts() to setup.py - Removes intermediate .o and .so files after build - Prevents stale artifacts from contaminating wheels 2. Create package marker __init__.py files - kamodo_ccmc/readers/OCTREE_BLOCK_GRID/__init__.py - kamodo_ccmc/readers/Tri2D/__init__.py - Ensures setuptools properly discovers extension packages 3. Fix import paths to use relative imports - swmfgm_4D.py: from .OCTREE_BLOCK_GRID._interpolate_amrdata - gameragm_4D.py: from .Tri2D._interpolate_tri2d - swmf_gm_octree.py: from .OCTREE_BLOCK_GRID._interpolate_amrdata 4. Retain graceful degradation pattern - Readers warn but don't crash when extensions unavailable - Clear error messages guide users to install compilers Combined with Claude's CI/CD infrastructure, this creates the optimal pip installation solution with comprehensive platform support. Co-Authored-By: Claude Opus 4.5 <[email protected]>

- OpenGGCM: Add FORTRAN_FUNCTIONS dict, type aliases (int4, real4), single name resolution, fail-fast on missing functions, ABI docs - OCTREE_BLOCK_GRID: Add c_int_p/c_float_p type aliases - Tri2D: Add c_int_p/c_float_p type aliases - Remove old CFFI build scripts Co-Authored-By: Claude Opus 4.5 <[email protected]>

- CI: Change to non-editable install (pip install . not -e) Fixes OpenGGCM not compiling due to PEP 660 changes in setuptools 64+ - CI: Make test fail if any extension fails to compile Previously showed "All extensions loaded successfully!" even when OpenGGCM was unavailable - now properly validates each extension - pyproject.toml: Add zip-safe = false for setuptools compatibility - README: Document proper editable install workflow for developers (build_ext --inplace before pip install -e) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

The zip-safe setting was causing build failures because setuptools expected a full [project] section when tool.setuptools was present. Kamodo uses setup.cfg for metadata, not pyproject.toml. The zip-safe setting wasn't needed anyway - the real PEP 660 fixes are: (1) CI uses non-editable install, (2) README documents manual build step for developer editable installs. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

Without ext_modules declared, setuptools skips build_ext entirely, even with a custom build_ext command registered. Adding dummy Extension objects with empty source lists makes setuptools call our custom build_ext.run() method, which handles the actual C/Fortran compilation. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

The original CFFI build scripts embedded C code inline that provided: - xyz_ranges() and interpolate_amrdata_multipos() for OCTREE - interpolate_tri2d_plus_1d_multipos() for Tri2D These functions were not in the tracked C files, causing undefined symbol errors when linking the shared libraries. This commit: - Extracts CFFI inline code to ctypes_wrappers.c for both modules - Adds globals.c to define extern variables (previously in CFFI wrapper) - Updates setup.py to compile the new files - Adds -Dno_idl flag to OCTREE compilation for ctypes-compatible signatures Co-Authored-By: Claude Opus 4.5 <[email protected]>

When running pip install . followed by python -c import kamodo_ccmc, Python was finding the local source kamodo_ccmc/ directory first instead of the installed package in site-packages. The source directory doesn't have the compiled extensions, causing the import tests to fail. Fix by running the test from a different directory (runner.temp) so Python imports from site-packages where the extensions are installed. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Co-Authored-By: Claude Opus 4.5 <[email protected]>

The existing C files already define the global variables. The globals.c files were duplicating these definitions, causing linker errors. Co-Authored-By: Claude Opus 4.5 <[email protected]>

shutil.move() was failing because outdir was a relative path like 'build/lib.linux-x86_64-cpython-311/...' but we had already chdir'd to the source directory for compilation. The relative path then resolved incorrectly, causing 'No such file or directory' errors. Fix: Use os.path.abspath() to convert outdir to an absolute path before changing directories. Co-Authored-By: Claude Opus 4.5 <[email protected]>

1. Add libinterpolate_*.so to macOS search list in loaders - distutils creates .so files on macOS (not .dylib) when using gcc/clang - The loaders were looking for .dylib but the compiled file was .so 2. Create gfortran symlink on macOS CI - brew install gcc installs gfortran as gfortran-XX (e.g., gfortran-14) - Create symlink so 'gfortran' command works Co-Authored-By: Claude Opus 4.5 <[email protected]>

sapols · 2026-02-20T07:13:02Z

@darrendezeeuw this is identical to the PR in my fork I already showed you, just opening here for better visibility after I cleaned up the cruft. No rush nor pressure to accept, just for your consideration!

sapols and others added 24 commits January 13, 2026 14:25

Merge branch 'nasa:master' into fix-pip-installation

183adf9

Remove unused build_tools.py (replaced by cffi_modules)

2f1945c

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Add Tri2D and libs to explicit package list in setup.cfg

b2396bf

Ensures Tri2D extension and bundled runtime libs are properly included in wheels by design, not by accident. Co-Authored-By: Claude Opus 4.5 <[email protected]>

Add verbose install and debug step to diagnose missing libraries

135c574

Co-Authored-By: Claude Opus 4.5 <[email protected]>

fix: Remove globals.c files causing multiple definition linker errors

b3de32c

The existing C files already define the global variables. The globals.c files were duplicating these definitions, causing linker errors. Co-Authored-By: Claude Opus 4.5 <[email protected]>

build: harden native packaging and abi3 wheel checks

244cd32

docs: align release guide and ignore native build artifacts

e2a4cd1

chore: clean native reader migration comments

9493dee

fix: load mingw .dll.a native libs on Windows

3a30cc9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix pip installation: Migrate from CFFI/f2py to ctypes (SpacePy-style)#155

Fix pip installation: Migrate from CFFI/f2py to ctypes (SpacePy-style)#155
sapols wants to merge 24 commits intonasa:masterfrom
sapols:fix-pip-installation-combined-v2

sapols commented Feb 20, 2026

Uh oh!

sapols commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

sapols commented Feb 20, 2026

Summary

What Changed

Build System (setup.py, pyproject.toml)

New C Wrapper Files

C Extension Loaders (OCTREE_BLOCK_GRID, Tri2D)

Fortran Extension Loader (OpenGGCM)

Reader Modules (swmfgm_4D, gameragm_4D, swmf_gm_octree)

CI/CD (.github/workflows/)

Comparison with SpacePy

Notable Behavioral Fix

Files Changed

How to Test

Local Build Test

Expected Outcome

References

Uh oh!

sapols commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Build System (`setup.py`, `pyproject.toml`)

CI/CD (`.github/workflows/`)