CLI options for controlling Cython extension compliation #567

sserita · 2025-04-01T23:07:45Z

This should fix both #566 and #559, giving the user better control over when Cython extensions are installed (and therefore us better control in our GitHub runners).

First, a clarification that took me a while to understand but is critical to understanding the issue in #559: whether or not pyGSTi's Cython extensions get compiled have nothing to do with which optional dependencies get installed, and everything to do with what is present in pip's isolated build system at install time. This, in turn, uses the build-system.requires section of pyproject.toml. Since Cython and numpy are included here, this means that pyGSTi will attempt to build Cython extensions even if the user is not specifying pygsti[extensions] (or a superset, like [testing] or [complete]).

It is not possible to make the build-system.requires field dynamic, so we instead always install Cython and provide a way for the user to turn this off via environment variables.

Solution

During installation, pyGSTi will now check for two environment variables: PYGSTI_CYTHON_SKIP and PYGSTI_CYTHON_EXCLUDE_FAILURES. If the first is defined, then extension compilation will be skipped entirely. If the second is defined, then extension compilation will be performed with exclude_failures=True defined - this used to be the default but can suppress errors in the case where one of the extensions fails but the others succeed. (This is uncommon and is most likely to be useful to developers who are writing Cython extensions, but I opted for the more verbose failure behavior by default).

Aside: Attempted PEP-517 Solution

In theory, it should be possible to do a pip-flag-based option via --config-settings. See an example package listed here using this for optional Cythonization.

However, I could not figure out how to get this working and gave up after several hours of banging my head against it. The CLI flag seems to be working fine.

New Behavior Examples

There is still a try-except block around the compilation. However, this now re-raises the error and I hope will prevent suppression of any error messages. We do still need it so that we can provide the user with the error message telling them about the new flags.

I'll use CC=broken to emulate a missing compiler that would trigger Cython compilation errors for testing. We can see that now we have our error message at the bottom.

$ CC=broken pip install -e .
[... MUCH ERROR MESSAGES ...]
          self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
        File "/private/var/folders/xb/r8b8yj9d1kjgw1fr5x8bkls0003q1t/T/pip-build-env-5di3gmpi/overlay/lib/python3.11/site-packages/setuptools/_distutils/compilers/C/unix.py", line 223, in _compile
          raise CompileError(msg)
      distutils.compilers.C.errors.CompileError: command 'broken' failed: No such file or directory
      /private/var/folders/xb/r8b8yj9d1kjgw1fr5x8bkls0003q1t/T/pip-build-env-5di3gmpi/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:1021: _DebuggingTips: Problem in editable installation.
      !!
      
              ********************************************************************************
              An error happened while installing `pygsti` in editable mode.
      
              The following steps are recommended to help debug this problem:
      
              - Try to install the project normally, without using the editable mode.
                Does the error still persist?
                (If it does, try fixing the problem before attempting the editable mode).
              - If you are using binary extensions, make sure you have all OS-level
                dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
              - Try the latest version of setuptools (maybe the error was already fixed).
              - If you (or your project dependencies) are using any setuptools extension
                or customization, make sure they support the editable mode.
      
              After following the steps above, if the problem still persists and
              you think this is related to how setuptools handles editable installations,
              please submit a reproducible example
              (see https://stackoverflow.com/help/minimal-reproducible-example) to:
      
                  https://github.com/pypa/setuptools/issues
      
              See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
              ********************************************************************************
      
      !!
        cmd_obj.run()
      
      An error occurred while compiling Cython extensions.
      Either fix the compilation issue or use the PYGSTI_CYTHON_SKIP to skip compilation,
      e.g. PYGSTI_CYTHON_SKIP=1 pip install pygsti
      To enable partial Cython failures (i.e. the exclude_failures=True flag of cythonize), use PYGSTI_CYTHON_EXCLUDE_FAILURES instead.
      
      error: command 'broken' failed: No such file or directory
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for pygsti
Failed to build pygsti
ERROR: Failed to build installable wheels for some pyproject.toml based projects (pygsti)

Note that there is the full stack trace, an error message telling the user what environment variables should be used to avoid this issue, and then the actual relevant line of the error message.

Now with our newly introduced flag:

$ CC=broken PYGSTI_SKIP_CYTHON=1 pip install -e .
[...]
Successfully built pygsti
Installing collected packages: pygsti
  Attempting uninstall: pygsti
    Found existing installation: pygsti 0.9.13.1.post1.dev1+g86142fd1a.feature.extension.option.d20250401
    Uninstalling pygsti-0.9.13.1.post1.dev1+g86142fd1a.feature.extension.option.d20250401:
      Successfully uninstalled pygsti-0.9.13.1.post1.dev1+g86142fd1a.feature.extension.option.d20250401
Successfully installed pygsti-0.9.13.1.post1.dev3+g7c8b1afb5.feature.extension.option.clean

No error! And we can check none of the stuff got compiled, and that it does if we don't include the variable.

$ PYGSTI_SKIP_CYTHON=1 pip install -e .
[...]
Successfully installed pygsti-0.9.13.1.post1.dev3+g7c8b1afb5.feature.extension.option.clean
$ find . -name '*.so'
$ pip install -e .
[... same output as above since extension compilation is silent if no errors ...]
$ find . -name '*.so'
./pygsti/tools/fastcalc.cpython-311-darwin.so
./pygsti/forwardsims/termforwardsim_calc_stabilizer.cpython-311-darwin.so
./pygsti/forwardsims/mapforwardsim_calc_densitymx.cpython-311-darwin.so
./pygsti/forwardsims/termforwardsim_calc_statevec.cpython-311-darwin.so
./pygsti/circuits/circuitparser/fastcircuitparser.cpython-311-darwin.so
./pygsti/baseobjs/opcalc/fastopcalc.cpython-311-darwin.so
./pygsti/evotypes/statevec/statereps.cpython-311-darwin.so
./pygsti/evotypes/statevec/termreps.cpython-311-darwin.so
./pygsti/evotypes/statevec/effectreps.cpython-311-darwin.so
./pygsti/evotypes/statevec/opreps.cpython-311-darwin.so
./pygsti/evotypes/stabilizer/statereps.cpython-311-darwin.so
./pygsti/evotypes/stabilizer/termreps.cpython-311-darwin.so
./pygsti/evotypes/stabilizer/effectreps.cpython-311-darwin.so
./pygsti/evotypes/stabilizer/opreps.cpython-311-darwin.so
./pygsti/evotypes/densitymx/statereps.cpython-311-darwin.so
./pygsti/evotypes/densitymx/effectreps.cpython-311-darwin.so
./pygsti/evotypes/densitymx/opreps.cpython-311-darwin.so
./pygsti/evotypes/basereps_cython.cpython-311-darwin.so

PYGSTI_CYTHON_SKIP being defined will fully skip extension compilation. PYGSTI_CYTHON_EXCLUDE_FAILURES will run cythonize with exclude_failures=True. Also removes the unneeded setup.cfg, and applies PYGSTI_CYTHON_SKIP to the "No Cython" GitHub Actions.

sserita · 2025-04-01T23:08:09Z

.github/workflows/reuseable-main.yml

@@ -69,7 +69,7 @@ jobs:
    - name: Install package (No Cython)
      if: ${{ inputs.use-cython != 'true' }}
      run: |
-        python -m pip install -e .[testing_no_cython]
+        PYGSTI_CYTHON_SKIP=1 python -m pip install -e .[testing_no_cython]


This should fix #559

sserita · 2025-04-01T23:08:43Z

pygsti/layouts/prefixtable.py

@@ -1208,6 +1207,8 @@ def _draw_graph(G, node_label_key='label', edge_label_key='promotion_cost', figu
        An optional size specifier passed into the matplotlib figure
        constructor to set the plot size.
    """
+    import matplotlib.pyplot as plt


Moved import that broke minimal install. Matplotlib is an optional dependency currently.

sserita · 2025-04-01T23:09:01Z

pyproject.toml

    "Programming Language :: Python",
    "Topic :: Scientific/Engineering :: Physics",
    "Operating System :: Microsoft :: Windows",
    "Operating System :: MacOS :: MacOS X",
    "Operating System :: Unix"
 ]
+license = "Apache-2.0"


Previous classifier was deprecated in favor of this.

sserita · 2025-04-01T23:10:23Z

setup.cfg

This file was unneeded - the [metadata] section is redundant with pyproject.toml, [bdist_wheel] for universal is actually deprecated with Python 2.7 EOL and would cause us issues in Aug 2025, and [nosetests] is out-of-date since we migrated to pytest.

sserita · 2025-04-01T23:10:48Z

setup.py

@@ -55,12 +54,219 @@ def build_extensions(self):
                ext.extra_compile_args = args
        build_ext.build_extensions(self)

+# Check if environment can try to build extensions


Most of this is just moving stuff around, I'll note the critical changes.

sserita · 2025-04-01T23:11:21Z

setup.py

+                extra_link_args=["-std=c++11"]
+            )
+        ]
+        extensions = cythonize(ext_modules, compiler_directives={'language_level': "3"}, exclude_failures="PYGSTI_CYTHON_EXCLUDE_FAILURES" in os.environ)


Here we went from exclude_failures=True to a check against PYGSTI_CYTHON_EXCLUDE_FAILURES

sserita · 2025-04-01T23:12:27Z

setup.py

+except ImportError:
+    warn("Extensions build tools are not available. Installing without Cython extensions...")
+    extensions = None


We still don't build extensions if no Cython. However, as mentioned in my main message, this should basically never trigger because it has to do with the build-system requirements, not the user-specified optional dependencies.

sserita · 2025-04-01T23:12:58Z

setup.py

-    # Extension compilation failed
-    warn("Error in extension compilation. Installing without Cython extensions...")
-    setup_with_extensions()
+except SystemExit as e:


We still have this try-except block so that we can tell the user about the env variables. However, we now re-raise the exception.

coreyostrove

Awesome work figuring this all out. I don't have any questions or concerns about any of these changes. Thanks, @sserita!

sserita added 3 commits April 1, 2025 15:34

Guard a minimal install matplotlib import

bc180a8

Minor typo.

f20b6d2

sserita commented Apr 1, 2025

View reviewed changes

sserita marked this pull request as ready for review April 1, 2025 23:13

sserita requested review from a team as code owners April 1, 2025 23:13

sserita requested a review from coreyostrove April 1, 2025 23:13

sserita added this to the 0.9.14 milestone Apr 1, 2025

coreyostrove approved these changes Apr 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLI options for controlling Cython extension compliation #567

CLI options for controlling Cython extension compliation #567

sserita commented Apr 1, 2025

sserita Apr 1, 2025

sserita Apr 1, 2025

sserita Apr 1, 2025

sserita Apr 1, 2025

sserita Apr 1, 2025

sserita Apr 1, 2025

sserita Apr 1, 2025

sserita Apr 1, 2025

coreyostrove left a comment

CLI options for controlling Cython extension compliation #567

Are you sure you want to change the base?

CLI options for controlling Cython extension compliation #567

Conversation

sserita commented Apr 1, 2025

Solution

Aside: Attempted PEP-517 Solution

New Behavior Examples

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coreyostrove left a comment

Choose a reason for hiding this comment