Simplify tests, shorten buildir name on windows

[mattip]

Taken from #135, without the pytest-xdist module name randomization

This should be smaller and easier to review:

• explicitly test c file is not regenerated rather than retesting, should speed up tests
• solve windows test crashes due to too-long filenames

[mattip]

I am not sure why CI is failing. When I run the tests locally they pass.

[arigo]

Force the tests to re-run (unsure how), and see if they still fail? If they do, force a re-run on the base (unsure how) and see if they still pass? If all that is true, then dig deeper.

[arigo]

The failure we get is genuine. I think you can get it too by running both test_verify.py and test_vgen.py in the same run. Then it tries to load a library made in the test_verify mode while assuming it was made in the test_vgen mode, or something.

[mattip]

When I run the tests locally the cffi.verifier.cleanup_tmpdir() call cleans out the old module. I am not sure why in this test only the cleanup does not work.

[arigo]

Maybe the CI runs pytest-xdist anyway? Or some other pytest version, or other plugins, etc.? Something that would make the order of execution different, e.g. test_vgen.setup_module could be called before the actual tests of test_verify run.

[mattip]

I am stumped. No pytest-xdist plugin is used:

platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0

and the tests are run in the correct order:

  ../../../project/cffi/testing/cffi0/test_verify.py ...................s. [ 42%]
  .....s.....s............................................................ [ 46%]
  .....................................................s...s.............s [ 50%]
  s...                                                                     [ 50%]
  ../../../project/cffi/testing/cffi0/test_version.py .....                [ 51%]
  ../../../project/cffi/testing/cffi0/test_vgen.py ...................s... [ 52%]
  ...s.....s.............................................................. [ 56%]
  ...................................................s...s.............ss. [ 60%]
  .F                                

When I can I will add debug cruft to cleanup_tempdir(), maybe somehow it is skipping the module in question.

[mattip]

Another strange thing I noticed in the codebase: there is both testing/cffi0/test_verify.py and testing/cffi1/test_verify1.py. These files echo each other: they have many of the same tests with slight variations. Some of these repeated tests are quite "expensive" on PyPy's nightly testing on windows. For instance:

• test_struct_bad_sized_integer takes 101 seconds
• test_global_const_int_size takes ~75 secs

[mattip]

Still stumped. I used cos to make sure the test is using a unique function call and print the remaining files in the cleanup_tmp() call at the beginning of test_vgen.py, there are no remnants that might trip up the call to verify(). It still seems like something is being cached from test_verify into test_vgen somewhere.

[mattip]

CI was passing so I removed the debug cruft and enabled windows tests. Let's see how long it takes. I am not sure why when using a module name I had to create a unique module name but 🤷. I guess #135 avoided this problem by making all the module names unique.

[mattip]

25 minutes for windows tests, I guess that is acceptable? It could be less if the repetitive tests are removed from cffi0/test_verify and cffi1/test_verify1.py

[arigo]

cffi0 checks the verify() call that was in cffi 0.x.y, while cffi1 checks the new compile() call. This is very different pieces of code, even if they both implement a large common part.

[mattip]

Ahh, makes sense, thanks for clarifying. CI is passing, including enabling windows tests.

[mattip]

Rebased on main and cleaned up history. CC @ngoldbaum. I would consider this ready to merge if CI passes, tests on windows do not take too long, and all the tests (except embedding on windows) run. We should maybe open an issue to revert disabling embedded tests on windows (commit 8be5d8a), although if I understand correctly they were not run in CI before this commit anyway.

I am not sure about the changes in the github CI yaml, each platform has a different CIBW_TEST_COMMAND call.
Linux uses

CIBW_TEST_COMMAND: PYTHONUNBUFFERED=1 python -m pytest ${{ matrix.test_args || '{project}' }}

macOS uses

CIBW_TEST_COMMAND: pip install pip --upgrade; cd {project}; PYTHONUNBUFFERED=1 pytest

and windows uses

CIBW_TEST_COMMAND: ${{ matrix.test_cmd || 'python -m pytest {package}/src/c' }}
which this PR changes to
CIBW_TEST_COMMAND: ${{ matrix.test_cmd || 'python -m pytest {package}' }}

[mattip]

CI test comparison:

platform	passed	skipped	xfailed	time
windows	1487	221	4	1332.06s (0:22:12)
macos	1584	135	4	391.34s (0:06:31)
linux	1622	97	4	206.63s (0:03:26)

So windows is still ~8x slower than linux while running fewer tests 🤷

[mattip]

This is the tricky part of this PR. As @arigo commented in #137

If you want to kill the test_v*2.py files and replace them with a single test that checks no C code is regenerated, then I'm fine with it.

so I did that.

[ngoldbaum]

I wonder if setting PYTHONBUFFERED=1 would make the windows tests even slower.

I personally find the linux version the most readable. The pip upgrade should happen elsewhere probably on Mac.

It looks like test_cmd is unused, so you can delete the matrix.test_cmd bit in the Windows invocation.

[mattip]

I wonder what the original intent of test_cmd was? @nitzmahone?