signer: run HSM tests as part of regular tests

Running HSM tests in a separate workflow on all Python version and
os combinations requires 15 additional runners. This is a bit of an
overkill.

This patch integrates the HSMTests with regular CI:

- Make hsm test module discoverable for aggregate_tests by
  changing the module name.
- Skip HSM tests if PYKCS11LIB is not set, so aggregate_tests can
  be executed without SoftHSM installed. (e.g. locally by devs)
- Remove dedicated HSM test env from tox.ini. Tests now run as part
  of default testenv.
- Require PYKCS11LIB to be set for that testenv.
- Remove dedicated HSM requirements files and add them to default
  requirement file, which already includes optional requirements.
- Re-compile pinned requirements file.

Signed-off-by: Lukas Puehringer <[email protected]>

Author: lukpueh

.github/workflows/ci.yml

@@ -52,5 +52,26 @@ jobs:
           python -m pip install --upgrade pip
           pip install --upgrade tox
 
+      - name:  Install system dependencies
+        shell: bash
+        run: |
+          if [ "$RUNNER_OS" == "Linux" ]; then
+            sudo apt-get install -y softhsm2
+            echo "PYKCS11LIB=/usr/lib/softhsm/libsofthsm2.so" >> $GITHUB_ENV
+
+          elif [ "$RUNNER_OS" == "macOS" ]; then
+            brew install softhsm
+            echo "PYKCS11LIB=$(brew --prefix softhsm)/lib/softhsm/libsofthsm2.so" >> $GITHUB_ENV
+
+          # TODO: Uncomment when testing on Windows
+          # elif [ "$RUNNER_OS" == "Windows" ]; then
+          #   choco install softhsm.install
+          #   echo "PYKCS11LIB=C:\SoftHSM2\lib\softhsm2-x64.dll" >> $GITHUB_ENV
+
+          else
+              echo "$RUNNER_OS not supported"
+              exit 1
+          fi
+
       - name: Run tox
         run: tox -e ${{ matrix.toxenv }}

.github/workflows/hsm.yml

@@ -1,6 +1,23 @@
-cffi==1.15.1              # via cryptography, pynacl
-cryptography==38.0.3
-pycparser==2.21           # via cffi
+#
+# This file is autogenerated by pip-compile with python 3.8
+# To update, run:
+#
+#    pip-compile --output-file=requirements-pinned.txt requirements.txt
+#
+asn1crypto==1.5.1
+    # via -r requirements.txt
+cffi==1.15.1
+    # via
+    #   cryptography
+    #   pynacl
+    #   pyspx
+cryptography==38.0.3 ; python_version >= "3"
+    # via -r requirements.txt
+pycparser==2.21
+    # via cffi
+pykcs11==1.5.11
+    # via -r requirements.txt
 pynacl==1.5.0
-six==1.16.0               # via pynacl
-PySPX==0.5.0
+    # via -r requirements.txt
+pyspx==0.5.0
+    # via -r requirements.txt

requirements-hsm-pinned.txt

@@ -34,3 +34,5 @@
 cryptography >= 37.0.0; python_version >= '3'
 pynacl
 PySPX
+PyKCS11
+asn1crypto

requirements-hsm.txt

@@ -26,6 +26,9 @@
 from securesystemslib.signer import HSMSigner, SSlibKey
 
 
+@unittest.skipUnless(
+    os.environ["PYKCS11LIB"], "set PYKCS11LIB to SoftHSM lib path"
+)
 class TestHSM(unittest.TestCase):
     """Test HSMSigner with SoftHSM
 

requirements-pinned.txt

@@ -11,12 +11,16 @@ skipsdist = True
 install_command =
     pip install {opts} {packages}
 
+passenv =
+    PYKCS11LIB
+
 deps =
     -r{toxinidir}/requirements-pinned.txt
     -r{toxinidir}/requirements-test.txt
 
 commands =
     python -m tests.check_gpg_available
+    python -c '"{env:PYKCS11LIB}"' # Required for 'test_hsm_signer'
     coverage run tests/aggregate_tests.py
     coverage report -m --fail-under 95
 
@@ -42,14 +46,6 @@ passenv =
 commands =
     python -m tests.check_kms_signers
 
-[testenv:hsm]
-deps =
-    -r{toxinidir}/requirements-hsm-pinned.txt
-passenv =
-    PYKCS11LIB
-commands =
-    python -m tests.check_hsm_signer
-
 # This checks that importing securesystemslib.gpg.constants doesn't shell out on
 # import.
 [testenv:py311-test-gpg-fails]