Dev/pytest plugin

.github/workflows/twister_tests.yml

@@ -49,10 +49,18 @@ jobs:
     - name: install-packages
       run: |
         pip3 install -r scripts/requirements-base.txt -r scripts/requirements-build-test.txt
-    - name: Run pytest
+    - name: Run pytest for twisterlib
       env:
         ZEPHYR_BASE: ./
         ZEPHYR_TOOLCHAIN_VARIANT: zephyr
       run: |
         echo "Run twister tests"
         PYTHONPATH=./scripts/tests pytest ./scripts/tests/twister
+    - name: Run pytest for pytest-twister-harness
+      env:
+        ZEPHYR_BASE: ./
+        ZEPHYR_TOOLCHAIN_VARIANT: zephyr
+        PYTHONPATH: ./scripts/pylib/pytest-twister-harness/src:${PYTHONPATH}
+      run: |
+        echo "Run twister tests"
+        pytest ./scripts/pylib/pytest-twister-harness/tests

doc/develop/test/index.rst

@@ -8,6 +8,7 @@ Testing
 
    ztest
    twister
+   pytest
    coverage
    BabbleSim<bsim>
    ztest_deprecated

doc/develop/test/pytest.rst

@@ -0,0 +1,84 @@
+.. integration-with-pytest:
+
+Integration with pytest test framework
+######################################
+
+*Please mind that integration of twister with pytest is still work in progress. Not every platform
+type is supported in pytest (yet). If you find any issue with the integration or have an idea for
+an improvement, please, let us know about it and open a GitHub issue/enhancement.*
+
+Introduction
+************
+
+Pytest is a python framework that *“makes it easy to write small, readable tests, and can scale to
+support complex functional testing for applications and libraries”* (`<https://docs.pytest.org/en/7.3.x/>`_).
+Python is known for its free libraries and ease of using it for scripting. In addition, pytest
+utilizes the concept of plugins and fixtures, increasing its expendability and reusability.
+A pytest plugin `pytest-twister-harness` was introduced to provide an integration between pytest
+and twister, allowing Zephyr’s community to utilize pytest functionality with keeping twister as
+the main framework.
+
+Integration with twister
+************************
+
+By default, there is nothing to be done to enable pytest support in twister. The plugin is
+developed as a part of Zephyr’s tree. To enable install-less operation, twister first extends
+``PYTHONPATH`` with path to this plugin, and then during pytest call, it appends the command with
+``-p twister_harness.plugin`` argument. If one prefers to use the installed version of the plugin,
+they must add ``--allow-installed-plugin`` flag to twister’s call.
+
+Pytest-based test suites are discovered the same way as other twister tests, i.e., by a presence
+of testcase/sample.yaml. Inside, a keyword ``harness`` tells twister how to handle a given test.
+In the case of ``harness: pytest``, most of twister workflow (test suites discovery,
+parallelization, building and reporting) remains the same as for other harnesses. The change
+happens during the execution step. The below picture presents a simplified overview of the
+integration.
+
+.. figure:: twister_and_pytest.svg
+   :figclass: align-center
+
+
+If ``harness: pytest`` is used, twister delegates the test execution to pytest, by calling it as
+a subprocess. Required parameters (such as build directory, device to be used, etc.) are passed
+through a CLI command. When pytest is done, twister looks for a pytest report (results.xml) and
+sets the test result accordingly.
+
+How to create a pytest test
+***************************
+
+An example of a pytest test is given at :zephyr_file:`samples/subsys/testsuite/pytest/shell/pytest/test_shell.py`.
+Twister calls pytest for each configuration from the .yaml file which uses ``harness: pytest``.
+By default, it points to ``pytest`` directory, located next to a directory with binary sources.
+A keyword ``pytest_root`` placed under ``harness_config`` section can be used to point to another
+location.
+
+Pytest scans the given folder looking for tests, following its default
+`discovery rules <https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#conventions-for-python-test-discovery>`_
+One can also pass some extra arguments to the pytest from yaml file using ``pytest_args`` keyword
+under ``harness_config``, e.g.: ``pytest_args: [‘-k=test_method’, ‘--log-level=DEBUG’]``.
+
+Two imports are important to include in .py sources:
+
+.. code-block:: python
+
+   import pytest  # noqa # pylint: disable=unused-import
+   from pytest_twister_harness.device.device_abstract import DeviceAbstract
+
+The first enables pytest-twister-harness plugin indirectly, as it is added with pytest.
+It also gives access to ``dut`` fixture. The second is important for type checking and enabling
+IDE hints for duts. The ``dut`` fixture is the core of pytest harness plugin. When used as an
+argument of a test function it gives access to a DeviceAbstract type object. The fixture yields a
+device prepared according to the requested type (native posix, qemu, hardware, etc.). All types of
+devices share the same API. This allows for writing tests which are device-type-agnostic.
+
+
+Limitations
+***********
+
+* Twister options like ``--pre-script``, ``--post-flash-script`` and ``--post-script`` are not handled by
+  pytest plugin during flashing/running application.
+* The whole pytest call is reported as one test in the final twister report (xml or json).
+* Device adapters in pytest plugin provide `iter_stdout` method to read from devices. In some
+  cases, it is not the most convenient way, and it will be considered how to improve this
+  (for example replace it with a simple read function with a given byte size and timeout arguments).
+* Not every platform type is supported in the plugin (yet).

samples/subsys/testsuite/pytest/testcase.yaml → samples/subsys/testsuite/pytest/basic/testcase.yaml

@@ -3,7 +3,7 @@ tests:
     platform_allow: native_posix
     harness: pytest
     harness_config:
-      pytest_args: ["--custom-pytest-arg", "foo"]
+      pytest_args: ["--custom-pytest-arg", "foo", "--cmdopt", "."]
     tags:
       - testing
       - pytest

samples/subsys/testsuite/pytest/shell/CMakeLists.txt

@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: Apache-2.0
+
+cmake_minimum_required(VERSION 3.20.0)
+find_package(Zephyr REQUIRED HINTS $ENV{ZEPHYR_BASE})
+project(shell)
+
+FILE(GLOB app_sources src/*.c)
+target_sources(app PRIVATE ${app_sources})

samples/subsys/testsuite/pytest/shell/prj.conf

@@ -0,0 +1,5 @@
+CONFIG_PRINTK=y
+CONFIG_SHELL=y
+CONFIG_LOG=y
+CONFIG_SHELL_BACKEND_SERIAL=y
+CONFIG_KERNEL_SHELL=y

samples/subsys/testsuite/pytest/shell/pytest/test_shell.py

@@ -0,0 +1,36 @@
+# Copyright (c) 2023 Nordic Semiconductor ASA
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import time
+import logging
+import pytest  # noqa # pylint: disable=unused-import
+
+from twister_harness.device.device_abstract import DeviceAbstract
+
+logger = logging.getLogger(__name__)
+
+
+def wait_for_message(iter_stdout, message, timeout=60):
+    time_started = time.time()
+    for line in iter_stdout:
+        if line:
+            logger.debug("#: " + line)
+        if message in line:
+            return True
+        if time.time() > time_started + timeout:
+            return False
+
+
+def test_shell_print_help(dut: DeviceAbstract):
+    time.sleep(1)  # wait for application initialization on DUT
+
+    dut.write(b'help\n')
+    assert wait_for_message(dut.iter_stdout, "see all available commands")
+
+
+def test_shell_print_version(dut: DeviceAbstract):
+    time.sleep(1)  # wait for application initialization on DUT
+
+    dut.write(b'kernel version\n')
+    assert wait_for_message(dut.iter_stdout, "Zephyr version")

samples/subsys/testsuite/pytest/shell/src/main.c

@@ -0,0 +1,14 @@
+/*
+ * Copyright (c) 2023 Nordic Semiconductor ASA
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+int main(void)
+{
+	/* Shell application source code is injected by applied Kconfg SHELL
+	 * options, no more "extra" functionalities are required for exemplary
+	 * pytest test.
+	 */
+	return 0;
+}

samples/subsys/testsuite/pytest/shell/testcase.yaml

@@ -0,0 +1,14 @@
+tests:
+  sample.pytest.shell:
+    filter: CONFIG_SERIAL and dt_chosen_enabled("zephyr,shell-uart")
+    min_ram: 40
+    harness: pytest
+    extra_configs:
+      - CONFIG_NATIVE_UART_0_ON_STDINOUT=y
+    integration_platforms:
+      - native_posix
+      - qemu_cortex_m3
+    tags:
+      - testing
+      - pytest
+      - shell

scripts/pylib/pytest-twister-harness/.gitignore

@@ -0,0 +1,62 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# Pycharm
+.idea/
+
+# VSCode
+.vscode/

scripts/pylib/pytest-twister-harness/README.rst

@@ -0,0 +1,47 @@
+==============
+Pytest Twister harness
+==============
+
+Installation
+------------
+
+If you plan to use this plugin with Twister, then you don't need to install it
+separately by pip. When Twister uses this plugin for pytest tests, it updates
+`PYTHONPATH` variable, and then extends pytest command by
+`-p twister_harness.plugin` argument.
+
+
+Usage
+-----
+
+Run exemplary test shell application by Twister:
+
+.. code-block:: sh
+
+  cd ${ZEPHYR_BASE}
+
+  # native_posix & QEMU
+  ./scripts/twister -p native_posix -p qemu_x86 -T samples/subsys/testsuite/pytest/shell
+
+  # hardware
+  ./scripts/twister -p nrf52840dk_nrf52840 --device-testing --device-serial /dev/ttyACM0 -T samples/subsys/testsuite/pytest/shell
+
+or build shell application by west and call pytest directly:
+
+.. code-block:: sh
+
+  export PYTHONPATH=${ZEPHYR_BASE}/scripts/pylib/pytest-twister-harness/src:${PYTHONPATH}
+
+  cd ${ZEPHYR_BASE}/samples/subsys/testsuite/pytest/shell
+
+  # native_posix
+  west build -p -b native_posix -- -DCONFIG_NATIVE_UART_0_ON_STDINOUT=y
+  pytest --twister-harness --device-type=native --build-dir=build -p twister_harness.plugin
+
+  # QEMU
+  west build -p -b qemu_x86 -- -DQEMU_PIPE=qemu-fifo
+  pytest --twister-harness --device-type=qemu --build-dir=build -p twister_harness.plugin
+
+  # hardware
+  west build -p -b nrf52840dk_nrf52840
+  pytest --twister-harness --device-type=hardware --device-serial=/dev/ttyACM0 --build-dir=build -p twister_harness.plugin

scripts/pylib/pytest-twister-harness/pyproject.toml

@@ -0,0 +1,6 @@
+[build-system]
+build-backend = "setuptools.build_meta"
+requires = [
+    "setuptools >= 48.0.0",
+    "wheel",
+]

scripts/pylib/pytest-twister-harness/setup.cfg

@@ -0,0 +1,33 @@
+[metadata]
+name = pytest-twister-harness
+version = attr: twister_harness.__version__
+description = Plugin for pytest to run tests which require interaction with real and simulated devices
+long_description = file: README.rst
+python_requires = ~=3.8
+classifiers =
+    Development Status :: 3 - Alpha
+    Intended Audience :: Developers
+    Topic :: Software Development :: Embedded Systems
+    Topic :: Software Development :: Quality Assurance
+    Operating System :: Posix :: Linux
+    Operating System :: Microsoft :: Windows
+    Programming Language :: Python :: 3.8
+    Programming Language :: Python :: 3.9
+    Programming Language :: Python :: 3.10
+    Programming Language :: Python :: 3.11
+
+[options]
+packages = find:
+package_dir =
+    =src
+install_requires =
+    psutil
+    pyserial
+    pytest>=7.0.0
+
+[options.packages.find]
+where = src
+
+[options.entry_points]
+pytest11 =
+    twister_harness = twister_harness.plugin

scripts/pylib/pytest-twister-harness/setup.py

@@ -0,0 +1,7 @@
+# Copyright (c) 2023 Nordic Semiconductor ASA
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import setuptools
+
+setuptools.setup()

scripts/pylib/pytest-twister-harness/src/twister_harness/__init__.py

@@ -0,0 +1,5 @@
+# Copyright (c) 2023 Nordic Semiconductor ASA
+#
+# SPDX-License-Identifier: Apache-2.0
+
+__version__ = '0.0.1'

scripts/pylib/pytest-twister-harness/src/twister_harness/constants.py

@@ -0,0 +1,8 @@
+# Copyright (c) 2023 Nordic Semiconductor ASA
+#
+# SPDX-License-Identifier: Apache-2.0
+
+from __future__ import annotations
+
+QEMU_FIFO_FILE_NAME: str = 'qemu-fifo'
+END_OF_DATA = object()  #: used for indicating that there will be no more data in queue

scripts/pylib/pytest-twister-harness/src/twister_harness/device/__init__.py

@@ -0,0 +1,3 @@
+# Copyright (c) 2023 Nordic Semiconductor ASA
+#
+# SPDX-License-Identifier: Apache-2.0