Added new localstack (SQS) broker (#374)

* Added new localstack (SQS) broker

* Added documentation

* Hotfix

Author: Nusnus

docs/getting-started/first-steps.rst

@@ -683,6 +683,27 @@ Fixtures
 
 A list of available fixtures for the broker can be found in the :mod:`pytest_celery.vendors.redis.broker.fixtures` module.
 
+Localstack (SQS) Broker
+-----------------------
+
+The Localstack broker uses the ``localstack/localstack`` version for the underlying container.
+
+Container
+#########
+
+The :class:`LocalstackContainer <pytest_celery.vendors.localstack.container.LocalstackContainer>` is used
+to describe the ``localstack/localstack`` docker image.
+
+Node
+####
+
+The :class:`LocalstackTestBroker <pytest_celery.vendors.localstack.api.LocalstackTestBroker>` is used to represent the broker node.
+
+Fixtures
+########
+
+A list of available fixtures for the broker can be found in the :mod:`pytest_celery.vendors.localstack.fixtures` module.
+
 .. _redis-backend:
 
 Redis Backend

docs/getting-started/introduction.rst

@@ -153,6 +153,7 @@ pytest-celery is…
 
             - RabbitMQ.
             - Redis.
+            - SQS (via Localstack)
             - Custom broker.
 
         - **Backends**

docs/getting-started/vendors.rst

@@ -20,13 +20,15 @@ the box.  This page lists the supported vendors and their status.
 Brokers
 ~~~~~~~
 
-+---------------+--------------+--------------+
-| **Name**      | **Status**   | **Enabled**  |
-+---------------+--------------+--------------+
-| *RabbitMQ*    | Stable       | Yes          |
-+---------------+--------------+--------------+
-| *Redis*       | Stable       | Yes          |
-+---------------+--------------+--------------+
++-----------------------+--------------+--------------+
+| **Name**              | **Status**   | **Enabled**  |
++-----------------------+--------------+--------------+
+| *RabbitMQ*            | Stable       | Yes          |
++-----------------------+--------------+--------------+
+| *Redis*               | Stable       | Yes          |
++-----------------------+--------------+--------------+
+| *Localstack (SQS)*    | Beta         | No           |
++-----------------------+--------------+--------------+
 
 Backends
 ~~~~~~~~
@@ -39,8 +41,7 @@ Backends
 | *Memcache*    | Experimental | No           |
 +---------------+--------------+--------------+
 
-Experimental brokers may be functional but are not confirmed to be
-production ready.
+Experimental or Beta status means it may be functional but are not confirmed to be production ready.
 
 Enabled means that it is automatically added to the test :ref:`setup-matrix`
 when running the test suite :ref:`if the vendor dependencies are installed <installation>`.
@@ -67,6 +68,61 @@ The Dockerfile is published with the source code and can be found using
    :language: docker
    :caption: pytest_celery.vendors.worker.Dockerfile
 
+.. _localstack-broker:
+
+Localstack (SQS) Broker
+=======================
+
+To use the Localstack broker, you will need add additional configuration to the test setup.
+
+You may add this to ``conftest.py`` to configure the Localstack broker.
+
+.. code-block:: python
+
+    import os
+
+    import pytest
+    from celery import Celery
+
+    from pytest_celery import LOCALSTACK_CREDS
+
+    @pytest.fixture
+    def default_worker_env(default_worker_env: dict) -> dict:
+        default_worker_env.update(LOCALSTACK_CREDS)
+        return default_worker_env
+
+
+    @pytest.fixture(scope="session", autouse=True)
+    def set_aws_credentials():
+        os.environ.update(LOCALSTACK_CREDS)
+
+
+    @pytest.fixture
+    def default_worker_app(default_worker_app: Celery) -> Celery:
+        app = default_worker_app
+        if app.conf.broker_url and app.conf.broker_url.startswith("sqs"):
+            app.conf.broker_transport_options["region"] = LOCALSTACK_CREDS["AWS_DEFAULT_REGION"]
+        return app
+
+And to enable the Localstack broker in the default :ref:`setup-matrix`, add the following configuration to ``conftest.py``.
+
+.. code-block:: python
+
+    from pytest_celery import ALL_CELERY_BROKERS
+    from pytest_celery import CELERY_LOCALSTACK_BROKER
+    from pytest_celery import CeleryTestBroker
+    from pytest_celery import _is_vendor_installed
+
+    if _is_vendor_installed("localstack"):
+        ALL_CELERY_BROKERS.add(CELERY_LOCALSTACK_BROKER)
+
+
+    @pytest.fixture(params=ALL_CELERY_BROKERS)
+    def celery_broker(request: pytest.FixtureRequest) -> CeleryTestBroker:  # type: ignore
+        broker: CeleryTestBroker = request.getfixturevalue(request.param)
+        yield broker
+        broker.teardown()
+
 .. _custom-vendors:
 
 Custom Vendors
@@ -210,6 +266,7 @@ There's no ``session`` vendor class for other vendors.
 
 - For RabbitMQ Broker use :func:`default_rabbitmq_broker_cls <pytest_celery.vendors.rabbitmq.fixtures.default_rabbitmq_broker_cls>`.
 - For Redis Broker use :func:`default_redis_broker_cls <pytest_celery.vendors.redis.broker.fixtures.default_redis_broker_cls>`.
+- For SQS Broker use :func:`default_localstack_broker_cls <pytest_celery.vendors.localstack.fixtures.default_localstack_broker_cls>`.
 - For Redis Backend use :func:`default_redis_backend_cls <pytest_celery.vendors.redis.backend.fixtures.default_redis_backend_cls>`.
 - For Memcache Backend use :func:`default_memcached_backend_cls <pytest_celery.vendors.memcached.fixtures.default_memcached_backend_cls>`.
 

docs/includes/installation.txt

@@ -36,6 +36,7 @@ To install the vendors, you may either install all of the dependencies manually,
 - ``all``: Installs all vendors.
 - ``redis``: Installs Redis vendor, providing **broker** and **result backend** components.
 - ``memcached``: Installs Memcached vendor, providing a **result backend** component.
+- ``sqs``: Installs Localstack vendor, providing an **SQS broker** component.
 
 The following extra is installed by default:
 
@@ -56,6 +57,15 @@ RabbitMQ & Redis combo
 
 This will configure the plugin to generate all possible setups using only RabbitMQ and Redis vendors.
 
+SQS & Redis combo
+-----------------
+
+.. code-block:: console
+
+    pip install -U "pytest-celery[redis,sqs]"
+
+This will configure the plugin to generate all possible setups using only Localstack and Redis vendors.
+
 All vendors
 -----------
 

docs/reference/pytest_celery.vendors.localstack.rst

@@ -0,0 +1,45 @@
+pytest\_celery.vendors.localstack package
+=========================================
+
+Submodules
+----------
+
+pytest\_celery.vendors.localstack.api module
+--------------------------------------------
+
+.. automodule:: pytest_celery.vendors.localstack.api
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+pytest\_celery.vendors.localstack.container module
+--------------------------------------------------
+
+.. automodule:: pytest_celery.vendors.localstack.container
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+pytest\_celery.vendors.localstack.defaults module
+-------------------------------------------------
+
+.. automodule:: pytest_celery.vendors.localstack.defaults
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+pytest\_celery.vendors.localstack.fixtures module
+-------------------------------------------------
+
+.. automodule:: pytest_celery.vendors.localstack.fixtures
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: pytest_celery.vendors.localstack
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/reference/pytest_celery.vendors.rst

@@ -7,6 +7,7 @@ Subpackages
 .. toctree::
    :maxdepth: 4
 
+   pytest_celery.vendors.localstack
    pytest_celery.vendors.memcached
    pytest_celery.vendors.rabbitmq
    pytest_celery.vendors.redis

docs/userguide/advanced-installation.rst

@@ -41,6 +41,7 @@ For example, let's assume you want to use the RabbitMQ/Redis combination.
 
 1. For RabbitMQ we will need :pypi:`kombu <kombu>`.
 2. For Redis we will need :pypi:`redis <redis>`.
+3. For SQS we will need :pypi:`kombu <kombu>` with the sqs extra.
 
 To install the plugin with the RabbitMQ/Redis combination, you will need to install the following dependencies:
 
@@ -53,8 +54,8 @@ Let's break down the command:
 - The ``pytest-celery`` is the plugin package, it will install the plugin alongside Celery and its dependencies,including **Kombu** (if not installed already).
 - The ``[redis]`` is the feature flag for the Redis vendor, it will install the :pypi:`redis <redis>` package and configure the plugin to use it which will add the Redis backend and Redis broker components to the default setup matrix.
 
-Experimental Vendors
-~~~~~~~~~~~~~~~~~~~~
+Experimental or Beta Vendors
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 :ref:`vendors` that are in not stable, will not be added to the default setup matrix.
 To use the experimental vendors, you will need to configure the setup matrix manually.

docs/userguide/setup-matrix.rst

@@ -128,3 +128,47 @@ Redis Broker and Redis Backend
         cluster = CeleryBackendCluster(celery_redis_backend)
         yield cluster
         cluster.teardown()
+
+SQS Broker and No Backend
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Use only Localstack as the broker.
+
+.. code-block:: python
+
+    @pytest.fixture
+    def celery_broker_cluster(celery_localstack_broker: LocalstackTestBroker) -> CeleryBrokerCluster:
+        cluster = CeleryBrokerCluster(celery_localstack_broker)
+        yield cluster
+        cluster.teardown()
+
+2. :ref:`Disable the backend <disable_backend>`.
+
+.. code-block:: python
+
+    @pytest.fixture
+    def celery_backend_cluster():
+        return None
+
+SQS Broker and Redis Backend
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Use only Localstack as the broker.
+
+.. code-block:: python
+
+    @pytest.fixture
+    def celery_broker_cluster(celery_localstack_broker: LocalstackTestBroker) -> CeleryBrokerCluster:
+        cluster = CeleryBrokerCluster(celery_localstack_broker)
+        yield cluster
+        cluster.teardown()
+
+2. Use Redis as the backend.
+
+.. code-block:: python
+
+    @pytest.fixture
+    def celery_backend_cluster(celery_redis_backend: RedisTestBackend) -> CeleryBackendCluster:
+        cluster = CeleryBackendCluster(celery_redis_backend)
+        yield cluster
+        cluster.teardown()

src/pytest_celery/__init__.py

@@ -40,6 +40,17 @@
 from pytest_celery.fixtures.worker import celery_worker_cluster_config
 from pytest_celery.vendors import _is_vendor_installed
 
+if _is_vendor_installed("localstack"):
+    from pytest_celery.vendors.localstack.api import LocalstackTestBroker
+    from pytest_celery.vendors.localstack.container import LocalstackContainer
+    from pytest_celery.vendors.localstack.defaults import *
+    from pytest_celery.vendors.localstack.fixtures import celery_localstack_broker
+    from pytest_celery.vendors.localstack.fixtures import default_localstack_broker
+    from pytest_celery.vendors.localstack.fixtures import default_localstack_broker_cls
+    from pytest_celery.vendors.localstack.fixtures import default_localstack_broker_env
+    from pytest_celery.vendors.localstack.fixtures import default_localstack_broker_image
+    from pytest_celery.vendors.localstack.fixtures import default_localstack_broker_ports
+
 if _is_vendor_installed("memcached"):
     from pytest_celery.vendors.memcached.api import MemcachedTestBackend
     from pytest_celery.vendors.memcached.container import MemcachedContainer

src/pytest_celery/defaults.py

@@ -13,6 +13,8 @@
 from pytest_docker_tools import network
 
 from pytest_celery.vendors import _is_vendor_installed
+from pytest_celery.vendors.localstack.defaults import CELERY_LOCALSTACK_BROKER
+from pytest_celery.vendors.localstack.defaults import *
 from pytest_celery.vendors.memcached.defaults import CELERY_MEMCACHED_BACKEND
 from pytest_celery.vendors.memcached.defaults import *
 from pytest_celery.vendors.rabbitmq.defaults import CELERY_RABBITMQ_BROKER
@@ -53,6 +55,11 @@
 if _is_vendor_installed("memcached") and False:
     ALL_CELERY_BACKENDS.add(CELERY_MEMCACHED_BACKEND)
 
+# Localstack is disabled by default regardless of its availability due to its beta status.
+if _is_vendor_installed("localstack") and False:
+    # Uses Kombu
+    ALL_CELERY_BROKERS.add(CELERY_LOCALSTACK_BROKER)
+
 # Worker setup is assumed to be always available.
 ALL_CELERY_WORKERS = (CELERY_SETUP_WORKER,)
 

src/pytest_celery/vendors/localstack/__init__.py

@@ -0,0 +1,5 @@
+"""The pytest-celery plugin provides a set of built-in components called
+:ref:`vendors`.
+
+This module is part of the Localstack Broker vendor.
+"""

src/pytest_celery/vendors/localstack/api.py

@@ -0,0 +1,13 @@
+"""The pytest-celery plugin provides a set of built-in components called
+:ref:`vendors`.
+
+This module is part of the Localstack Broker vendor.
+"""
+
+from __future__ import annotations
+
+from pytest_celery.api.broker import CeleryTestBroker
+
+
+class LocalstackTestBroker(CeleryTestBroker):
+    pass