This file contains project-specific context for AI coding agents. The project is a Python client library for the PGMQ PostgreSQL extension.
pgmq is the official Python client for PGMQ (Postgres Message Queue). It exposes synchronous and asynchronous APIs, with both raw driver (psycopg/asyncpg) and SQLAlchemy-based backends. The public API surface is intentionally identical across all four client variants so users can swap implementations with minimal changes.
- Package name:
pgmq - Version:
1.1.1(pyproject.toml); runtime__version__is read from installed package metadata viaimportlib.metadata.version("pgmq")(falls back to"unknown"). - License: Apache-2.0
- Python support:
>=3.10(classifiers: 3.10–3.14) - Authors: Adam Hendel, Ali Tavallaie
- Repository: https://github.com/pgmq/pgmq-py
- Documentation: https://pgmq.github.io/pgmq-py/
| Layer | Choices |
|---|---|
| Build / package manager | uv (build backend uv_build) |
| Sync driver | psycopg[binary,pool]>=3.2.10 |
| Async driver | asyncpg>=0.30.0 (optional extra [async]) |
| SQLAlchemy | sqlalchemy>=2.0.0 (optional extras [sqlalchemy] / [sqlalchemy-async]) |
| JSON | orjson>=3.11.3 |
| Lint / format | ruff>=0.12.12 |
| Logging | stdlib logging with optional loguru fallback |
| Benchmarks | locust, pandas, pyyaml, scipy, typer |
| Documentation | mkdocs, mkdocs-material, mkdocstrings, mike |
src/pgmq/
__init__.py # Public exports, backward-compat aliases, dynamic version
base.py # PGMQConfig dataclass + BaseQueue (shared init/logging)
_client_fields.py # Shared dataclass fields for all PGMQueue clients
sync_operations.py # SyncPGMQueueOperationsMixin — all sync public methods (write once)
async_operations.py # AsyncPGMQueueOperationsMixin — all async public methods (write once)
queue.py # Thin sync psycopg backend (pool + _execute + JSON encoding)
async_queue.py # Thin async asyncpg backend
sqlalchemy_queue.py # Thin sync SQLAlchemy backend
sqlalchemy_async_queue.py # Thin async SQLAlchemy backend
_sql.py # All SQL templates + conversion helpers (%s → $N, %s → :param_N)
messages.py # Dataclasses mapping PGMQ composite types
decorators.py # Transaction decorators (transaction, async_transaction, sqlalchemy_transaction, sqlalchemy_async_transaction)
logger.py # LoggingManager with dual stdlib/loguru backend
notify_listener.py # SyncNotificationListener + AsyncNotificationListener (PostgreSQL NOTIFY/LISTEN)
tests/
utils.py # PGMQTestCase base class + env-driven PG_* constants via PGMQConfig
test_integration.py # Sync psycopg integration tests
test_async_integration.py # Async asyncpg integration tests
test_sqlalchemy_integration.py # Sync SQLAlchemy integration tests
test_sqlalchemy_async_integration.py # Async SQLAlchemy integration tests
test_features.py # Partitioning, notifications, validation utilities
test_routing.py # Topic routing (bind/send/unbind/test_routing)
test_notify_listener.py # NOTIFY/LISTEN tests for all four backends
test_sql_conversion.py # Pure unit tests for _sql.py conversions (no DB required)
test_logger.py # Logger unit tests
example/
example_app_sync.py # Transaction decorator usage examples
example_app_async.py # Async transaction usage examples
benches/
bench.py / runner.py / ... # Locust-based load testing (dependency-group "bench")
docs/
index.md # Documentation homepage
getting_started.md # Installation & quick start
configuration.md # PGMQConfig reference
clients.md # Four backend clients
queue_management.md # Create, drop, list, purge queues
messages.md # Dataclasses (Message, QueueMetrics, ...)
sending_messages.md # send, send_batch, headers, delay
reading_messages.md # read, read_with_poll, FIFO variants, conditional reads
deleting_and_archiving.md # delete, archive, pop, purge
visibility_timeout.md # set_vt
topic_routing.md # Topic-based routing
metrics.md # Queue statistics
notifications.md # NOTIFY/LISTEN & listeners
transactions.md # Decorators & manual transactions
logging.md # Logging configuration
utilities.md # Validation, FIFO indexes
backward_compatibility.md # Migration notes
development.md # Tests, contributing, MkDocs/Mike
mkdocs.yml # MkDocs configuration (Material theme, Mike versioning)
All common tasks are wrapped in the Makefile:
# Install everything (dev + all extras + bench)
uv sync --all-groups --all-extras
# Format code and auto-fix lint issues
make format
# → uv run ruff format src/
# → uv run ruff check --fix --exit-zero src/
# Run lint checks without modifying files
make lint
# → uv run ruff check src/
# → uv run ruff format --check src/
# Run the full test suite (spins up Docker PostgreSQL automatically)
make test
# → docker rm -f pgmq-postgres
# → docker run -d --name pgmq-postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 ghcr.io/pgmq/pg18-pgmq:latest
# → sleep 10
# → uv run python -m unittest discover -s tests -p "test_*.py"
# Run tests against an existing Postgres (no Docker)
make test-env
# → uv run python -m unittest discover -s tests -p "test_*.py"If you already have a Postgres with PGMQ running:
uv run python -m unittest discover -s tests -p "test_*.py"# Start a local PGMQ-enabled Postgres
make run-pgmq-postgres
# → docker run -d --name pgmq-postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 ghcr.io/pgmq/pg18-pgmq:latest
# Tear it down
make clear-postgres# Serve docs locally with live reload
make docs-serve
# → uv run --group docs mkdocs serve
# Build static site
make docs-build
# → uv run --group docs mkdocs build
# Deploy a versioned release (uses mike)
make docs-deploy VERSION=1.1.1
# → uv run --group docs mike deploy 1.1.1 latest
# → uv run --group docs mike set-default latest- Formatter / Linter:
ruffonly (no black, no isort, no flake8). - Line length: default ruff line length (88).
- Import style:
ruffhandles import sorting; do not add# isort: skipunless absolutely necessary. - Docstrings: Every module and public class/method should have a docstring. Use imperative mood ("Create a queue", not "Creates a queue").
- Type hints: Use modern
typingimports (Optional,List,Dict,Union,Any). The codebase targets Python 3.10+. - Logging: Never use bare
print()in library code. Uselog_with_context(self.logger, logging.DEBUG, "...", key=value). - Section dividers: In large classes (e.g.,
PGMQueue), use====comment blocks to group methods by feature area (Queue Management, Sending Messages, Reading Messages, etc.).
- A running PostgreSQL instance with the
pgmqextension installed. - Default connection falls back to
localhost:5432,postgres/postgres/postgres. - Override via environment variables:
PG_HOST,PG_PORT,PG_DATABASE,PG_USERNAME,PG_PASSWORD,DATABASE_URL. tests/utils.pyusesPGMQConfig()to resolve connection settings the same way production code does.
tests/utils.pydefinesPGMQTestCase(sync base class). It creates a uniquely-named queue per class, purges it insetUp, and drops it intearDownClass.- Async tests extend
unittest.IsolatedAsyncioTestCaseand useasyncSetUp/asyncTearDown. - Tests that depend on newer PGMQ features (topic routing, conditional reads, validation functions, partitioning, advanced notify) must gracefully skip when
UndefinedFunction,RaiseException, or related errors are raised, because the CI image may lag behind the bleeding-edge extension. test_sql_conversion.pyis the only test file that does not need a database.
- If it is sync + psycopg, add to
tests/test_integration.pyor create a new file and inherit fromtests.utils.PGMQTestCase. - If it is async + asyncpg, add to
tests/test_async_integration.pyand inherit fromunittest.IsolatedAsyncioTestCase. - If it tests SQLAlchemy variants, add to the corresponding
test_sqlalchemy_*_integration.py. - If it tests pure Python logic (no DB), add to
tests/test_sql_conversion.pyor a new file.
The library maintains four PGMQueue classes with identical public method signatures. Public methods are implemented once in shared operation mixins; each backend file only provides connection execution and JSON encoding.
| Layer | Module | Responsibility |
|---|---|---|
| Operations (sync) | sync_operations.py |
SyncPGMQueueOperationsMixin — all sync PGMQ methods |
| Operations (async) | async_operations.py |
AsyncPGMQueueOperationsMixin — all async PGMQ methods |
| Shared fields | _client_fields.py |
PGMQClientFields — connection/config dataclass fields |
| Backend adapter | queue.py, etc. |
_execute*, _encode_jsonb, pool/engine init |
Transaction decorators are applied automatically via __init_subclass__ when a backend sets _transaction_decorator (sync) or _async_transaction_decorator (async).
The four concrete clients:
| Module | Class | Backend | Transaction decorator |
|---|---|---|---|
pgmq.queue |
PGMQueue |
psycopg + ConnectionPool |
@transaction |
pgmq.async_queue |
PGMQueue |
asyncpg + Pool |
@async_transaction |
pgmq.sqlalchemy_queue |
PGMQueue |
SQLAlchemy Engine (sync) |
@sqlalchemy_transaction |
pgmq.sqlalchemy_async_queue |
PGMQueue |
SQLAlchemy AsyncEngine |
@sqlalchemy_async_transaction |
pgmq/__init__.py re-exports aliases:
PGMQueue→ sync psycopg version (backward compatible)SyncPGMQueue,AsyncPGMQueueSQLAlchemyPGMQueue,SQLAlchemyAsyncPGMQueue
Optional backends are imported inside try/except ImportError blocks — AsyncPGMQueue, SQLAlchemyPGMQueue, and SQLAlchemyAsyncPGMQueue are set to None when their dependencies are not installed.
Each PGMQueue exposes the same methods, grouped by feature area:
| Area | Methods |
|---|---|
| Queue management | create_queue, create_partitioned_queue, drop_queue, list_queues, validate_queue_name |
| Sending | send, send_batch, send_topic, send_batch_topic |
| Topic routing | bind_topic, unbind_topic, list_topic_bindings, test_routing |
| Reading | read, read_batch, read_with_poll, read_grouped, read_grouped_with_poll, read_grouped_rr, read_grouped_rr_with_poll |
| Pop | pop |
| Delete / archive | delete, delete_batch, archive, archive_batch, purge |
| Visibility timeout | set_vt (single or batch msg_id; int or datetime vt) |
| Metrics | metrics, metrics_all |
| Notifications | enable_notify, disable_notify, update_notify, list_notify_throttles |
| Utilities | validate_routing_key, validate_topic_pattern, create_fifo_index, create_fifo_indexes_all, convert_archive_partitioned, detach_archive |
Async-only lifecycle methods: init() (required before operations), close() (asyncpg pool teardown).
PGMQConfig is a dataclass that reads from environment variables by default but allows explicit overrides. It supports:
- Individual fields (
host,port,database,username,password) - Full connection strings (
conn_stringarg orDATABASE_URLenv var) in URI or libpq format - Robust URI parsing: splits on the last
@so passwords containing@work; usespsycopg.conninfo.conninfo_to_dictfor libpq strings init_extension: bool = True— clients auto-runCREATE EXTENSION IF NOT EXISTS pgmq CASCADEon initdsnproperty (libpq format) andasync_dsnproperty (URI format with URL-encoded credentials)
Clients always connect via the re-assembled config.dsn / config.async_dsn, not the raw conn_string, to fix malformed URIs.
BaseQueue handles logger creation via LoggingManager.get_logger(...).
All SQL lives in src/pgmq/_sql.py:
- Constants use psycopg
%splaceholders. get_send_sql,get_send_batch_sql,get_send_topic_sql,get_send_batch_topic_sql,get_set_vt_sqlreturn the correct template based on flags (has_headers,has_delay,delay_is_timestamp,is_batch,vt_is_timestamp)._iter_placeholdersskips%sinside SQL string literals before conversion._convert_psycopg_to_asyncpgconverts%s→$1,$2, etc. Results are pre-computed intoASYNC_SQL_MAPat import time.convert_sql_paramsconverts%s+ tuple params into SQLAlchemy:param_N+ dict form, with special JSONB handling for dicts/lists and::cast escaping.
Decorators check whether conn is already provided in kwargs. If not, they acquire a connection/transaction from the pool or engine, inject it as conn, and commit/rollback automatically. This design lets users compose multiple operations into a single transaction by passing the same conn around, or by using the decorator.
- psycopg:
with conn.transaction() - asyncpg: explicit
txn.start()/commit()/rollback() - SQLAlchemy:
engine.begin()(sync and async)
| Class | PGMQ type | Notes |
|---|---|---|
Message |
message_record |
from_row with optional json_parser |
QueueRecord |
queue_record |
list_queues() return type |
QueueMetrics |
metrics_result |
Handles old (6-col) and new (7-col) schemas via queue_visible_length |
TopicBinding |
topic binding row | |
RoutingResult |
test_routing result |
|
BatchTopicResult |
send_batch_topic result |
Used internally, not in __all__ |
NotificationThrottle |
notify throttle row |
Row-to-object mapping uses from_row(cls, row, json_parser=None) classmethods. Rows can be tuples or mappings (psycopg/asyncpg/SQLAlchemy return different types). _get_value helper abstracts over both, preferring name-based access with index fallback.
SyncNotificationListener— dedicated psycopg connection withautocommit=True, listens on channelpgmq.q_{queue_name}.INSERTAsyncNotificationListener— asyncpg connection withadd_listener, dispatches viaasyncio.Queue- Both quote channel names and handle empty/malformed JSON payloads gracefully
LoggingManager.get_logger()— library-safe by default (no handlers injected unlessverbose,log_filename, etc. are set)- Dual backend: stdlib
loggingorloguru(when installed) LoggingManager.configure_global_logging()for app-wide setuplog_with_context(),log_performance()decorator,PGMQLoggeralias for backward compatLoggingManager._test_modedisables loguruenqueuefor synchronous test assertions
- No SQL string interpolation of user data. All queries in
_sql.pyuse placeholders (%s,$N, or:param_N). Queue names, routing keys, and patterns are passed as bound parameters to the PGMQ extension functions. - Connection string privacy:
PGMQConfig.conn_stringhasrepr=Falseto avoid leaking credentials in logs or tracebacks. - Queue name validation:
validate_queue_namedelegates topgmq.validate_queue_name()in the database. - Notification channels: The listener code quotes channel names (
LISTEN "{channel}") to avoid identifier injection.
- Triggers on PR/push to
mainwhensrc/**,tests/**,pyproject.toml, or the workflow itself changes. - Python 3.13 via
uv python install 3.13. - Jobs:
lints—uv sync --all-groups+make linttests—uv sync --all-groups --all-extras+make test(requires Docker)
- Triggers on tag push (
v*). - Runs full test suite,
uv build,uv publishto PyPI (OIDC), and creates a GitHub Release with generated notes. - Only runs on
pgmq/pgmq-pyrepository.
.pre-commit-config.yaml runs ruff with --fix and ruff-format.
- Triggers on push to
mainand on tags (v*). - Uses
miketo deploy versioned documentation to thegh-pagesbranch. - Jobs:
- Push to
main— deploys asmainversion, updateslatestalias. - Tag push (
v1.1.1) — deploys as1.1.1version, updateslatestalias. - Sets
latestas the default version on every deploy.
- Push to
- GitHub Pages source must be set to the
gh-pagesbranch (/root) in repo settings. - Live site: https://pgmq.github.io/pgmq-py/
-
Async clients require explicit initialization.
queue = AsyncPGMQueue(...) await queue.init() # required before any operation
The sync psycopg and sync SQLAlchemy clients initialize automatically in
__post_init__. -
SQLAlchemy async client also requires
await queue.init(). It can accept an externalAsyncEnginevia theenginekwarg. -
Do not duplicate SQL strings. If you need to add a new query, add it to
src/pgmq/_sql.py, add it to_ALL_SQL_CONSTANTS, and useget_*_sqlhelpers if there are parameter variants. -
Keep
ASYNC_SQL_MAPin mind. Any new SQL constant added to_sql.pymust be included in_ALL_SQL_CONSTANTSso the asyncpg pre-computed map picks it up. -
Backward compatibility:
tzis an alias fordelayinsend().list_queues()returnsList[QueueRecord](it used to returnList[str]). AUserWarningis emitted.PGMQueueinpgmq/__init__.pyis aliased to the sync version.read_batch()is a backward-compat alias forread(..., qty=batch_size).
-
Version bumping: Update
pyproject.tomlversion when releasing. Runtime__version__is derived from installed package metadata automatically. -
Tests assume feature availability: Newer PGMQ features (topic routing, conditional reads, validation utilities, partitioning, advanced notify) are not guaranteed in all DB versions. Tests should
self.skipTest(...)or catchUndefinedFunction/RaiseException. -
init_extension=False: Set onPGMQConfigor client kwargs when the extension is pre-installed (e.g., hosted Postgres with SQL-only PGMQ install) to skipCREATE EXTENSION. -
Conditional reads:
read()andread_with_poll()accept an optionalconditional: Dict[str, Any]parameter for JSONB-based message filtering. Requires a recent PGMQ extension version. -
Connection strings with special characters: Always rely on
PGMQConfigparsing — clients use the re-builtdsn/async_dsn, not the raw input string.
Everything users import from from pgmq import ...:
| Category | Names |
|---|---|
| Clients | PGMQueue (sync alias), SyncPGMQueue, AsyncPGMQueue, SQLAlchemyPGMQueue, SQLAlchemyAsyncPGMQueue |
| Dataclasses | Message, QueueMetrics, QueueRecord, TopicBinding, RoutingResult, NotificationThrottle |
| Decorators | transaction, async_transaction, sqlalchemy_transaction, sqlalchemy_async_transaction |
| Logging | PGMQLogger, create_logger, log_performance |
| Version | __version__ |
Not exported but available via submodules: PGMQConfig, BaseQueue, SyncNotificationListener, AsyncNotificationListener, BatchTopicResult.
Optional clients (AsyncPGMQueue, SQLAlchemyPGMQueue, SQLAlchemyAsyncPGMQueue) are None when their extra is not installed.
pip install pgmq[async] # asyncpg
pip install pgmq[sqlalchemy] # SQLAlchemy sync
pip install pgmq[sqlalchemy-async] # SQLAlchemy async + asyncpg| Group | Purpose |
|---|---|
dev |
ruff, pre-commit, loguru, python-dotenv |
docs |
mkdocs, mkdocs-material, mkdocstrings, mike |
bench |
locust, pandas, pyyaml, scipy, typer |
All four clients share the same public method signatures but differ in how they run SQL internally. Add or change public methods in the operation mixins (sync_operations.py / async_operations.py), not in each backend file.
- Pool:
psycopg_pool.ConnectionPool, auto-opened in__post_init__ - Execute:
conn.execute(sql, params)with raw%sSQL from_sql.py - JSONB params: wrap dicts/lists in
psycopg.types.json.Jsonb - Row parsing:
Message.from_row(row, lambda x: x)— psycopg returns dicts directly - Transaction:
@transaction→with conn.transaction()
- Pool:
asyncpg.create_pool(config.async_dsn), requiresawait init() - Execute: looks up
_sql.ASYNC_SQL_MAP[sql]for$1placeholders, passes*params - JSONB params:
orjson.dumps(m).decode("utf-8")for message arrays; asyncpg handles encoding - Row parsing:
Message.from_row(row, _parse_jsonb)— may needorjson.loadson strings - Transaction:
@async_transaction→ explicittxn.start()/commit()/rollback() - Cleanup:
await close()shuts down the pool
- Engine:
create_engine(config.async_dsn.replace("postgresql://", "postgresql+psycopg://")) - Execute:
_sql.convert_sql_params(sql, params)→text(converted_sql)with:param_Ndict - JSONB:
convert_sql_paramsauto-json.dumpsdicts/lists when::jsonbcast follows placeholder - Row parsing:
Message.from_row(row, _parse_jsonb) - Transaction:
@sqlalchemy_transaction→with engine.begin() as conn - External engine: pass
engine=kwarg to skip auto-creation
- Engine:
create_async_enginewithpostgresql+asyncpg://driver prefix - Execute: same
convert_sql_params+text()pattern as sync SQLAlchemy - Requires
await init(); accepts externalAsyncEngineviaengine=kwarg - Transaction:
@sqlalchemy_async_transaction→async with engine.begin() as conn
- Add SQL to
src/pgmq/_sql.pyas a constant with%splaceholders. - Register the constant in
_ALL_SQL_CONSTANTS(required for asyncpg). - Add a
get_*_sqlhelper if the method has parameter variants (headers, delay, batch, etc.). - Implement once in each operations mixin:
- Sync method →
src/pgmq/sync_operations.py(SyncPGMQueueOperationsMixin) - Async method →
src/pgmq/async_operations.py(AsyncPGMQueueOperationsMixin) - Use
self._encode_jsonb()/self._encode_jsonb_list()for JSONB params - Use
Message.from_row(row, self._json_parser)for message rows
- Sync method →
- If the method needs a transaction, add its name to
TRANSACTIONAL_SYNC_METHODSorTRANSACTIONAL_ASYNC_METHODSin the same file (decorators are applied automatically). - Do not duplicate logic in
queue.py,async_queue.py,sqlalchemy_queue.py, orsqlalchemy_async_queue.py— those files only implement_execute*and encoding. - Add tests in the matching integration test files (all four backends if applicable).
- Gracefully skip in tests if the PGMQ function may not exist yet (
UndefinedFunction). - Update user docs in
docs/and nav entry inmkdocs.ymlif the feature is user-facing.
- Define in
src/pgmq/messages.pywithfrom_row(cls, row)using_get_value. - Export in
pgmq/__init__.pyand__all__if part of the public API. - Document in
docs/messages.md.
- Edit only
_sql.py— never inline SQL in client files. - If placeholders change, verify
convert_sql_paramsandASYNC_SQL_MAPstill work (runtest_sql_conversion.py). - Check all
get_*_sqlhelpers that reference the modified constant.
| Method | Return type | Notes |
|---|---|---|
send |
int |
Message ID; -1 if no result |
send_batch |
List[int] |
Empty list if messages is empty |
send_topic |
int |
Routed message ID |
send_batch_topic |
List[BatchTopicResult] |
Per-queue routing results |
read |
Message | List[Message] | None |
Single if qty=1, list otherwise |
read_with_poll |
List[Message] |
Always returns a list |
pop |
Message | List[Message] | None |
Same qty semantics as read |
delete / archive |
bool |
|
delete_batch / archive_batch |
List[int] |
IDs successfully processed |
drop_queue |
bool |
|
purge |
int |
Count of purged messages |
set_vt |
Message | List[Message] | None |
Batch if msg_id is a list |
list_queues |
List[QueueRecord] |
Emits UserWarning |
metrics |
QueueMetrics |
Raises ValueError if queue not found |
metrics_all |
List[QueueMetrics] |
|
validate_queue_name |
bool |
Raises on invalid name |
validate_routing_key / validate_topic_pattern |
bool |
Returns False on invalid, no raise |
Common kwargs on almost every method: conn=None (for manual transaction composition).
Delay parameters accept int (seconds) or datetime (timestamptz). send() also accepts tz as a delay alias.
Load-testing harness using Locust. Not part of CI.
uv sync --group bench --all-extras
uv run python -m benches.runner benches/config.yamlKey files: bench.py (core bench logic), runner.py (YAML-driven multi-bench), locustfile.py, payloads/ (sample JSON sizes).
- Bump
versioninpyproject.toml. - Run
make lintandmake test. - Tag
vX.Y.Zonmain— triggersrelease.yml(PyPI publish + GitHub Release) anddocs.yml(mike deploy). - Verify https://pgmq.github.io/pgmq-py/ shows the new version.
- Only modify
src/for library changes; keep SQL in_sql.py. - Update the operation mixins when changing the public API — backend files should stay thin adapters.
- Do not add new dependencies to
[project.dependencies]without explicit user request. - Do not use bare
print()— uselog_with_context. - Do not create markdown files the user did not ask for (except
AGENTS.mdupdates). - Prefer extending existing patterns over introducing new abstractions.
- Run
make lintafter code changes; runmake testormake test-envwhen DB is available.
- PGMQ Extension: https://github.com/pgmq/pgmq
- Python Client Repo: https://github.com/pgmq/pgmq-py
- Documentation: https://pgmq.github.io/pgmq-py/
- PyPI: https://pypi.org/project/pgmq/