diff --git a/bus_alt_connection/README.rst b/bus_alt_connection/README.rst new file mode 100644 index 00000000000..25e41842933 --- /dev/null +++ b/bus_alt_connection/README.rst @@ -0,0 +1,189 @@ +.. image:: https://odoo-community.org/readme-banner-image + :target: https://odoo-community.org/get-involved?utm_source=readme + :alt: Odoo Community Association + +================== +Bus Alt Connection +================== + +.. + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + !! This file is generated by oca-gen-addon-readme !! + !! changes will be overwritten. !! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + !! source digest: sha256:ab7e1b9d5721f8cb27f93c58ed77e5034bc0099105ac6d5097f1bdc74a4e6973 + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png + :target: https://odoo-community.org/page/development-status + :alt: Beta +.. |badge2| image:: https://img.shields.io/badge/license-AGPL--3-blue.png + :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html + :alt: License: AGPL-3 +.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fserver--tools-lightgray.png?logo=github + :target: https://github.com/OCA/server-tools/tree/19.0/bus_alt_connection + :alt: OCA/server-tools +.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png + :target: https://translation.odoo-community.org/projects/server-tools-19-0/server-tools-19-0-bus_alt_connection + :alt: Translate me on Weblate +.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png + :target: https://runboat.odoo-community.org/builds?repo=OCA/server-tools&target_branch=19.0 + :alt: Try me on Runboat + +|badge1| |badge2| |badge3| |badge4| |badge5| + +This module makes it possible to use +`PgBouncer `__ as a connection pooler for +odoo. + +Why isn't odoo's connection pooling good enough? +------------------------------------------------ + +Odoo's builtin connection pooling works at process level: each Odoo +process has its own +`ConnectionPool `__, +limited to ``db_maxconn``. + +It does the job of re-using open connections available in the pool. But +it never closes these connections, `unless reaching +db_maxconn `__. + +In practice, we observe that each odoo worker will end up with up to 3 +open connection in its pool. With 10 http workers, that's up to 30 +connection continuously open just for one single instance. + +Here comes PgBouncer +-------------------- + +PgBouncer will help to limit this number of open connections, by sharing +a pool of connections at the instance level, between all workers. Odoo +workers will still have up to 3 open connections, but these will be +connections to PgBouncer, that on its side will close unnecessary +connections to pg. + +This has proven to help performances on Odoo deployments with multiple +instances. + +It allows you to define how resources should be shared, according to +your priorities, e.g. : + +- key odoo instance on host A can open up to 30 connections +- while odoo instance on host B, dedicated to reports, can open up to 10 + connections only + +And most importantly, it helps you to ensure that ``max_connections`` +will never be reached on pg server side. + +Why is this module needed? +-------------------------- + +When configuring PgBouncer, you can choose between 2 transaction pooling +modes: + +- pool_mode = session +- pool_mode = transaction + +If we choose pool_mode = session, then one server connection will be +tied to a given odoo process until its death, which is exactly what +we're trying to change. Thus, to release the server connection once the +transaction is complete, we use pool_mode = transaction. + +This works fine, except for Odoo's longpolling features that relies on +the +`LISTEN/NOTIFY `__ +mechanism from pg, which is `not +compatible `__ with that +mode. + +To be more precise, NOTIFY statements are properly transfered by +PgBouncer in that mode; only the LISTEN statement isn't (because it +needs to keep the server connection open). + +So for the unique "listening" connection per instance that requires this +statement +(`here `__), +we need odoo to connect directly to the pg server, bypassing PgBouncer. + +That's what this module implements, by overriding the relevant method of +the +`Dispatcher `__. + +**Table of contents** + +.. contents:: + :local: + +Installation +============ + +You don't need to install this module in the database(s) to enable it. + +But you need to load it server-wide: + +- By starting Odoo with ``--load=web,bus_alt_connection`` +- Or by updating its configuration file: + +.. code:: ini + + [options] + (...) + server_wide_modules = web,bus_alt_connection + +Configuration +============= + +You need to define how to connect directly to the database: + +- Either by defining environment variables: + + - ``IMDISPATCHER_DB_HOST=db-01`` + - ``IMDISPATCHER_DB_PORT=5432`` + +- Or in Odoo's configuration file: + +.. code:: ini + + [options] + (...) + imdispatcher_db_host = db-01 + imdispatcher_db_port = 5432 + +Bug Tracker +=========== + +Bugs are tracked on `GitHub Issues `_. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us to smash it by providing a detailed and welcomed +`feedback `_. + +Do not contact contributors directly about support or help with technical issues. + +Credits +======= + +Authors +------- + +* Trobz + +Contributors +------------ + +- Nils Hamerlinck + +Maintainers +----------- + +This module is maintained by the OCA. + +.. image:: https://odoo-community.org/logo.png + :alt: Odoo Community Association + :target: https://odoo-community.org + +OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use. + +This module is part of the `OCA/server-tools `_ project on GitHub. + +You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute. diff --git a/bus_alt_connection/__init__.py b/bus_alt_connection/__init__.py new file mode 100644 index 00000000000..1bcfe0e2f29 --- /dev/null +++ b/bus_alt_connection/__init__.py @@ -0,0 +1,3 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). +from . import models diff --git a/bus_alt_connection/__manifest__.py b/bus_alt_connection/__manifest__.py new file mode 100644 index 00000000000..2a3496e86a4 --- /dev/null +++ b/bus_alt_connection/__manifest__.py @@ -0,0 +1,15 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). + +{ + "name": "Bus Alt Connection", + "summary": "Needed when using PgBouncer as a connection pooler", + "version": "19.0.1.0.0", + "author": "Trobz,Odoo Community Association (OCA)", + "website": "https://github.com/OCA/server-tools", + "category": "Extra Tools", + "license": "AGPL-3", + "depends": ["bus"], + "installable": True, + "auto_install": False, +} diff --git a/bus_alt_connection/i18n/bus_alt_connection.pot b/bus_alt_connection/i18n/bus_alt_connection.pot new file mode 100644 index 00000000000..4d8b20f912f --- /dev/null +++ b/bus_alt_connection/i18n/bus_alt_connection.pot @@ -0,0 +1,13 @@ +# Translation of Odoo Server. +# This file contains the translation of the following modules: +# +msgid "" +msgstr "" +"Project-Id-Version: Odoo Server 14.0\n" +"Report-Msgid-Bugs-To: \n" +"Last-Translator: \n" +"Language-Team: \n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: \n" +"Plural-Forms: \n" diff --git a/bus_alt_connection/models/__init__.py b/bus_alt_connection/models/__init__.py new file mode 100644 index 00000000000..3dc9bd99358 --- /dev/null +++ b/bus_alt_connection/models/__init__.py @@ -0,0 +1,3 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). +from . import bus diff --git a/bus_alt_connection/models/bus.py b/bus_alt_connection/models/bus.py new file mode 100644 index 00000000000..f3e77f21424 --- /dev/null +++ b/bus_alt_connection/models/bus.py @@ -0,0 +1,67 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). + +import json +import logging +import os +import selectors + +import psycopg2 + +import odoo +from odoo.tools import config + +import odoo.addons.bus.models.bus +from odoo.addons.bus.models.bus import TIMEOUT, hashable, stop_event + +_logger = logging.getLogger(__name__) + + +def _connection_info_for(db_name): + db_or_uri, connection_info = odoo.sql_db.connection_info_for(db_name) + + for p in ("host", "port"): + cfg = os.environ.get(f"ODOO_IMDISPATCHER_DB_{p.upper()}") or config.get( + "imdispatcher_db_" + p + ) + if cfg: + connection_info[p] = cfg + return connection_info + + +class ImDispatch(odoo.addons.bus.models.bus.ImDispatch): + def loop(self): + """Dispatch postgres notifications to the relevant + polling threads/greenlets""" + connection_info = _connection_info_for("postgres") + _logger.info( + "Bus.loop listen imbus on db postgres (via %(host)s:%(port)s)", + connection_info, + ) + conn = psycopg2.connect(**connection_info) + with conn.cursor() as cr, selectors.DefaultSelector() as sel: + cr.execute("listen imbus") + conn.commit() + sel.register(conn, selectors.EVENT_READ) + while not stop_event.is_set(): + if sel.select(TIMEOUT): + conn.poll() + channels = [] + while conn.notifies: + channels.extend(json.loads(conn.notifies.pop().payload)) + # relay notifications to websockets that have + # subscribed to the corresponding channels. + websockets = set() + for channel in channels: + websockets.update( + self._channels_to_ws.get(hashable(channel), []) + ) + for websocket in websockets: + websocket.trigger_notification_dispatching() + + +odoo.addons.bus.models.bus.ImDispatch = ImDispatch +dispatch = ImDispatch() +odoo.addons.bus.models.bus.dispatch = dispatch +odoo.addons.bus.models.ir_websocket.dispatch = dispatch +odoo.addons.bus.websocket.dispatch = dispatch diff --git a/bus_alt_connection/pyproject.toml b/bus_alt_connection/pyproject.toml new file mode 100644 index 00000000000..4231d0cccb3 --- /dev/null +++ b/bus_alt_connection/pyproject.toml @@ -0,0 +1,3 @@ +[build-system] +requires = ["whool"] +build-backend = "whool.buildapi" diff --git a/bus_alt_connection/readme/CONFIGURE.md b/bus_alt_connection/readme/CONFIGURE.md new file mode 100644 index 00000000000..06479bff9af --- /dev/null +++ b/bus_alt_connection/readme/CONFIGURE.md @@ -0,0 +1,15 @@ +You need to define how to connect directly to the database: + +- Either by defining environment variables: + + > - `IMDISPATCHER_DB_HOST=db-01` + > - `IMDISPATCHER_DB_PORT=5432` + +- Or in Odoo's configuration file: + +``` ini +[options] +(...) +imdispatcher_db_host = db-01 +imdispatcher_db_port = 5432 +``` diff --git a/bus_alt_connection/readme/CONTRIBUTORS.md b/bus_alt_connection/readme/CONTRIBUTORS.md new file mode 100644 index 00000000000..2f9ef686333 --- /dev/null +++ b/bus_alt_connection/readme/CONTRIBUTORS.md @@ -0,0 +1 @@ +- Nils Hamerlinck \<\> diff --git a/bus_alt_connection/readme/DESCRIPTION.md b/bus_alt_connection/readme/DESCRIPTION.md new file mode 100644 index 00000000000..1784d3fb88e --- /dev/null +++ b/bus_alt_connection/readme/DESCRIPTION.md @@ -0,0 +1,71 @@ +This module makes it possible to use +[PgBouncer](https://pgbouncer.github.io/) as a connection pooler for +odoo. + +## Why isn't odoo's connection pooling good enough? + +Odoo's builtin connection pooling works at process level: each Odoo +process has its own +[ConnectionPool](https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525), +limited to `db_maxconn`. + +It does the job of re-using open connections available in the pool. But +it never closes these connections, [unless reaching +db_maxconn](https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593). + +In practice, we observe that each odoo worker will end up with up to 3 +open connection in its pool. With 10 http workers, that's up to 30 +connection continuously open just for one single instance. + +## Here comes PgBouncer + +PgBouncer will help to limit this number of open connections, by sharing +a pool of connections at the instance level, between all workers. Odoo +workers will still have up to 3 open connections, but these will be +connections to PgBouncer, that on its side will close unnecessary +connections to pg. + +This has proven to help performances on Odoo deployments with multiple +instances. + +It allows you to define how resources should be shared, according to +your priorities, e.g. : + +- key odoo instance on host A can open up to 30 connections +- while odoo instance on host B, dedicated to reports, can open up to 10 + connections only + +And most importantly, it helps you to ensure that `max_connections` will +never be reached on pg server side. + +## Why is this module needed? + +When configuring PgBouncer, you can choose between 2 transaction pooling +modes: + +- pool_mode = session +- pool_mode = transaction + +If we choose pool_mode = session, then one server connection will be +tied to a given odoo process until its death, which is exactly what +we're trying to change. Thus, to release the server connection once the +transaction is complete, we use pool_mode = transaction. + +This works fine, except for Odoo's longpolling features that relies on +the +[LISTEN/NOTIFY](https://www.postgresql.org/docs/9.6/static/sql-notify.html) +mechanism from pg, which is [not +compatible](https://wiki.postgresql.org/wiki/PgBouncer) with that mode. + +To be more precise, NOTIFY statements are properly transfered by +PgBouncer in that mode; only the LISTEN statement isn't (because it +needs to keep the server connection open). + +So for the unique "listening" connection per instance that requires this +statement +([here](https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166)), +we need odoo to connect directly to the pg server, bypassing PgBouncer. + +That's what this module implements, by overriding the relevant method of +the +[Dispatcher](https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105). diff --git a/bus_alt_connection/readme/INSTALL.md b/bus_alt_connection/readme/INSTALL.md new file mode 100644 index 00000000000..f5532609239 --- /dev/null +++ b/bus_alt_connection/readme/INSTALL.md @@ -0,0 +1,12 @@ +You don't need to install this module in the database(s) to enable it. + +But you need to load it server-wide: + +- By starting Odoo with `--load=web,bus_alt_connection` +- Or by updating its configuration file: + +``` ini +[options] +(...) +server_wide_modules = web,bus_alt_connection +``` diff --git a/bus_alt_connection/static/description/icon.png b/bus_alt_connection/static/description/icon.png new file mode 100644 index 00000000000..3a0328b516c Binary files /dev/null and b/bus_alt_connection/static/description/icon.png differ diff --git a/bus_alt_connection/static/description/index.html b/bus_alt_connection/static/description/index.html new file mode 100644 index 00000000000..010b37e2784 --- /dev/null +++ b/bus_alt_connection/static/description/index.html @@ -0,0 +1,525 @@ + + + + + +README.rst + + + +
+ + + +Odoo Community Association + +
+

Bus Alt Connection

+ +

Beta License: AGPL-3 OCA/server-tools Translate me on Weblate Try me on Runboat

+

This module makes it possible to use +PgBouncer as a connection pooler for +odoo.

+
+

Why isn’t odoo’s connection pooling good enough?

+

Odoo’s builtin connection pooling works at process level: each Odoo +process has its own +ConnectionPool, +limited to db_maxconn.

+

It does the job of re-using open connections available in the pool. But +it never closes these connections, unless reaching +db_maxconn.

+

In practice, we observe that each odoo worker will end up with up to 3 +open connection in its pool. With 10 http workers, that’s up to 30 +connection continuously open just for one single instance.

+
+
+

Here comes PgBouncer

+

PgBouncer will help to limit this number of open connections, by sharing +a pool of connections at the instance level, between all workers. Odoo +workers will still have up to 3 open connections, but these will be +connections to PgBouncer, that on its side will close unnecessary +connections to pg.

+

This has proven to help performances on Odoo deployments with multiple +instances.

+

It allows you to define how resources should be shared, according to +your priorities, e.g. :

+
    +
  • key odoo instance on host A can open up to 30 connections
    • +
  • while odoo instance on host B, dedicated to reports, can open up to 10 +connections only
    • +
+

And most importantly, it helps you to ensure that max_connections +will never be reached on pg server side.

+
+
+

Why is this module needed?

+

When configuring PgBouncer, you can choose between 2 transaction pooling +modes:

+
    +
  • pool_mode = session
    • +
  • pool_mode = transaction
    • +
+

If we choose pool_mode = session, then one server connection will be +tied to a given odoo process until its death, which is exactly what +we’re trying to change. Thus, to release the server connection once the +transaction is complete, we use pool_mode = transaction.

+

This works fine, except for Odoo’s longpolling features that relies on +the +LISTEN/NOTIFY +mechanism from pg, which is not +compatible with that +mode.

+

To be more precise, NOTIFY statements are properly transfered by +PgBouncer in that mode; only the LISTEN statement isn’t (because it +needs to keep the server connection open).

+

So for the unique “listening” connection per instance that requires this +statement +(here), +we need odoo to connect directly to the pg server, bypassing PgBouncer.

+

That’s what this module implements, by overriding the relevant method of +the +Dispatcher.

+

Table of contents

+ +
+

Installation

+

You don’t need to install this module in the database(s) to enable it.

+

But you need to load it server-wide:

+
    +
  • By starting Odoo with --load=web,bus_alt_connection
    • +
  • Or by updating its configuration file:
    • +
+
+[options]
+(...)
+server_wide_modules = web,bus_alt_connection
+
+
+
+

Configuration

+

You need to define how to connect directly to the database:

+
    +

  • Either by defining environment variables:

    +
    +
      +
    • IMDISPATCHER_DB_HOST=db-01
      • +
    • IMDISPATCHER_DB_PORT=5432
      • +
    +
    +
    • +

  • Or in Odoo’s configuration file:

    +
    • +
+
+[options]
+(...)
+imdispatcher_db_host = db-01
+imdispatcher_db_port = 5432
+
+
+
+

Bug Tracker

+

Bugs are tracked on GitHub Issues. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us to smash it by providing a detailed and welcomed +feedback.

+

Do not contact contributors directly about support or help with technical issues.

+
+ +
+
+

Authors

+
    +
  • Trobz
    • +
+
+
+

Contributors

+ +
+
+

Maintainers

+

This module is maintained by the OCA.

+ +Odoo Community Association + +

OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use.

+

This module is part of the OCA/server-tools project on GitHub.

+

You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

+
+
+
+ +