feat: support storing, retrieving and refreshing encrypted secrets at account or asset level

documentation/changelog.rst

@@ -7,6 +7,8 @@ FlexMeasures Changelog
 v1.0.0 | July XX, 2026
 ============================
 
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+
 New features
 -------------
 * In the UI, asset and sensor lists can be filtered by ID prefix through API-backed search fields [see `PR #2231 <https://www.github.com/FlexMeasures/flexmeasures/pull/2231>`_]
@@ -15,11 +17,11 @@ New features
 * Let storage scheduling infer missing ``power-capacity`` from directional device capacities before falling back to site capacity, and default the missing opposite capacity to zero when only a non-zero ``consumption-capacity`` or ``production-capacity`` is configured [see `PR #2222 <https://www.github.com/FlexMeasures/flexmeasures/pull/2222>`_]
 * CLI support for adding/editing account attributes [see `PR #2242 <https://www.github.com/FlexMeasures/flexmeasures/pull/2242>`_]
 
-
 Infrastructure / Support
 ----------------------
 * Upgraded dependencies [see `PR #1485 <https://www.github.com/FlexMeasures/flexmeasures/pull/1485>`_ and `PR #2215 <https://www.github.com/FlexMeasures/flexmeasures/pull/2215>`_]
 * Prepare the ``device_scheduler`` to deal with commitments per device group [see `PR #1934 <https://www.github.com/FlexMeasures/flexmeasures/pull/1934>`_]
+* Support storing encrypted connection secrets on organisations and assets, including utility functions, encryption key configuration, CLI commands to set and delete secrets, and UI tables that show stored secret names and optional expiration times without exposing their values [see `PR #2236 <https://www.github.com/FlexMeasures/flexmeasures/pull/2236>`_]
 * Documentation section on the modelling choice for recording measurements, forecasts and schedules under one or multiple sensors [see `PR #2217 <https://www.github.com/FlexMeasures/flexmeasures/pull/2217>`_]
 
 Bugfixes

documentation/cli/change_log.rst

@@ -4,6 +4,12 @@
 FlexMeasures CLI Changelog
 **********************
 
+since v1.0.0 | July XX, 2026
+=================================
+
+* Add ``flexmeasures edit secret`` to store an encrypted secret on an account or asset.
+* Add ``flexmeasures delete secret`` to remove an encrypted secret from an account or asset.
+
 since v0.33.0 | June 01, 2026
 =================================
 

documentation/cli/commands.rst

@@ -68,12 +68,12 @@ of which some are referred to in this documentation.
 
 ================================================= =======================================
 ``flexmeasures edit attribute``                   Edit (or add) an asset attribute or sensor attribute.
+``flexmeasures edit secret``                      Edit (or add) an encrypted secret on an account or asset.
 ``flexmeasures edit resample-data``               Assign a new event resolution to an existing sensor and resample its data accordingly.
 ``flexmeasures edit transfer-parenthood``         (Re)assign parent assets.
 ``flexmeasures edit transfer-ownership``          Transfer the ownership of an asset and its children to a different account.
 ================================================= =======================================
 
-
 ``delete`` - Delete data
 --------------
 
@@ -83,6 +83,7 @@ of which some are referred to in this documentation.
 ``flexmeasures delete account``                   Delete a tenant account & also their users (with assets and power measurements).
 ``flexmeasures delete user``                      Delete a user & also their assets and power measurements.
 ``flexmeasures delete asset``                     Delete an asset & also its sensors and data.
+``flexmeasures delete secret``                    Delete an encrypted secret from an account or asset.
 ``flexmeasures delete sensor``                    Delete a sensor and all beliefs about it.
 ``flexmeasures delete beliefs``                   Delete time series data (beliefs).
 ``flexmeasures delete measurements``              Delete measurements (with horizon <= 0).

documentation/configuration.rst

@@ -508,6 +508,25 @@ Extra password salt (a.k.a. pepper)
 
 Default: ``None`` (falls back to ``SECRET_KEY``\ )
 
+.. _flexmeasures_secrets_encryption_keys:
+
+FLEXMEASURES_SECRETS_ENCRYPTION_KEYS
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Keyring (dictionary with key IDs mapped to key material), used to encrypt and decrypt
+connection secrets stored for accounts and assets. This follows the same shape
+as ``SECURITY_TOTP_SECRETS``, for example
+``{"1": "old-secret", "2": "current-secret"}``.
+
+Newly encrypted values use the highest numeric key ID in this dictionary, or
+the last key ID in lexical order if the IDs are not numeric. Keep previous keys
+in this dictionary until all stored secrets have been re-encrypted with the
+current key.
+
+This setting must be configured before connection secrets can be stored or decrypted.
+
+Default: ``None``
+
 SECURITY_TOKEN_AUTHENTICATION_HEADER
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

documentation/dev/connection-secrets.rst

@@ -0,0 +1,71 @@
+.. _connection_secrets_dev:
+
+Connection secrets
+==================
+
+If you make a secure connection to an external platform, FlexMeasures can store credentials,
+API keys and tokens in the ``secrets`` JSON field of the account or asset that owns the connection.
+Each secret value is encrypted separately and stored together with metadata such as its encryption
+key ID and timestamps. Developers normally do not need to read or modify this
+JSON structure directly.
+
+For implementation examples, token lifecycle strategies and manually seeding a
+credential through the CLI, see :ref:`storing_connection_secrets`. 
+
+The encrypted values are protected by
+``FLEXMEASURES_SECRETS_ENCRYPTION_KEYS``, see :ref:`flexmeasures_secrets_encryption_keys` 
+ This setting accepts arbitrary non-empty strings, which FlexMeasures derives into Fernet-compatible keys.
+Hosts must configure this keyring before secrets can be stored
+- FlexMeasures will print a warning if it is not set and hints how to initialize it.
+
+
+Recommended practices
+---------------------
+
+Write-only API and UI
+^^^^^^^^^^^^^^^^^^^^^
+
+Treat secrets as write-only in API and UI flows. Accept new or replacement
+values, but never return plaintext secrets. Responses should contain only
+redacted information such as whether a value is set and when it expires.
+
+For administrator-level maintenance, use ``flexmeasures edit secret`` to store
+or replace one account or asset secret and ``flexmeasures delete secret`` to
+remove one. Use ``--secret`` for dot-separated paths or ``--secret-path`` once
+or twice. Prefer the edit command's ``--prompt`` option so secret values do not
+enter shell history.
+
+
+Use the secret utilities
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+FlexMeasures provides a complete utility API in :mod:`flexmeasures.utils.secrets_utils`.
+
+Use :func:`flexmeasures.utils.secrets_utils.set_secret` for individual
+credentials. For token refresh flows, prefer
+:class:`flexmeasures.utils.secrets_utils.TokenRefreshResult` together with
+:func:`flexmeasures.utils.secrets_utils.apply_token_refresh_result`. They safely
+handle replacement tokens, rotated refresh tokens and providers that only
+extend the expiry of an existing token.
+
+Use :func:`flexmeasures.utils.secrets_utils.get_secret_overview` to build safe
+secret listings with their paths and optional expiration times, without
+exposing encrypted values or unrelated metadata.
+When ``expires_at`` is stored without a timezone offset, FlexMeasures treats it
+as UTC when rendering the overview.
+
+Store secrets at one or two path levels only: a top-level namespace such as a
+provider name, optionally followed by a secret name within that namespace.
+String paths therefore look like ``provider`` or ``provider.refresh_token``.
+If a secret name itself contains a dot, pass the path as a tuple or list of
+parts, or use ``--secret-path`` in the CLI.
+
+
+Refresh early in multi-worker deployments
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Store reusable access tokens and their expiry metadata in ``secrets`` so all
+workers share the same token state. Refresh with time to spare before expiry to
+allow for clock differences, retries and concurrent workers. In high-traffic
+integrations, use database locking so only one worker performs a refresh. See
+the refresh-leeway example in :ref:`initiating_connection_tokens`.

documentation/index.rst

@@ -244,6 +244,7 @@ In :ref:`getting_started`, we have some helpful tips how to dive into this docum
     plugin/introduction
     plugin/showcase
     plugin/customisation
+    plugin/storing-platform-secrets
 
 
 .. toctree::
@@ -258,6 +259,7 @@ In :ref:`getting_started`, we have some helpful tips how to dive into this docum
     dev/auth
     dev/dependency-management
     dev/api
+    dev/connection-secrets
     dev/automated-deploy-via-GHActions
 
 .. autosummary::
@@ -288,4 +290,3 @@ In :ref:`getting_started`, we have some helpful tips how to dive into this docum
 .. * :ref:`genindex`
 .. * :ref:`modindex`
 .. * :ref:`search`
-

documentation/plugin/introduction.rst

@@ -32,4 +32,6 @@ To hit the ground running with that approach, we provide a `CookieCutter templat
 It also includes a few Blueprint examples and best practices.
 
 
-Continue reading the :ref:`plugin_showcase` or possibilities to do :ref:`plugin_customization`.
+Continue reading the :ref:`plugin_showcase`. Continue to see more possibilities to do :ref:`plugin_customization`.
+And if your plugin connects to a 3rd-party platform, read about support for :ref:`storing_connection_secrets`.
+

documentation/plugin/storing-platform-secrets.rst

@@ -0,0 +1,168 @@
+.. _storing_connection_secrets:
+
+Storing connection secrets
+==========================
+
+Plugins that connect FlexMeasures accounts or assets to external platforms often
+need to store credentials, refresh tokens, access tokens or connection-specific
+passwords. Store such values in the ``secrets`` JSON field of the relevant
+account or asset, rather than in ``attributes`` or plugin configuration files.
+
+Secrets live in a dictionary and support one or two path levels. For example:
+
+.. code-block:: json
+
+    {
+        "3rdparty-platform": {
+            "refresh_token": "encrypted-value",
+            "access_token": "encrypted-value"
+        }
+    }
+
+(where "encrypted-value" is not the actual string you pass in, FlexMeasures handles the encryption seamlessly)
+
+The ``secrets`` field is intended to be write-only from API and UI flows: users
+can provide or replace secret values, but normal responses should return only
+redacted metadata such as whether a value is set and when it expires. Trusted
+server-side plugin code can decrypt and use the value when it performs work for
+the account or asset.
+
+Use ``flexmeasures.utils.secrets_utils`` for secret handling:
+
+.. code-block:: python
+
+    from flexmeasures.utils.secrets_utils import (
+        SecretsEncryptor,
+        get_secret,
+        redact_secrets,
+        set_secret,
+    )
+
+    encryptor = SecretsEncryptor.from_current_app()
+
+    my_account.secrets = set_secret(
+        my_account.secrets,
+        "3rdparty-platform.refresh_token",
+        refresh_token,
+        encryptor=encryptor,
+        metadata={"expires_at": refresh_token_expires_at.isoformat()},
+    )
+
+    refresh_token = get_secret(
+        my_account.secrets,
+        "3rdparty-platform.refresh_token",
+        encryptor=encryptor,
+    )
+
+    response_payload = redact_secrets(my_account.secrets)
+
+The metadata string ``expires_at`` (see example above) is useful to add, as it will be used by the token handling and also shown in the UI.
+FlexMeasures will also handle ``created-at`` and ``updated-at`` metadata automatically.
+
+More details and best practices for storing connection secrets are in the :ref:`connection_secrets_dev` section.
+
+
+Token lifecycle strategies
+-----------------------------
+
+External platforms do not all require the same interactions to maintain a connection.
+Initial login, token refresh and token expiry can all work rather differently.
+You could say that they implement different "token lifecycle strategies".
+
+That's why our advice is to keep the provider-specific HTTP calls in your plugin,
+and use FlexMeasures utilities only for encryption, redaction and updating stored token state.
+
+The following token lifecycle strategies are supported by
+``TokenRefreshResult`` and ``apply_token_refresh_result``:
+
+* A refresh operation returns a new access token: set
+  ``TokenRefreshResult.access_token`` and ``access_token_expires_at``.
+* A refresh operation rotates the refresh token: set
+  ``TokenRefreshResult.refresh_token`` and ``refresh_token_expires_at``.
+* A refresh operation extends an existing access token without returning a new token:
+  leave ``access_token`` as ``None`` and set ``access_token_expires_at``.
+* Access tokens are minted separately from refresh-token rotation: call
+  ``apply_token_refresh_result`` once for the refreshed long-lived credential
+  and again when a short-lived access token is minted.
+
+Your task is to translate the HTTP response from the platform provider into a
+``TokenRefreshResult`` and let FlexMeasures update the encrypted JSON state.
+Let's look at an example:
+
+.. code-block:: python
+
+    from flexmeasures.utils.secrets_utils import (
+        SecretsEncryptor,
+        TokenRefreshResult,
+        apply_token_refresh_result,
+        get_secret,
+    )
+
+    encryptor = SecretsEncryptor.from_current_app()
+
+    refresh_token = get_secret(
+        my_account.secrets,
+        "3rdparty-platform.refresh_token",
+        encryptor=encryptor,
+    )
+
+    # this function would be written by you
+    token_response = refresh_with_external_platform(refresh_token)
+
+    # here you translate the response - consult the platform docs
+    my_account.secrets = apply_token_refresh_result(
+        my_account.secrets,
+        "3rdparty-platform",
+        TokenRefreshResult(
+            access_token=token_response.get("access_token"),
+            refresh_token=token_response.get("refresh_token"),
+            access_token_expires_at=token_response.get("access_token_expires_at"),
+            refresh_token_expires_at=token_response.get("refresh_token_expires_at"),
+            token_type=token_response.get("token_type"),
+            metadata={"scope": token_response.get("scope")},
+        ),
+        encryptor=encryptor,
+    )
+
+This varies by platform - for instance, if a platform only extends the lifetime
+of the existing access token (instead of returning a new one), keep
+``access_token`` set to ``None`` and provide the new ``access_token_expires_at``.
+The existing encrypted token is kept, and only its metadata is updated.
+
+Your plugin should also decide when to request a refreshed access token, e.g.:
+
+.. code-block:: python
+
+    3RDPARTY_PLATFORM_TOKEN_LEEWAY = timedelta(seconds=120)
+    # you might get this info with `secret_utils.get_secret()`
+    if current_access_token_expires_at > now + 3RDPARTY_PLATFORM_TOKEN_LEEWAY:
+        return access_token
+    response = send_request_to_3rdparty_platform()
+    access_token = response.text
+    refresh_account.secrets = apply_token_refresh_result(...)
+
+
+
+.. _initiating_connection_tokens:
+
+Initiating tokens (before app startup)
+-----------------------------------------
+
+Most providers will require the true credentials only in the first interaction:
+For example: username & password to get the access & refresh token.
+From then on, the refresh token helps to get by (as long as it does not expire).
+
+In your plugin, you can write a CLI command to perform this login, get your first
+refresh token and save it as a secret (see `set_secret()` and `apply_token_refresh_result()`
+in utils/secrets_utils.py).
+The token lifecycle strategy (see above) will depend on the platform you connect to.
+
+Also advisable: if no current token is in the database, let your plugin code fail
+explicitly and advise the user to call your login CLI command.
+
+
+Alternatively, you can manually store a known credential: Use ``flexmeasures edit secret`` with
+an account or asset ID, plus either ``--secret`` or one or two ``--secret-path``
+options, and either ``--value`` or ``--prompt`` (to paste the secret instead of typing it).
+Use ``--metadata`` for non-secret JSON metadata such as expiry
+timestamps.

documentation/views/asset-data.rst

@@ -162,6 +162,8 @@ Properties page
 ---------------
 
 The properties page allows you to view and edit the properties of the asset.
+It lists stored secret names without exposing their values and shows a
+human-readable expiration time when one is available.
 
 You can also delete the asset by clicking on the "Delete this asset" button.
 
@@ -204,4 +206,3 @@ This is how the audit log looks for the history of actions taken on an asset:
 ..    :scale: 40%
 
 |
-

flexmeasures/app.py

@@ -46,11 +46,10 @@ def create(  # noqa C901
         get_flexmeasures_env,
     )
     from flexmeasures.utils.app_utils import (
-        set_secret_key,
-        set_totp_secrets,
         init_sentry,
     )
     from flexmeasures.utils.error_utils import add_basic_error_handlers
+    from flexmeasures.utils.secrets_utils import set_secret_key, set_totp_secrets
 
     configure_logging()  # do this first, see https://flask.palletsprojects.com/en/2.0.x/logging
     cfg_location = find_flexmeasures_cfg()  # Find flexmeasures.cfg location