From 6217825fba090dbcd5540286a84796fc56d06fa7 Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 4 Apr 2025 13:07:38 -0700 Subject: [PATCH 01/51] docs: restructuring introduction and adding a begin section --- content/shared/v3-core-plugins/_index.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index e4661f43bb..3faec38ab1 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -1,5 +1,6 @@ -Use the InfluxDB 3 Processing engine to run Python code directly in your -{{% product-name %}} database to automatically process data and respond to database events. +# Get Started with the Processing Engine and Plugins + +Use the Processing Engine in InfluxDB 3 to extend database functionality with custom Python code directly in your {{% product-name %}}. The Processing Engine runs Python plugins in response to database events like data writes, scheduled tasks, or HTTP requests. This guide walks you through setting up the engine, writing your first plugin, and triggering it. The Processing engine is an embedded Python VM that runs inside your InfluxDB 3 database and lets you: @@ -10,6 +11,16 @@ The Processing engine is an embedded Python VM that runs inside your InfluxDB 3 Learn how to create, configure, run, and extend Python plugins that execute when specific events occur. +## Before you begin + +Ensure you have: +- A working influxDB 3 Core instance +- Access to command line +- Python installed if you're writing your own plugin +- Basic knowledge of the InfluxDB CLI + +Learn how to create, configure, run, and extend Python plugins that execute when specific events occur. + 1. [Set up the Processing engine](#set-up-the-processing-engine) 2. [Add a Processing engine plugin](#add-a-processing-engine-plugin) - [Get example plugins](#get-example-plugins) From 524bcf91283441d2505f1b5f891b22d2c8218303 Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 4 Apr 2025 14:21:00 -0700 Subject: [PATCH 02/51] docs: updating setp up section and adding more direction --- content/shared/v3-core-plugins/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index 3faec38ab1..712a968dc3 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -19,7 +19,7 @@ Ensure you have: - Python installed if you're writing your own plugin - Basic knowledge of the InfluxDB CLI -Learn how to create, configure, run, and extend Python plugins that execute when specific events occur. +Once you have all the prerequisites in place, follow these steps to implement the Processing engine for your data automation needs. 1. [Set up the Processing engine](#set-up-the-processing-engine) 2. [Add a Processing engine plugin](#add-a-processing-engine-plugin) From c030f68ad9b5c8a403844206f868801d317dca7f Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 4 Apr 2025 15:02:58 -0700 Subject: [PATCH 03/51] docs: revising introduction and restructuring the first part of adding a processing engine plugin section --- content/shared/v3-core-plugins/_index.md | 65 ++++++++++++++---------- 1 file changed, 38 insertions(+), 27 deletions(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index 712a968dc3..507579bf80 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -1,20 +1,23 @@ # Get Started with the Processing Engine and Plugins -Use the Processing Engine in InfluxDB 3 to extend database functionality with custom Python code directly in your {{% product-name %}}. The Processing Engine runs Python plugins in response to database events like data writes, scheduled tasks, or HTTP requests. This guide walks you through setting up the engine, writing your first plugin, and triggering it. +Extend InfluxDB 3 with custom Python code that responds to database events. The Processing Engine lets you automate workflows, transform data, and create API endpoints directly within your {{% product-name %}}. -The Processing engine is an embedded Python VM that runs inside your InfluxDB 3 database and lets you: +## What is the Processing Engine? -- Process data as it's written to the database -- Run code on a schedule -- Create API endpoints that execute Python code -- Maintain state between executions with an in-memory cache +The Processing Engine is an embedded Python virtual machine that runs inside your InfluxDB 3 database. It executes Python code in response to: -Learn how to create, configure, run, and extend Python plugins that execute when specific events occur. +- **Data writes** - Process and transform data as it enters the database +- **Scheduled events** - Run code at specific intervals or times +- **HTTP requests** - Create custom API endpoints that execute your code + +The engine maintains state between executions using an in-memory cache, allowing you to build stateful applications directly in your database. + +This guide shows you how to set up the Processing Engine, create your first plugin, and configure triggers that execute your code when specific events occur. ## Before you begin Ensure you have: -- A working influxDB 3 Core instance +- A working InfluxDB 3 Core instance - Access to command line - Python installed if you're writing your own plugin - Basic knowledge of the InfluxDB CLI @@ -23,7 +26,7 @@ Once you have all the prerequisites in place, follow these steps to implement th 1. [Set up the Processing engine](#set-up-the-processing-engine) 2. [Add a Processing engine plugin](#add-a-processing-engine-plugin) - - [Get example plugins](#get-example-plugins) + - [Clone and use an example plugin](#Clone-and-use-an-example-plugin) - [Create a plugin](#create-a-plugin) 3. [Create a trigger to run a plugin](#create-a-trigger-to-run-a-plugin) - [Create a trigger for data writes](#create-a-trigger-for-data-writes) @@ -38,7 +41,7 @@ Once you have all the prerequisites in place, follow these steps to implement th ## Set up the Processing engine -To enable the Processing engine, start your InfluxDB server with the `--plugin-dir` option: +To enable the Processing engine, start your InfluxDB server with the `--plugin-dir` flag to specify where your plugin files are stored. ```bash influxdb3 serve \ @@ -47,36 +50,44 @@ influxdb3 serve \ --plugin-dir /path/to/plugins ``` +Replace: +- `` with a unique identifier for your instance +- `` with the type of object store (e.g., file, memory, s3) +- /absolute/path/to/plugins with the path to your plugin directory + +The plugin directory must exist before you start InfluxDB. + ## Add a Processing engine plugin -A plugin is a Python file that contains a specific function signature that corresponds to a trigger type. -Plugins: +A plugin is a Python file that contains a specific function signature that corresponds to a trigger type. InfluxData maintains a repository of contributed plugins that you can use as-is or as a starting point for your own plugin. + +### Clone and use an example plugin + +1. Clone the repo: +```bash +git clone https://github.com/influxdata/influxdb3_plugins.git +``` +You can find example plugins to use here: [influxdb3_plugins repository](https://github.com/influxdata/influxdb3_plugins). + +2. Copy a plugin into your configured plugin directory +```bash +cp influxdb3_plugins/examples/ /path/to/plugins/ +``` +3. Restart InfluxDB if it's already running. + +Plugins have various functions such as: - Receive plugin-specific arguments (such as written data, call time, or an HTTP request) - Can receive keyword arguments (as `args`) from _trigger arguments_ - Can access the `influxdb3_local` shared API for writing, querying, and managing state -Get started using example plugins or create your own: - -- [Get example plugins](#get-example-plugins) +If you would like to create your own plugin you can follow: - [Create a plugin](#create-a-plugin) -### Get example plugins - -InfluxData maintains a repository of contributed plugins that you can use as-is or as a starting point for your own plugin. - #### From local files You can copy example plugins from the [influxdb3_plugins repository](https://github.com/influxdata/influxdb3_plugins) to your local plugin directory: -```bash -# Clone the repository -git clone https://github.com/influxdata/influxdb3_plugins.git - -# Copy example plugins to your plugin directory -cp -r influxdb3_plugins/examples/wal_plugin/* /path/to/plugins/ -``` - #### Directly from GitHub You can use plugins directly from GitHub without downloading them first by using the `gh:` prefix in the plugin filename: From 2396ba1fc2619b57c3c57559c2169d7e81783d02 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 7 Apr 2025 10:51:03 -0700 Subject: [PATCH 04/51] docs: updating Add a Processing engine plugin section --- content/shared/v3-core-plugins/_index.md | 66 +++++++++++------------- 1 file changed, 29 insertions(+), 37 deletions(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index 507579bf80..b6ef5bbd83 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -26,7 +26,7 @@ Once you have all the prerequisites in place, follow these steps to implement th 1. [Set up the Processing engine](#set-up-the-processing-engine) 2. [Add a Processing engine plugin](#add-a-processing-engine-plugin) - - [Clone and use an example plugin](#Clone-and-use-an-example-plugin) + - [Use example plugins](#use-example-plugins) - [Create a plugin](#create-a-plugin) 3. [Create a trigger to run a plugin](#create-a-trigger-to-run-a-plugin) - [Create a trigger for data writes](#create-a-trigger-for-data-writes) @@ -61,39 +61,39 @@ The plugin directory must exist before you start InfluxDB. A plugin is a Python file that contains a specific function signature that corresponds to a trigger type. InfluxData maintains a repository of contributed plugins that you can use as-is or as a starting point for your own plugin. -### Clone and use an example plugin +You have two main options for adding plugins to your InfluxDB instance: -1. Clone the repo: -```bash -git clone https://github.com/influxdata/influxdb3_plugins.git -``` -You can find example plugins to use here: [influxdb3_plugins repository](https://github.com/influxdata/influxdb3_plugins). +1. [Use example plugins](#use-example-plugins) - Quickest way to get started +2. [Create your own plugin](#create-a-plugin) - For custom functionality -2. Copy a plugin into your configured plugin directory -```bash -cp influxdb3_plugins/examples/ /path/to/plugins/ -``` -3. Restart InfluxDB if it's already running. +### Use example plugins -Plugins have various functions such as: +The InfluxData team maintains a repository of example plugins you can use immediately: -- Receive plugin-specific arguments (such as written data, call time, or an HTTP request) -- Can receive keyword arguments (as `args`) from _trigger arguments_ -- Can access the `influxdb3_local` shared API for writing, querying, and managing state +1. **Browse available plugins**: Visit the [influxdb3_plugins repository](https://github.com/influxdata/influxdb3_plugins) to find examples for: + - **Data transformation**: Process and transform incoming data + - **Alerting**: Send notifications based on data thresholds + - **Aggregation**: Calculate statistics on time series data + - **Integration**: Connect to external services and APIs + - **System monitoring**: Track resource usage and health metrics -If you would like to create your own plugin you can follow: -- [Create a plugin](#create-a-plugin) +2. **Choose how to access plugins**: -#### From local files - -You can copy example plugins from the [influxdb3_plugins repository](https://github.com/influxdata/influxdb3_plugins) to your local plugin directory: +**Option A: Copy plugins to your local directory** + +```bash +# Clone the repository +git clone https://github.com/influxdata/influxdb3_plugins.git + +# Copy a plugin to your configured plugin directory +cp influxdb3_plugins/examples/schedule/system_metrics/system_metrics.py /path/to/plugins/ +``` -#### Directly from GitHub +**Option B: Use plugins directly from GitHub** You can use plugins directly from GitHub without downloading them first by using the `gh:` prefix in the plugin filename: - + ```bash -# Use a plugin directly from GitHub influxdb3 create trigger \ --trigger-spec "every:1m" \ --plugin-filename "gh:examples/schedule/system_metrics/system_metrics.py" \ @@ -101,19 +101,11 @@ influxdb3 create trigger \ system_metrics ``` -> [!Note] -> #### Find and contribute plugins -> -> The plugins repository includes examples for various use cases: -> -> - **Data transformation**: Process and transform incoming data -> - **Alerting**: Send notifications based on data thresholds -> - **Aggregation**: Calculate statistics on time series data -> - **Integration**: Connect to external services and APIs -> - **System monitoring**: Track resource usage and health metrics -> -> Visit [influxdata/influxdb3_plugins](https://github.com/influxdata/influxdb3_plugins) -> to browse available plugins or contribute your own. +Plugins have various functions such as: + +- Receive plugin-specific arguments (such as written data, call time, or an HTTP request) +- Can receive keyword arguments (as `args`) from _trigger arguments_ +- Can access the `influxdb3_local` shared API for writing, querying, and managing state ### Create a plugin From 0f853ec0d46d4164db8d702d6442b79849cfa438 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 7 Apr 2025 11:22:27 -0700 Subject: [PATCH 05/51] docs: revising Create a custom plugin section --- content/shared/v3-core-plugins/_index.md | 53 ++++++++++++++++++++---- 1 file changed, 45 insertions(+), 8 deletions(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index b6ef5bbd83..d1358b2b9c 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -27,7 +27,7 @@ Once you have all the prerequisites in place, follow these steps to implement th 1. [Set up the Processing engine](#set-up-the-processing-engine) 2. [Add a Processing engine plugin](#add-a-processing-engine-plugin) - [Use example plugins](#use-example-plugins) - - [Create a plugin](#create-a-plugin) + - [Create a custom plugin](#create-a-custom-plugin) 3. [Create a trigger to run a plugin](#create-a-trigger-to-run-a-plugin) - [Create a trigger for data writes](#create-a-trigger-for-data-writes) - [Create a trigger for scheduled events](#create-a-trigger-for-scheduled-events) @@ -64,7 +64,7 @@ A plugin is a Python file that contains a specific function signature that corre You have two main options for adding plugins to your InfluxDB instance: 1. [Use example plugins](#use-example-plugins) - Quickest way to get started -2. [Create your own plugin](#create-a-plugin) - For custom functionality +2. [Create a custom plugin](#create-a-custom-plugin) - For custom functionality ### Use example plugins @@ -107,12 +107,32 @@ Plugins have various functions such as: - Can receive keyword arguments (as `args`) from _trigger arguments_ - Can access the `influxdb3_local` shared API for writing, querying, and managing state -### Create a plugin +### Create a custom plugin + +When you need custom functionality, you can create your own plugin be doing the following: + +#### Step 1: Choose your plugin type + +First, determine which type of plugin you need based on your automation goals: + +| Plugin Type | Best For | Trigger Type | +|-------------|----------|-------------| +| **Data write** | Processing data as it arrives | `table:` or `all_tables` | +| **Scheduled** | Running code at specific times | `every:` or `cron:` | +| **HTTP request** | Creating API endpoints | `path:` | + +#### Step 2: Create your plugin file 1. Create a `.py` file in your plugins directory -2. Define a function with one of the following signatures: +2. Add the appropriate function signature based on your chosen plugin type +3. Implement your processing logic inside the function + +##### Option A: Create a data write plugin -#### For data write events +Data write plugins process incoming data as it's written to the database. They're ideal for: +- Data transformation and enrichment +- Alerting on incoming values +- Creating derived metrics ```python def process_writes(influxdb3_local, table_batches, args=None): @@ -131,7 +151,13 @@ def process_writes(influxdb3_local, table_batches, args=None): influxdb3_local.write(line) ``` -#### For scheduled events +##### Option B: Create a scheduled plugin + +Scheduled plugins run at specific intervals or times. They're perfect for: + +- Periodic data aggregation +- Report generation +- System health checks ```python def process_scheduled_call(influxdb3_local, call_time, args=None): @@ -147,7 +173,13 @@ def process_scheduled_call(influxdb3_local, call_time, args=None): influxdb3_local.warn("No recent metrics found") ``` -#### For HTTP requests +##### Option C: Create an HTTP requests plugin + +HTTP request plugins respond to API calls. They're excellent for: + +- Creating custom API endpoints +- Web hooks for external integrations +- User interfaces for data interaction ```python def process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None): @@ -166,7 +198,12 @@ def process_request(influxdb3_local, query_parameters, request_headers, request_ return {"status": "success", "message": "Request processed"} ``` -After adding your plugin, you can [install Python dependencies](#install-python-dependencies) or learn how to [extend plugins with API features and state management](#extend-plugins-with-api-features-and-state-management). +#### Step 3: Next Steps + +After adding your plugin: +- You can [install Python dependencies](#install-python-dependencies) +- Learn how to [extend plugins with API features and state management](#extend-plugins-with-api-features-and-state-management) +- Create a trigger to connect your plugin to database events ## Create a trigger to run a plugin From c652fcd722b1bf1701c3e1daa6b71c030c5baab8 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 7 Apr 2025 14:07:28 -0700 Subject: [PATCH 06/51] docs: updating before you begin section and step three --- content/shared/v3-core-plugins/_index.md | 45 ++++++++++++++++-------- 1 file changed, 30 insertions(+), 15 deletions(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index d1358b2b9c..2a5fac3256 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -29,9 +29,9 @@ Once you have all the prerequisites in place, follow these steps to implement th - [Use example plugins](#use-example-plugins) - [Create a custom plugin](#create-a-custom-plugin) 3. [Create a trigger to run a plugin](#create-a-trigger-to-run-a-plugin) - - [Create a trigger for data writes](#create-a-trigger-for-data-writes) - - [Create a trigger for scheduled events](#create-a-trigger-for-scheduled-events) - - [Create a trigger for HTTP requests](#create-a-trigger-for-http-requests) + - [Understand trigger types](#understand-trigger-types) + - [Create a trigger](#create-a-trigger) + - [Choose a trigger specification](#choose-a-trigger-specification) - [Use community plugins from GitHub](#use-community-plugins-from-github) - [Pass arguments to plugins](#pass-arguments-to-plugins) - [Control trigger execution](#control-trigger-execution) @@ -173,7 +173,7 @@ def process_scheduled_call(influxdb3_local, call_time, args=None): influxdb3_local.warn("No recent metrics found") ``` -##### Option C: Create an HTTP requests plugin +##### Option C: Create an HTTP request plugin HTTP request plugins respond to API calls. They're excellent for: @@ -211,17 +211,36 @@ A trigger connects your plugin to a specific database event. The plugin function signature in your plugin file determines which _trigger specification_ you can choose for configuring and activating your plugin. -Create a trigger with the `influxdb3 create trigger` command. +After setting up your plugin, you need to connect it to specific database events using triggers. + +### Understand trigger types + +| Plugin Type | Trigger Specification | When Plugin Runs | +|------------|----------------------|-----------------| +| Data write | `table:` or `all_tables` | When data is written to tables | +| Scheduled | `every:` or `cron:` | At specified time intervals | +| HTTP request | `path:` | When HTTP requests are received | + +### Create a trigger + +Use the `influxdb3 create trigger` command with the appropriate trigger specification: + +```bash +influxdb3 create trigger \ + --trigger-spec "" \ + --plugin-filename "" \ + --database \ + + ``` > [!Note] > When specifying a local plugin file, the `--plugin-filename` parameter > _is relative to_ the `--plugin-dir` configured for the server. > You don't need to provide an absolute path. -### Create a trigger for data writes +### Choose a trigger specification -Use the `table:` or the `all_tables` trigger specification to configure -and run a [plugin for data write events](#for-data-write-events)--for example: +#### Option A: For data write events ```bash # Trigger on writes to a specific table @@ -245,10 +264,7 @@ to the Write-Ahead Log (WAL) in the Object store (default is every second). The plugin receives the written data and table information. -### Create a trigger for scheduled events - -Use the `every:` or the `cron:` trigger specification -to configure and run a [plugin for scheduled events](#for-scheduled-events)--for example: +#### Option B: For scheduled events ```bash # Run every 5 minutes @@ -268,9 +284,7 @@ influxdb3 create trigger \ The plugin receives the scheduled call time. -### Create a trigger for HTTP requests - -[For an HTTP request plugin], use the `path:` trigger specification to configure and enable a [plugin for HTTP requests](#for-http-requests)--for example: +#### Option C: For HTTP requests ```bash # Create an endpoint at /api/v3/engine/webhook @@ -290,6 +304,7 @@ curl http://{{% influxdb/host %}}/api/v3/engine/webhook The plugin receives the HTTP request object with methods, headers, and body. +-----[Pick up here]------- ### Use community plugins from GitHub You can reference plugins directly from the GitHub repository by using the `gh:` prefix: From 3572d79c0920c68d8eb0b49b78844407db5f33e0 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 7 Apr 2025 14:15:40 -0700 Subject: [PATCH 07/51] WIP: porting over extened plugins with api feature --- .../v3-core-plugins/extended-plugin-api.md | 292 ++++++++++++++++++ 1 file changed, 292 insertions(+) create mode 100644 content/shared/v3-core-plugins/extended-plugin-api.md diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md new file mode 100644 index 0000000000..14a5f88115 --- /dev/null +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -0,0 +1,292 @@ +## Extend plugins with API features and state management + +The Processing engine includes API capabilities that allow your plugins to +interact with InfluxDB data and maintain state between executions. +These features let you build more sophisticated plugins that can transform, analyze, and respond to data. + +### Use the shared API + +All plugins have access to the shared API to interact with the database. + +#### Write data + +Use the `LineBuilder` API to create line protocol data: + +```python +# Create a line protocol entry +line = LineBuilder("weather") +line.tag("location", "us-midwest") +line.float64_field("temperature", 82.5) +line.time_ns(1627680000000000000) + +# Write the data to the database +influxdb3_local.write(line) +``` + +Writes are buffered while the plugin runs and are flushed when the plugin completes. + +{{% expand-wrapper %}} +{{% expand "View the `LineBuilder` Python implementation" %}} + +```python +from typing import Optional +from collections import OrderedDict + +class InfluxDBError(Exception): + """Base exception for InfluxDB-related errors""" + pass + +class InvalidMeasurementError(InfluxDBError): + """Raised when measurement name is invalid""" + pass + +class InvalidKeyError(InfluxDBError): + """Raised when a tag or field key is invalid""" + pass + +class InvalidLineError(InfluxDBError): + """Raised when a line protocol string is invalid""" + pass + +class LineBuilder: + def __init__(self, measurement: str): + if ' ' in measurement: + raise InvalidMeasurementError("Measurement name cannot contain spaces") + self.measurement = measurement + self.tags: OrderedDict[str, str] = OrderedDict() + self.fields: OrderedDict[str, str] = OrderedDict() + self._timestamp_ns: Optional[int] = None + + def _validate_key(self, key: str, key_type: str) -> None: + """Validate that a key does not contain spaces, commas, or equals signs.""" + if not key: + raise InvalidKeyError(f"{key_type} key cannot be empty") + if ' ' in key: + raise InvalidKeyError(f"{key_type} key '{key}' cannot contain spaces") + if ',' in key: + raise InvalidKeyError(f"{key_type} key '{key}' cannot contain commas") + if '=' in key: + raise InvalidKeyError(f"{key_type} key '{key}' cannot contain equals signs") + + def tag(self, key: str, value: str) -> 'LineBuilder': + """Add a tag to the line protocol.""" + self._validate_key(key, "tag") + self.tags[key] = str(value) + return self + + def uint64_field(self, key: str, value: int) -> 'LineBuilder': + """Add an unsigned integer field to the line protocol.""" + self._validate_key(key, "field") + if value < 0: + raise ValueError(f"uint64 field '{key}' cannot be negative") + self.fields[key] = f"{value}u" + return self + + def int64_field(self, key: str, value: int) -> 'LineBuilder': + """Add an integer field to the line protocol.""" + self._validate_key(key, "field") + self.fields[key] = f"{value}i" + return self + + def float64_field(self, key: str, value: float) -> 'LineBuilder': + """Add a float field to the line protocol.""" + self._validate_key(key, "field") + # Check if value has no decimal component + self.fields[key] = f"{int(value)}.0" if value % 1 == 0 else str(value) + return self + + def string_field(self, key: str, value: str) -> 'LineBuilder': + """Add a string field to the line protocol.""" + self._validate_key(key, "field") + # Escape quotes and backslashes in string values + escaped_value = value.replace('"', '\\"').replace('\\', '\\\\') + self.fields[key] = f'"{escaped_value}"' + return self + + def bool_field(self, key: str, value: bool) -> 'LineBuilder': + """Add a boolean field to the line protocol.""" + self._validate_key(key, "field") + self.fields[key] = 't' if value else 'f' + return self + + def time_ns(self, timestamp_ns: int) -> 'LineBuilder': + """Set the timestamp in nanoseconds.""" + self._timestamp_ns = timestamp_ns + return self + + def build(self) -> str: + """Build the line protocol string.""" + # Start with measurement name (escape commas only) + line = self.measurement.replace(',', '\\,') + + # Add tags if present + if self.tags: + tags_str = ','.join( + f"{k}={v}" for k, v in self.tags.items() + ) + line += f",{tags_str}" + + # Add fields (required) + if not self.fields: + raise InvalidLineError(f"At least one field is required: {line}") + + fields_str = ','.join( + f"{k}={v}" for k, v in self.fields.items() + ) + line += f" {fields_str}" + + # Add timestamp if present + if self._timestamp_ns is not None: + line += f" {self._timestamp_ns}" + + return line +``` +{{% /expand %}} +{{% /expand-wrapper %}} + +#### Query data + +Execute SQL queries and get results: + +```python +# Simple query +results = influxdb3_local.query("SELECT * FROM metrics WHERE time > now() - INTERVAL '1 hour'") + +# Parameterized query for safer execution +params = {"table": "metrics", "threshold": 90} +results = influxdb3_local.query("SELECT * FROM $table WHERE value > $threshold", params) +``` + +The shared API `query` function returns results as a `List` of `Dict[String, Any]`, where the key is the column name and the value is the column value. + +#### Log information + +The shared API `info`, `warn`, and `error` functions accept multiple arguments, +convert them to strings, and log them as a space-separated message to the database log, +which is output in the server logs and captured in system tables that you can +query using SQL. + +Add logging to track plugin execution: + +```python +influxdb3_local.info("Starting data processing") +influxdb3_local.warn("Could not process some records") +influxdb3_local.error("Failed to connect to external API") + +# Log structured data +obj_to_log = {"records": 157, "errors": 3} +influxdb3_local.info("Processing complete", obj_to_log) +``` + +#### Use the in-memory cache + +The Processing engine provides an in-memory cache system that enables plugins to persist and retrieve data between executions. + +Use the shared API `cache` property to access the cache API. + +```python +# Basic usage pattern +influxdb3_local.cache.METHOD(PARAMETERS) +``` + +| Method | Parameters | Returns | Description | +|--------|------------|---------|-------------| +| `put` | `key` (str): The key to store the value under
`value` (Any): Any Python object to cache
`ttl` (Optional[float], default=None): Time in seconds before expiration
`use_global` (bool, default=False): If True, uses global namespace | None | Stores a value in the cache with an optional time-to-live | +| `get` | `key` (str): The key to retrieve
`default` (Any, default=None): Value to return if key not found
`use_global` (bool, default=False): If True, uses global namespace | Any | Retrieves a value from the cache or returns default if not found | +| `delete` | `key` (str): The key to delete
`use_global` (bool, default=False): If True, uses global namespace | bool | Deletes a value from the cache. Returns True if deleted, False if not found | + +##### Cache namespaces + +The cache system offers two distinct namespaces: + +| Namespace | Scope | Best For | +| --- | --- | --- | +| **Trigger-specific** (default) | Isolated to a single trigger | Plugin state, counters, timestamps specific to one plugin | +| **Global** | Shared across all triggers | Configuration, lookup tables, service states that should be available to all plugins | + +##### Store and retrieve cached data + +```python +# Store a value +influxdb3_local.cache.put("last_run_time", time.time()) + +# Retrieve a value with a default if not found +last_time = influxdb3_local.cache.get("last_run_time", default=0) + +# Delete a cached value +influxdb3_local.cache.delete("temporary_data") +``` + +##### Store cached data with expiration + +```python +# Cache with a 5-minute TTL (time-to-live) +influxdb3_local.cache.put("api_response", response_data, ttl=300) +``` + +##### Share data across plugins + +```python +# Store in the global namespace +influxdb3_local.cache.put("config", {"version": "1.0"}, use_global=True) + +# Retrieve from the global namespace +config = influxdb3_local.cache.get("config", use_global=True) +``` + +##### Track state between executions + +```python +# Get current counter or default to 0 +counter = influxdb3_local.cache.get("execution_count", default=0) + +# Increment counter +counter += 1 + +# Store the updated value +influxdb3_local.cache.put("execution_count", counter) + +influxdb3_local.info(f"This plugin has run {counter} times") +``` + +#### Best practices for in-memory caching + +- [Use the trigger-specific namespace](#use-the-trigger-specific-namespace) +- [Use TTL appropriately](#use-ttl-appropriately) +- [Cache computation results](#cache-computation-results) +- [Warm the cache](#warm-the-cache) +- [Consider cache limitations](#consider-cache-limitations) + +##### Use the trigger-specific namespace + +The cache is designed to support stateful operations while maintaining isolation between different triggers. Use the trigger-specific namespace for most operations and the global namespace only when data sharing across triggers is necessary. + +##### Use TTL appropriately +Set realistic expiration times based on how frequently data changes. + +```python +# Cache external API responses for 5 minutes +influxdb3_local.cache.put("weather_data", api_response, ttl=300) +``` + +##### Cache computation results +Store the results of expensive calculations that need to be utilized frequently. +```python +# Cache aggregated statistics +influxdb3_local.cache.put("daily_stats", calculate_statistics(data), ttl=3600) +``` + +##### Warm the cache +For critical data, prime the cache at startup. This can be especially useful for global namespace data where multiple triggers need the data. + +```python +# Check if cache needs to be initialized +if not influxdb3_local.cache.get("lookup_table"): + influxdb3_local.cache.put("lookup_table", load_lookup_data()) +``` + +##### Consider cache limitations + +- **Memory Usage**: Since cache contents are stored in memory, monitor your memory usage when caching large datasets. +- **Server Restarts**: Because the cache is cleared when the server restarts, design your plugins to handle cache initialization (as noted above). +- **Concurrency**: Be cautious of accessing inaccurate or out-of-date data when multiple trigger instances might simultaneously update the same cache key. \ No newline at end of file From 797c74fe47c6bff0df9f2d582b59a50d5a420dc3 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 9 Apr 2025 11:40:31 -0700 Subject: [PATCH 08/51] docs: updating gh triggers --- content/shared/v3-core-plugins/_index.md | 307 ++--------------------- 1 file changed, 18 insertions(+), 289 deletions(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index 2a5fac3256..611716d599 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -153,7 +153,7 @@ def process_writes(influxdb3_local, table_batches, args=None): ##### Option B: Create a scheduled plugin -Scheduled plugins run at specific intervals or times. They're perfect for: +Scheduled plugins run at specific intervals or times. They can be used for: - Periodic data aggregation - Report generation @@ -175,7 +175,7 @@ def process_scheduled_call(influxdb3_local, call_time, args=None): ##### Option C: Create an HTTP request plugin -HTTP request plugins respond to API calls. They're excellent for: +HTTP request plugins respond to API calls. They can be used for: - Creating custom API endpoints - Web hooks for external integrations @@ -295,7 +295,7 @@ influxdb3 create trigger \ webhook_processor ``` -The trigger makes your endpoint available at `/api/v3/engine/`. +Access your endpoint available at `/api/v3/engine/`. To run the plugin, send a `GET` or `POST` request to the endpoint--for example: ```bash @@ -304,7 +304,6 @@ curl http://{{% influxdb/host %}}/api/v3/engine/webhook The plugin receives the HTTP request object with methods, headers, and body. ------[Pick up here]------- ### Use community plugins from GitHub You can reference plugins directly from the GitHub repository by using the `gh:` prefix: @@ -388,298 +387,28 @@ influxdb3 create trigger \ auto_disable_processor ``` -## Extend plugins with API features and state management +## Advanced trigger configuration -The Processing engine includes API capabilities that allow your plugins to -interact with InfluxDB data and maintain state between executions. -These features let you build more sophisticated plugins that can transform, analyze, and respond to data. +After creating basic triggers, you can enhance your plugins with these advanced features: -### Use the shared API +### Step 1: Access community plugins from GitHub -All plugins have access to the shared API to interact with the database. +Skip downloading plugins by referencing them directly from GitHub: -#### Write data - -Use the `LineBuilder` API to create line protocol data: - -```python -# Create a line protocol entry -line = LineBuilder("weather") -line.tag("location", "us-midwest") -line.float64_field("temperature", 82.5) -line.time_ns(1627680000000000000) - -# Write the data to the database -influxdb3_local.write(line) -``` - -Writes are buffered while the plugin runs and are flushed when the plugin completes. - -{{% expand-wrapper %}} -{{% expand "View the `LineBuilder` Python implementation" %}} - -```python -from typing import Optional -from collections import OrderedDict - -class InfluxDBError(Exception): - """Base exception for InfluxDB-related errors""" - pass - -class InvalidMeasurementError(InfluxDBError): - """Raised when measurement name is invalid""" - pass - -class InvalidKeyError(InfluxDBError): - """Raised when a tag or field key is invalid""" - pass - -class InvalidLineError(InfluxDBError): - """Raised when a line protocol string is invalid""" - pass - -class LineBuilder: - def __init__(self, measurement: str): - if ' ' in measurement: - raise InvalidMeasurementError("Measurement name cannot contain spaces") - self.measurement = measurement - self.tags: OrderedDict[str, str] = OrderedDict() - self.fields: OrderedDict[str, str] = OrderedDict() - self._timestamp_ns: Optional[int] = None - - def _validate_key(self, key: str, key_type: str) -> None: - """Validate that a key does not contain spaces, commas, or equals signs.""" - if not key: - raise InvalidKeyError(f"{key_type} key cannot be empty") - if ' ' in key: - raise InvalidKeyError(f"{key_type} key '{key}' cannot contain spaces") - if ',' in key: - raise InvalidKeyError(f"{key_type} key '{key}' cannot contain commas") - if '=' in key: - raise InvalidKeyError(f"{key_type} key '{key}' cannot contain equals signs") - - def tag(self, key: str, value: str) -> 'LineBuilder': - """Add a tag to the line protocol.""" - self._validate_key(key, "tag") - self.tags[key] = str(value) - return self - - def uint64_field(self, key: str, value: int) -> 'LineBuilder': - """Add an unsigned integer field to the line protocol.""" - self._validate_key(key, "field") - if value < 0: - raise ValueError(f"uint64 field '{key}' cannot be negative") - self.fields[key] = f"{value}u" - return self - - def int64_field(self, key: str, value: int) -> 'LineBuilder': - """Add an integer field to the line protocol.""" - self._validate_key(key, "field") - self.fields[key] = f"{value}i" - return self - - def float64_field(self, key: str, value: float) -> 'LineBuilder': - """Add a float field to the line protocol.""" - self._validate_key(key, "field") - # Check if value has no decimal component - self.fields[key] = f"{int(value)}.0" if value % 1 == 0 else str(value) - return self - - def string_field(self, key: str, value: str) -> 'LineBuilder': - """Add a string field to the line protocol.""" - self._validate_key(key, "field") - # Escape quotes and backslashes in string values - escaped_value = value.replace('"', '\\"').replace('\\', '\\\\') - self.fields[key] = f'"{escaped_value}"' - return self - - def bool_field(self, key: str, value: bool) -> 'LineBuilder': - """Add a boolean field to the line protocol.""" - self._validate_key(key, "field") - self.fields[key] = 't' if value else 'f' - return self - - def time_ns(self, timestamp_ns: int) -> 'LineBuilder': - """Set the timestamp in nanoseconds.""" - self._timestamp_ns = timestamp_ns - return self - - def build(self) -> str: - """Build the line protocol string.""" - # Start with measurement name (escape commas only) - line = self.measurement.replace(',', '\\,') - - # Add tags if present - if self.tags: - tags_str = ','.join( - f"{k}={v}" for k, v in self.tags.items() - ) - line += f",{tags_str}" - - # Add fields (required) - if not self.fields: - raise InvalidLineError(f"At least one field is required: {line}") - - fields_str = ','.join( - f"{k}={v}" for k, v in self.fields.items() - ) - line += f" {fields_str}" - - # Add timestamp if present - if self._timestamp_ns is not None: - line += f" {self._timestamp_ns}" - - return line -``` -{{% /expand %}} -{{% /expand-wrapper %}} - -#### Query data - -Execute SQL queries and get results: - -```python -# Simple query -results = influxdb3_local.query("SELECT * FROM metrics WHERE time > now() - INTERVAL '1 hour'") - -# Parameterized query for safer execution -params = {"table": "metrics", "threshold": 90} -results = influxdb3_local.query("SELECT * FROM $table WHERE value > $threshold", params) -``` - -The shared API `query` function returns results as a `List` of `Dict[String, Any]`, where the key is the column name and the value is the column value. - -#### Log information - -The shared API `info`, `warn`, and `error` functions accept multiple arguments, -convert them to strings, and log them as a space-separated message to the database log, -which is output in the server logs and captured in system tables that you can -query using SQL. - -Add logging to track plugin execution: - -```python -influxdb3_local.info("Starting data processing") -influxdb3_local.warn("Could not process some records") -influxdb3_local.error("Failed to connect to external API") - -# Log structured data -obj_to_log = {"records": 157, "errors": 3} -influxdb3_local.info("Processing complete", obj_to_log) -``` - -#### Use the in-memory cache - -The Processing engine provides an in-memory cache system that enables plugins to persist and retrieve data between executions. - -Use the shared API `cache` property to access the cache API. - -```python -# Basic usage pattern -influxdb3_local.cache.METHOD(PARAMETERS) -``` - -| Method | Parameters | Returns | Description | -|--------|------------|---------|-------------| -| `put` | `key` (str): The key to store the value under
`value` (Any): Any Python object to cache
`ttl` (Optional[float], default=None): Time in seconds before expiration
`use_global` (bool, default=False): If True, uses global namespace | None | Stores a value in the cache with an optional time-to-live | -| `get` | `key` (str): The key to retrieve
`default` (Any, default=None): Value to return if key not found
`use_global` (bool, default=False): If True, uses global namespace | Any | Retrieves a value from the cache or returns default if not found | -| `delete` | `key` (str): The key to delete
`use_global` (bool, default=False): If True, uses global namespace | bool | Deletes a value from the cache. Returns True if deleted, False if not found | - -##### Cache namespaces - -The cache system offers two distinct namespaces: - -| Namespace | Scope | Best For | -| --- | --- | --- | -| **Trigger-specific** (default) | Isolated to a single trigger | Plugin state, counters, timestamps specific to one plugin | -| **Global** | Shared across all triggers | Configuration, lookup tables, service states that should be available to all plugins | - -##### Store and retrieve cached data - -```python -# Store a value -influxdb3_local.cache.put("last_run_time", time.time()) - -# Retrieve a value with a default if not found -last_time = influxdb3_local.cache.get("last_run_time", default=0) - -# Delete a cached value -influxdb3_local.cache.delete("temporary_data") -``` - -##### Store cached data with expiration - -```python -# Cache with a 5-minute TTL (time-to-live) -influxdb3_local.cache.put("api_response", response_data, ttl=300) -``` - -##### Share data across plugins - -```python -# Store in the global namespace -influxdb3_local.cache.put("config", {"version": "1.0"}, use_global=True) - -# Retrieve from the global namespace -config = influxdb3_local.cache.get("config", use_global=True) -``` - -##### Track state between executions - -```python -# Get current counter or default to 0 -counter = influxdb3_local.cache.get("execution_count", default=0) - -# Increment counter -counter += 1 - -# Store the updated value -influxdb3_local.cache.put("execution_count", counter) - -influxdb3_local.info(f"This plugin has run {counter} times") -``` - -#### Best practices for in-memory caching - -- [Use the trigger-specific namespace](#use-the-trigger-specific-namespace) -- [Use TTL appropriately](#use-ttl-appropriately) -- [Cache computation results](#cache-computation-results) -- [Warm the cache](#warm-the-cache) -- [Consider cache limitations](#consider-cache-limitations) - -##### Use the trigger-specific namespace - -The cache is designed to support stateful operations while maintaining isolation between different triggers. Use the trigger-specific namespace for most operations and the global namespace only when data sharing across triggers is necessary. - -##### Use TTL appropriately -Set realistic expiration times based on how frequently data changes. - -```python -# Cache external API responses for 5 minutes -influxdb3_local.cache.put("weather_data", api_response, ttl=300) -``` - -##### Cache computation results -Store the results of expensive calculations that need to be utilized frequently. -```python -# Cache aggregated statistics -influxdb3_local.cache.put("daily_stats", calculate_statistics(data), ttl=3600) -``` - -##### Warm the cache -For critical data, prime the cache at startup. This can be especially useful for global namespace data where multiple triggers need the data. - -```python -# Check if cache needs to be initialized -if not influxdb3_local.cache.get("lookup_table"): - influxdb3_local.cache.put("lookup_table", load_lookup_data()) +```bash +# Create a trigger using a plugin from GitHub +influxdb3 create trigger \ + --trigger-spec "every:1m" \ + --plugin-filename "gh:examples/schedule/system_metrics/system_metrics.py" \ + --database my_database \ + system_metrics ``` -##### Consider cache limitations +This approach: -- **Memory Usage**: Since cache contents are stored in memory, monitor your memory usage when caching large datasets. -- **Server Restarts**: Because the cache is cleared when the server restarts, design your plugins to handle cache initialization (as noted above). -- **Concurrency**: Be cautious of accessing inaccurate or out-of-date data when multiple trigger instances might simultaneously update the same cache key. +- Ensures you're using the latest version +- Simplifies updates and maintenance +- Reduces local storage requirements ## Install Python dependencies From 0a66059f274b99038b8443c43bd27e5d48d413e4 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 9 Apr 2025 11:57:59 -0700 Subject: [PATCH 09/51] docs: updating configure error handling and python dependencies --- content/shared/v3-core-plugins/_index.md | 60 ++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index 611716d599..7c9a665253 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -410,6 +410,66 @@ This approach: - Simplifies updates and maintenance - Reduces local storage requirements +### Step 2: Configure your tiggers + +#### Pass configuration arguments + +provide runtine configuration to your plugins: + +# Pass threshold and email settings to a plugin + +```bash +influxdb3 create trigger \ + --trigger-spec "every:1h" \ + --plugin-filename "threshold_check.py" \ + --trigger-arguments threshold=90,notify_email=admin@example.com \ + --database my_database \ + threshold_monitor +``` +Your plugin accesses these values through the `args` parameter: + +```python +def process_scheduled_call(influxdb3_local, call_time, args=None): + if args and "threshold" in args: + threshold = float(args["threshold"]) + email = args.get("notify_email", "default@example.com") + + # Use the arguments in your logic + influxdb3_local.info(f"Checking threshold {threshold}, will notify {email}") +``` +#### Set execution mode + +Choose between synchronous (default) or asynchronous excution: + +```bash +# Allow multiple trigger instances to run simultaneously +influxdb3 create trigger \ + --trigger-spec "table:metrics" \ + --plugin-filename "heavy_process.py" \ + --run-asynchronous \ + --database my_database \ + async_processor +``` + +Use asynchronous execution when: + +- Processing might take longer than the trigger interval +- Multiple events need to be handled simultaneously +- Performance is more important than sequential execution + +#### Configure error handling + +Control how your trigger responds to errors: +```bash +# Automatically retry on error +influxdb3 create trigger \ + --trigger-spec "table:important_data" \ + --plugin-filename "critical_process.py" \ + --error-behavior retry \ + --database my_database \ + critical_processor +``` + ## Install Python dependencies If your plugin needs additional Python packages, use the `influxdb3 install` command: From d7c63d5d0f1dce8560a45a98a479900d584bbfd0 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 9 Apr 2025 12:13:32 -0700 Subject: [PATCH 10/51] docs: first read through for mistakes --- content/shared/v3-core-plugins/_index.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index 7c9a665253..79b08c9578 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -63,8 +63,8 @@ A plugin is a Python file that contains a specific function signature that corre You have two main options for adding plugins to your InfluxDB instance: -1. [Use example plugins](#use-example-plugins) - Quickest way to get started -2. [Create a custom plugin](#create-a-custom-plugin) - For custom functionality +- [Use example plugins](#use-example-plugins) - Quickest way to get started +- [Create a custom plugin](#create-a-custom-plugin) - For custom functionality ### Use example plugins @@ -109,7 +109,7 @@ Plugins have various functions such as: ### Create a custom plugin -When you need custom functionality, you can create your own plugin be doing the following: +When you need custom functionality, you can create your own plugin by doing the following: #### Step 1: Choose your plugin type @@ -410,15 +410,14 @@ This approach: - Simplifies updates and maintenance - Reduces local storage requirements -### Step 2: Configure your tiggers +### Step 2: Configure your triggers #### Pass configuration arguments -provide runtine configuration to your plugins: - -# Pass threshold and email settings to a plugin +Provide runtine configuration to your plugins: ```bash +# Pass threshold and email settings to a plugin influxdb3 create trigger \ --trigger-spec "every:1h" \ --plugin-filename "threshold_check.py" \ @@ -439,7 +438,7 @@ def process_scheduled_call(influxdb3_local, call_time, args=None): ``` #### Set execution mode -Choose between synchronous (default) or asynchronous excution: +Choose between synchronous (default) or asynchronous execution: ```bash # Allow multiple trigger instances to run simultaneously From 2e2da735ecb6b9bee3c7571592c79fa21238d8fe Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 9 Apr 2025 12:26:11 -0700 Subject: [PATCH 11/51] WIP: Setting up file and path structure and confirming functionality --- content/influxdb3/enterprise/extended-plugin.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 content/influxdb3/enterprise/extended-plugin.md diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md new file mode 100644 index 0000000000..247edfea68 --- /dev/null +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -0,0 +1,15 @@ +--- +title: Extend plugins with API features and state management +description: | + The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. +menu: + influxdb3_enterprise: + name: Extended Plugins +weight: 4 +influxdb3/enterprise/tags: [processing engine, plugins, API, python] +source: /shared/v3-core-plugins/extended-plugin-api.md +--- + + \ No newline at end of file From 7350053f96af9c5edbc5fca223e3dc3e0f7b2d7e Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 14 Apr 2025 14:38:38 -0700 Subject: [PATCH 12/51] adding extebded-plugin.md to core --- content/influxdb3/core/extended-plugin.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 content/influxdb3/core/extended-plugin.md diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md new file mode 100644 index 0000000000..75c2b93ade --- /dev/null +++ b/content/influxdb3/core/extended-plugin.md @@ -0,0 +1,15 @@ +--- +title: Extend plugins with API features and state management +description: | + The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. +menu: + influxdb3_core: + name: Extended Plugins +weight: 4 +influxdb3/enterprise/tags: [processing engine, plugins, API, python] +source: /shared/v3-core-plugins/extended-plugin-api.md +--- + + \ No newline at end of file From 098942ea7381085e41979977ea4caa7e04f7a166 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 14 Apr 2025 14:40:13 -0700 Subject: [PATCH 13/51] updates to extended-plugin.md in core --- content/influxdb3/core/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md index 75c2b93ade..11fad20a3b 100644 --- a/content/influxdb3/core/extended-plugin.md +++ b/content/influxdb3/core/extended-plugin.md @@ -6,7 +6,7 @@ menu: influxdb3_core: name: Extended Plugins weight: 4 -influxdb3/enterprise/tags: [processing engine, plugins, API, python] +influxdb3/core/tags: [processing engine, plugins, API, python] source: /shared/v3-core-plugins/extended-plugin-api.md --- From 6ec728a639ecba738d577be3945f19388528b280 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 14 Apr 2025 15:08:16 -0700 Subject: [PATCH 14/51] updating enterprise extended-plugin.md --- content/influxdb3/core/extended-plugin.md | 1 + content/influxdb3/enterprise/extended-plugin.md | 1 + 2 files changed, 2 insertions(+) diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md index 11fad20a3b..5a77a62b9a 100644 --- a/content/influxdb3/core/extended-plugin.md +++ b/content/influxdb3/core/extended-plugin.md @@ -5,6 +5,7 @@ description: | menu: influxdb3_core: name: Extended Plugins + parent: Processing Engine and Python plugins weight: 4 influxdb3/core/tags: [processing engine, plugins, API, python] source: /shared/v3-core-plugins/extended-plugin-api.md diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md index 247edfea68..414e8ded4a 100644 --- a/content/influxdb3/enterprise/extended-plugin.md +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -5,6 +5,7 @@ description: | menu: influxdb3_enterprise: name: Extended Plugins + parent: Processing Engine and Python plugins weight: 4 influxdb3/enterprise/tags: [processing engine, plugins, API, python] source: /shared/v3-core-plugins/extended-plugin-api.md From 639481278914ea02e893e471d4698833587bfbef Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 14 Apr 2025 15:14:27 -0700 Subject: [PATCH 15/51] Checking file paths and dropdowns --- content/influxdb3/core/extended-plugin.md | 4 +- .../influxdb3/enterprise/extended-plugin.md | 4 +- content/influxdb3/enterprise/plugins.md | 2 +- content/shared/influxdb3-processing-engine.md | 84 +++++++++++++++++++ 4 files changed, 89 insertions(+), 5 deletions(-) create mode 100644 content/shared/influxdb3-processing-engine.md diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md index 5a77a62b9a..f3fb30ecf2 100644 --- a/content/influxdb3/core/extended-plugin.md +++ b/content/influxdb3/core/extended-plugin.md @@ -4,8 +4,8 @@ description: | The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. menu: influxdb3_core: - name: Extended Plugins - parent: Processing Engine and Python plugins + name: Extended plugins + parent: Processing engine and Python plugins weight: 4 influxdb3/core/tags: [processing engine, plugins, API, python] source: /shared/v3-core-plugins/extended-plugin-api.md diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md index 414e8ded4a..f7e99a6d23 100644 --- a/content/influxdb3/enterprise/extended-plugin.md +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -4,8 +4,8 @@ description: | The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. menu: influxdb3_enterprise: - name: Extended Plugins - parent: Processing Engine and Python plugins + name: Extended plugins + parent: Processing engine and Python plugins weight: 4 influxdb3/enterprise/tags: [processing engine, plugins, API, python] source: /shared/v3-core-plugins/extended-plugin-api.md diff --git a/content/influxdb3/enterprise/plugins.md b/content/influxdb3/enterprise/plugins.md index 73ad8c3c3f..6862a163fe 100644 --- a/content/influxdb3/enterprise/plugins.md +++ b/content/influxdb3/enterprise/plugins.md @@ -5,7 +5,7 @@ description: | code on different events in an {{< product-name >}} instance. menu: influxdb3_enterprise: - name: Processing Engine and Python plugins + name: Processing engine and Python plugins weight: 4 influxdb3/enterprise/tags: [processing engine, python] related: diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md new file mode 100644 index 0000000000..81fec66323 --- /dev/null +++ b/content/shared/influxdb3-processing-engine.md @@ -0,0 +1,84 @@ +# Processing engine + +The Processing engine is an embedded Python virtual machine that runs inside an {{% product-name %}} database server. It executes Python code in response to triggers and database events without requiring external application servers or middleware. + +## How it works + +### Architecture + +The Processing engine runs Python code directly within a {{% product-name %}} server process. This design provides high performance and direct access to database resources. + +- **Embedded execution**: Code runs in the same process space as the database server +- **Direct data access**: Zero-copy access to data +- **Event-driven**: Responds to database writes, scheduled events, and HTTP requests +- **Isolated contexts**: Maintains separation between different plugin executions +- **Cache integration**: Access to system caches including Last values and Distinct values + +### Event processing flow + +When events occur in the database, the Processing engine handles them through a consistent sequence: + +1. A **trigger** activates the plugin based on one of three event types: + - Data writes to specific tables or all tables + - Scheduled events (time-based or cron expressions) + - HTTP requests to configured endpoints +2. The engine loads the associated **plugin** specified in the trigger configuration +3. The plugin receives context data specific to the trigger type: + - Write triggers: the written data and table information + - Schedule triggers: the scheduled call time + - HTTP triggers: the request object with methods, headers, and body +4. The plugin processes the received data, can query the database, call external tools, and write results back +5. Execution completes and the engine returns to waiting state + +## Key components + +### Trigger system + +Triggers connect database events to Python code execution based on specific conditions: + +- **Data write triggers**: Execute on WAL flush events, when data is written to the object store, for a specific table or all tables in a database +- **Scheduled triggers**: Run at intervals or according to cron expressions +- **HTTP triggers**: Respond to HTTP requests to custom endpoints + +### Plugin registry + +The registry manages all Python code available to the Processing engine: + +- Indexes plugins by filename and location +- Tracks which plugins are used by which triggers +- Manages plugin versioning and dependencies + +### Memory management + +The Processing engine implements specialized memory handling to ensure stability and performance: + +- **Execution isolation**: Each plugin runs in its own context +- **Cache system**: Maintains state between executions +- **Resource limits**: Controls memory usage and execution time + +## Performance characteristics + +The Processing engine is designed for high-performance operation with minimal overhead: + +- **Low latency**: Activates triggers in sub-millisecond time +- **Efficient access**: Accesses database directly without network overhead +- **Controlled resources**: Limits memory and CPU usage through configuration +- **Execution policies**: Offers synchronous or asynchronous processing options + +## Reliability features + +The Processing engine includes multiple features to ensure consistent and dependable execution: + +- **Error handling**: Configures behaviors for failure scenarios (log, retry, or disable) +- **Execution tracking**: Tracks plugin performance and resource usage +- **State persistence**: Persists cache state across server restarts + +## Extension capabilities + +Extend and customize the Processing engine through several built-in mechanisms: + +- **Package management**: Installs custom Python dependencies +- **Plugin distribution**: Distributes plugins via Git repositories +- **Shared API**: Provides consistent interface for database operations + +For a step-by-step guide to setting up and using the Processing engine, see the [Getting started with plugins](/influxdb3/core/plugins/) documentation. \ No newline at end of file From e568a12168eb252622e562d79cae014ccff8eb19 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 14 Apr 2025 15:42:37 -0700 Subject: [PATCH 16/51] docs: updating introduction sections for extended plugins --- content/shared/v3-core-plugins/extended-plugin-api.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md index 14a5f88115..09130c8add 100644 --- a/content/shared/v3-core-plugins/extended-plugin-api.md +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -4,9 +4,13 @@ The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. These features let you build more sophisticated plugins that can transform, analyze, and respond to data. -### Use the shared API +The plugin API lets you: +- Write and query data directly from your python code +- Track information between plugin executions with the in-memory cache +- Log messages for monitoring and debugging +- Build data processing workflows -All plugins have access to the shared API to interact with the database. +Let's explore how to use these fatures in your pligins. #### Write data @@ -262,6 +266,7 @@ influxdb3_local.info(f"This plugin has run {counter} times") The cache is designed to support stateful operations while maintaining isolation between different triggers. Use the trigger-specific namespace for most operations and the global namespace only when data sharing across triggers is necessary. ##### Use TTL appropriately + Set realistic expiration times based on how frequently data changes. ```python @@ -270,6 +275,7 @@ influxdb3_local.cache.put("weather_data", api_response, ttl=300) ``` ##### Cache computation results + Store the results of expensive calculations that need to be utilized frequently. ```python # Cache aggregated statistics @@ -277,6 +283,7 @@ influxdb3_local.cache.put("daily_stats", calculate_statistics(data), ttl=3600) ``` ##### Warm the cache + For critical data, prime the cache at startup. This can be especially useful for global namespace data where multiple triggers need the data. ```python From ce3974c80032bf323ac24d1c51b1c558e7b59aba Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 14 Apr 2025 15:49:23 -0700 Subject: [PATCH 17/51] updating and adding a get started section --- .../shared/v3-core-plugins/extended-plugin-api.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md index 09130c8add..8bffd5e5f6 100644 --- a/content/shared/v3-core-plugins/extended-plugin-api.md +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -12,9 +12,13 @@ The plugin API lets you: Let's explore how to use these fatures in your pligins. +### Getting started with the shared API + +Every plugin automatically has access to the shared API through the influxdb3_local object. You don't need to import any libraries - this API is available as soon as your plugin runs. + #### Write data -Use the `LineBuilder` API to create line protocol data: +To write data into your database use the `LineBuilder` API to create line protocol data: ```python # Create a line protocol entry @@ -27,7 +31,7 @@ line.time_ns(1627680000000000000) influxdb3_local.write(line) ``` -Writes are buffered while the plugin runs and are flushed when the plugin completes. +Your writes are buffered while the plugin runs and are flushed when the plugin completes. {{% expand-wrapper %}} {{% expand "View the `LineBuilder` Python implementation" %}} @@ -150,7 +154,7 @@ class LineBuilder: #### Query data -Execute SQL queries and get results: +Your plugins can execute SQL queries and process the results directly: ```python # Simple query @@ -161,7 +165,9 @@ params = {"table": "metrics", "threshold": 90} results = influxdb3_local.query("SELECT * FROM $table WHERE value > $threshold", params) ``` -The shared API `query` function returns results as a `List` of `Dict[String, Any]`, where the key is the column name and the value is the column value. +Query results come back as a `List` of `Dict[String, Any]`, where each dictionary represents a row with column names as keys and column values as values. This makes it easy to process the results in your plugin code. + + #### Log information From 149b175fb7ee2a3d387af1827537f36809c181bd Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 14 Apr 2025 17:05:01 -0700 Subject: [PATCH 18/51] docs: finishing up resturcting and did first proofread --- .../v3-core-plugins/extended-plugin-api.md | 51 ++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md index 8bffd5e5f6..0f3a75a255 100644 --- a/content/shared/v3-core-plugins/extended-plugin-api.md +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -1,10 +1,9 @@ -## Extend plugins with API features and state management - The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. -These features let you build more sophisticated plugins that can transform, analyze, and respond to data. +These features let you build plugins that can transform, analyze, and respond to data. The plugin API lets you: + - Write and query data directly from your python code - Track information between plugin executions with the in-memory cache - Log messages for monitoring and debugging @@ -14,7 +13,7 @@ Let's explore how to use these fatures in your pligins. ### Getting started with the shared API -Every plugin automatically has access to the shared API through the influxdb3_local object. You don't need to import any libraries - this API is available as soon as your plugin runs. +Every plugin has access to the shared API through the `influxdb3_local` object. You don't need to import any libraries. This API is available as soon as your plugin runs. #### Write data @@ -154,7 +153,7 @@ class LineBuilder: #### Query data -Your plugins can execute SQL queries and process the results directly: +Your plugins can execute SQL queries and process results directly: ```python # Simple query @@ -167,14 +166,10 @@ results = influxdb3_local.query("SELECT * FROM $table WHERE value > $threshold", Query results come back as a `List` of `Dict[String, Any]`, where each dictionary represents a row with column names as keys and column values as values. This makes it easy to process the results in your plugin code. - - #### Log information The shared API `info`, `warn`, and `error` functions accept multiple arguments, -convert them to strings, and log them as a space-separated message to the database log, -which is output in the server logs and captured in system tables that you can -query using SQL. +convert them to strings, and log them as a space-separated message to the database log. Add logging to track plugin execution: @@ -187,17 +182,19 @@ influxdb3_local.error("Failed to connect to external API") obj_to_log = {"records": 157, "errors": 3} influxdb3_local.info("Processing complete", obj_to_log) ``` +All log messages appear in the server logs and are stored in system tables that you can query using SQL. -#### Use the in-memory cache +#### Maintaining state with the in-memory cache The Processing engine provides an in-memory cache system that enables plugins to persist and retrieve data between executions. -Use the shared API `cache` property to access the cache API. +You can access the cache through the `cache` property of the shared API: ```python # Basic usage pattern influxdb3_local.cache.METHOD(PARAMETERS) ``` +Here are the available methods: | Method | Parameters | Returns | Description | |--------|------------|---------|-------------| @@ -205,7 +202,7 @@ influxdb3_local.cache.METHOD(PARAMETERS) | `get` | `key` (str): The key to retrieve
`default` (Any, default=None): Value to return if key not found
`use_global` (bool, default=False): If True, uses global namespace | Any | Retrieves a value from the cache or returns default if not found | | `delete` | `key` (str): The key to delete
`use_global` (bool, default=False): If True, uses global namespace | bool | Deletes a value from the cache. Returns True if deleted, False if not found | -##### Cache namespaces +##### Understanding cache namespaces The cache system offers two distinct namespaces: @@ -214,6 +211,9 @@ The cache system offers two distinct namespaces: | **Trigger-specific** (default) | Isolated to a single trigger | Plugin state, counters, timestamps specific to one plugin | | **Global** | Shared across all triggers | Configuration, lookup tables, service states that should be available to all plugins | +### Common cache operations +Here are some examples of how to use the cache in your plugins + ##### Store and retrieve cached data ```python @@ -243,8 +243,9 @@ influxdb3_local.cache.put("config", {"version": "1.0"}, use_global=True) # Retrieve from the global namespace config = influxdb3_local.cache.get("config", use_global=True) ``` +#### Building a counter -##### Track state between executions +You can track how many times a plugin has run: ```python # Get current counter or default to 0 @@ -261,6 +262,8 @@ influxdb3_local.info(f"This plugin has run {counter} times") #### Best practices for in-memory caching +To get the most out of the in-memory cache, follow these guidlines: + - [Use the trigger-specific namespace](#use-the-trigger-specific-namespace) - [Use TTL appropriately](#use-ttl-appropriately) - [Cache computation results](#cache-computation-results) @@ -273,7 +276,7 @@ The cache is designed to support stateful operations while maintaining isolation ##### Use TTL appropriately -Set realistic expiration times based on how frequently data changes. +Set realistic expiration times based on how frequently data changes: ```python # Cache external API responses for 5 minutes @@ -282,7 +285,8 @@ influxdb3_local.cache.put("weather_data", api_response, ttl=300) ##### Cache computation results -Store the results of expensive calculations that need to be utilized frequently. +Store the results of expensive calculations that need to be utilized frequently: + ```python # Cache aggregated statistics influxdb3_local.cache.put("daily_stats", calculate_statistics(data), ttl=3600) @@ -290,7 +294,7 @@ influxdb3_local.cache.put("daily_stats", calculate_statistics(data), ttl=3600) ##### Warm the cache -For critical data, prime the cache at startup. This can be especially useful for global namespace data where multiple triggers need the data. +For critical data, prime the cache at startup. This can be especially useful for global namespace data where multiple triggers need the data: ```python # Check if cache needs to be initialized @@ -302,4 +306,15 @@ if not influxdb3_local.cache.get("lookup_table"): - **Memory Usage**: Since cache contents are stored in memory, monitor your memory usage when caching large datasets. - **Server Restarts**: Because the cache is cleared when the server restarts, design your plugins to handle cache initialization (as noted above). -- **Concurrency**: Be cautious of accessing inaccurate or out-of-date data when multiple trigger instances might simultaneously update the same cache key. \ No newline at end of file +- **Concurrency**: Be cautious of accessing inaccurate or out-of-date data when multiple trigger instances might simultaneously update the same cache key. + +### Next Steps + +Now that you understand the Extended Plugin API, you're ready to build data processing workflows that can transform, analyze, and respond to your time series data. + +Try combining these features to create plugins that: + +- Process incoming data and write transformed results +- Maintain state between executions using the cache +- Log detailed information about their operation +- Share configuration through the global cache \ No newline at end of file From 33e9e9df23252796eb146b186702298daa3ef34f Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 17:11:09 -0700 Subject: [PATCH 19/51] Update content/shared/v3-core-plugins/extended-plugin-api.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- content/shared/v3-core-plugins/extended-plugin-api.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md index 0f3a75a255..8ba77702c0 100644 --- a/content/shared/v3-core-plugins/extended-plugin-api.md +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -9,8 +9,7 @@ The plugin API lets you: - Log messages for monitoring and debugging - Build data processing workflows -Let's explore how to use these fatures in your pligins. - +Let's explore how to use these features in your plugins. ### Getting started with the shared API Every plugin has access to the shared API through the `influxdb3_local` object. You don't need to import any libraries. This API is available as soon as your plugin runs. From 10c82daacac0d1742f282fdfda93d9844b4b1649 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:17:38 -0700 Subject: [PATCH 20/51] Update content/influxdb3/core/extended-plugin.md Co-authored-by: Jason Stirnaman --- content/influxdb3/core/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md index f3fb30ecf2..046cbfa447 100644 --- a/content/influxdb3/core/extended-plugin.md +++ b/content/influxdb3/core/extended-plugin.md @@ -1,7 +1,7 @@ --- title: Extend plugins with API features and state management description: | - The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. + The Processing engine includes API capabilities that allow your plugins to interact with your data and maintain state between executions. menu: influxdb3_core: name: Extended plugins From e90656e0430f134d5f99111273feed1fbfe3fbab Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:17:56 -0700 Subject: [PATCH 21/51] Update content/influxdb3/core/extended-plugin.md Co-authored-by: Jason Stirnaman --- content/influxdb3/core/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md index 046cbfa447..da5b625031 100644 --- a/content/influxdb3/core/extended-plugin.md +++ b/content/influxdb3/core/extended-plugin.md @@ -12,5 +12,5 @@ source: /shared/v3-core-plugins/extended-plugin-api.md --- \ No newline at end of file From 2e11cbaaec1cb2d54923968cd5078e9e159e6b9f Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:18:09 -0700 Subject: [PATCH 22/51] Update content/influxdb3/enterprise/extended-plugin.md Co-authored-by: Jason Stirnaman --- content/influxdb3/enterprise/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md index f7e99a6d23..6a76f3a8c6 100644 --- a/content/influxdb3/enterprise/extended-plugin.md +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -1,7 +1,7 @@ --- title: Extend plugins with API features and state management description: | - The Processing engine includes API capabilities that allow your plugins to interact with InfluxDB data and maintain state between executions. + The Processing engine includes API capabilities that allow your plugins to interact with your data and maintain state between executions. menu: influxdb3_enterprise: name: Extended plugins From c1a1ba80b41e0e4a0c7d3fc12f5cdb1c34d0642b Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:18:24 -0700 Subject: [PATCH 23/51] Update content/influxdb3/enterprise/extended-plugin.md Co-authored-by: Jason Stirnaman --- content/influxdb3/enterprise/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md index 6a76f3a8c6..b23ded5ee8 100644 --- a/content/influxdb3/enterprise/extended-plugin.md +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -12,5 +12,5 @@ source: /shared/v3-core-plugins/extended-plugin-api.md --- \ No newline at end of file From 22c9617ef487dee8cd1b6cebc5e9a1ad80ef25f0 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:19:03 -0700 Subject: [PATCH 24/51] Update content/shared/influxdb3-processing-engine.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index 81fec66323..7b695fe944 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -12,7 +12,7 @@ The Processing engine runs Python code directly within a {{% product-name %}} se - **Direct data access**: Zero-copy access to data - **Event-driven**: Responds to database writes, scheduled events, and HTTP requests - **Isolated contexts**: Maintains separation between different plugin executions -- **Cache integration**: Access to system caches including Last values and Distinct values +- **Cache integration**: Access to system caches, such as Last Value Cache and Distinct Value Cache ### Event processing flow From 8f543a6052abaeb432a793690988bdcbcc6d0a1f Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:19:38 -0700 Subject: [PATCH 25/51] Update content/shared/influxdb3-processing-engine.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index 7b695fe944..c2977743cd 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -18,7 +18,7 @@ The Processing engine runs Python code directly within a {{% product-name %}} se When events occur in the database, the Processing engine handles them through a consistent sequence: -1. A **trigger** activates the plugin based on one of three event types: +1. A _trigger_ configures execution parameters and executes the plugin based on one of three event types: - Data writes to specific tables or all tables - Scheduled events (time-based or cron expressions) - HTTP requests to configured endpoints From 851fa2c9f514d0112ea5b16eceac811f35a46ba2 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:19:48 -0700 Subject: [PATCH 26/51] Update content/shared/influxdb3-processing-engine.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-processing-engine.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index c2977743cd..13c5e2757d 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -27,6 +27,7 @@ When events occur in the database, the Processing engine handles them through a - Write triggers: the written data and table information - Schedule triggers: the scheduled call time - HTTP triggers: the request object with methods, headers, and body + The plugin also receives any keyword arguments that you pass as `trigger-arguments`. 4. The plugin processes the received data, can query the database, call external tools, and write results back 5. Execution completes and the engine returns to waiting state From 811cf2d2dcc6ceab568ce39eb242053fedf935e3 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:20:10 -0700 Subject: [PATCH 27/51] Update content/shared/influxdb3-processing-engine.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index 13c5e2757d..158c029ba5 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -28,7 +28,7 @@ When events occur in the database, the Processing engine handles them through a - Schedule triggers: the scheduled call time - HTTP triggers: the request object with methods, headers, and body The plugin also receives any keyword arguments that you pass as `trigger-arguments`. -4. The plugin processes the received data, can query the database, call external tools, and write results back +4. When the plugin runs, it can process the received data and arguments, access the shared API through `influxdb3_local` for writing, querying, and caching data, and integrate with external systems. 5. Execution completes and the engine returns to waiting state ## Key components From f65b3378d1f430d82b690263ddaa585288b7738f Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:20:22 -0700 Subject: [PATCH 28/51] Update content/shared/influxdb3-processing-engine.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index 158c029ba5..313357004a 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -29,7 +29,7 @@ When events occur in the database, the Processing engine handles them through a - HTTP triggers: the request object with methods, headers, and body The plugin also receives any keyword arguments that you pass as `trigger-arguments`. 4. When the plugin runs, it can process the received data and arguments, access the shared API through `influxdb3_local` for writing, querying, and caching data, and integrate with external systems. -5. Execution completes and the engine returns to waiting state +5. Execution completes and the engine returns to waiting state. ## Key components From e4eb00c7281bb0ee4a811fe90c869e01ea5b4fde Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:20:43 -0700 Subject: [PATCH 29/51] Update content/shared/v3-core-plugins/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/v3-core-plugins/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md index 8ba77702c0..5254989b8b 100644 --- a/content/shared/v3-core-plugins/extended-plugin-api.md +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -4,7 +4,7 @@ These features let you build plugins that can transform, analyze, and respond to The plugin API lets you: -- Write and query data directly from your python code +- Write and query data directly from your Python code - Track information between plugin executions with the in-memory cache - Log messages for monitoring and debugging - Build data processing workflows From 464a816f26547c0ebf07d6d781b002c4e42aa963 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 14 Apr 2025 19:20:53 -0700 Subject: [PATCH 30/51] Update content/shared/v3-core-plugins/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/v3-core-plugins/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md index 5254989b8b..131ee0fa19 100644 --- a/content/shared/v3-core-plugins/extended-plugin-api.md +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -1,5 +1,5 @@ The Processing engine includes API capabilities that allow your plugins to -interact with InfluxDB data and maintain state between executions. +interact with your data and maintain state between executions. These features let you build plugins that can transform, analyze, and respond to data. The plugin API lets you: From c099310efcae6c5daca292f01604d63f43f67216 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 16 Apr 2025 09:12:32 -0700 Subject: [PATCH 31/51] Update content/shared/influxdb3-processing-engine.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index 313357004a..842b030f60 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -22,7 +22,7 @@ When events occur in the database, the Processing engine handles them through a - Data writes to specific tables or all tables - Scheduled events (time-based or cron expressions) - HTTP requests to configured endpoints -2. The engine loads the associated **plugin** specified in the trigger configuration +2. The engine loads the associated plugin specified in the trigger configuration 3. The plugin receives context data specific to the trigger type: - Write triggers: the written data and table information - Schedule triggers: the scheduled call time From 78d8a431d7aba3d9d7fa4fa441b7458232f344fe Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 16 Apr 2025 12:14:19 -0700 Subject: [PATCH 32/51] updating cache integration --- content/shared/influxdb3-processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index 842b030f60..e29397848e 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -12,7 +12,7 @@ The Processing engine runs Python code directly within a {{% product-name %}} se - **Direct data access**: Zero-copy access to data - **Event-driven**: Responds to database writes, scheduled events, and HTTP requests - **Isolated contexts**: Maintains separation between different plugin executions -- **Cache integration**: Access to system caches, such as Last Value Cache and Distinct Value Cache +- **Cache integration**: Access to system caches, such as [Last Value Cache](v3-core-get-started/_index.md/#last-values-cache) and [Distinct Value Cache]() ### Event processing flow From 29d9186d6f216d588cdfa428e91242c812f704a4 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 16 Apr 2025 12:22:19 -0700 Subject: [PATCH 33/51] update to cache integrations --- content/shared/influxdb3-processing-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-processing-engine.md b/content/shared/influxdb3-processing-engine.md index e29397848e..111931a59f 100644 --- a/content/shared/influxdb3-processing-engine.md +++ b/content/shared/influxdb3-processing-engine.md @@ -12,7 +12,7 @@ The Processing engine runs Python code directly within a {{% product-name %}} se - **Direct data access**: Zero-copy access to data - **Event-driven**: Responds to database writes, scheduled events, and HTTP requests - **Isolated contexts**: Maintains separation between different plugin executions -- **Cache integration**: Access to system caches, such as [Last Value Cache](v3-core-get-started/_index.md/#last-values-cache) and [Distinct Value Cache]() +- **Cache integration**: Access to system caches, such as [Last Value Cache](/influxdb3/core/get-started/#last-values-cache) and [Distinct Value Cache](/influxdb3/core/get-started/#distinct-values-cache) ### Event processing flow From 8f4d914a7d58dc2286bf826eee921abdd92db52e Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 24 Apr 2025 18:11:23 -0700 Subject: [PATCH 34/51] Update content/shared/v3-core-plugins/extended-plugin-api.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- content/shared/v3-core-plugins/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/v3-core-plugins/extended-plugin-api.md index 131ee0fa19..90451369dc 100644 --- a/content/shared/v3-core-plugins/extended-plugin-api.md +++ b/content/shared/v3-core-plugins/extended-plugin-api.md @@ -261,7 +261,7 @@ influxdb3_local.info(f"This plugin has run {counter} times") #### Best practices for in-memory caching -To get the most out of the in-memory cache, follow these guidlines: +To get the most out of the in-memory cache, follow these guidelines: - [Use the trigger-specific namespace](#use-the-trigger-specific-namespace) - [Use TTL appropriately](#use-ttl-appropriately) From a56a8d87b33d00ca79554f84774977272d267617 Mon Sep 17 00:00:00 2001 From: meelahme Date: Thu, 24 Apr 2025 18:38:23 -0700 Subject: [PATCH 35/51] updating file paths --- content/influxdb3/core/extended-plugin.md | 7 ++++--- content/influxdb3/enterprise/extended-plugin.md | 4 ++-- .../shared/{v3-core-plugins => }/extended-plugin-api.md | 0 content/shared/v3-core-plugins/_index.md | 2 -- 4 files changed, 6 insertions(+), 7 deletions(-) rename content/shared/{v3-core-plugins => }/extended-plugin-api.md (100%) diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md index da5b625031..3e7ed886d2 100644 --- a/content/influxdb3/core/extended-plugin.md +++ b/content/influxdb3/core/extended-plugin.md @@ -8,9 +8,10 @@ menu: parent: Processing engine and Python plugins weight: 4 influxdb3/core/tags: [processing engine, plugins, API, python] -source: /shared/v3-core-plugins/extended-plugin-api.md +source: /shared/extended-plugin-api.md --- \ No newline at end of file +// SOURCE content/shared/extended-plugin-api.md +--> + diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md index b23ded5ee8..08fce6e602 100644 --- a/content/influxdb3/enterprise/extended-plugin.md +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -8,9 +8,9 @@ menu: parent: Processing engine and Python plugins weight: 4 influxdb3/enterprise/tags: [processing engine, plugins, API, python] -source: /shared/v3-core-plugins/extended-plugin-api.md +source: /shared/extended-plugin-api.md --- \ No newline at end of file diff --git a/content/shared/v3-core-plugins/extended-plugin-api.md b/content/shared/extended-plugin-api.md similarity index 100% rename from content/shared/v3-core-plugins/extended-plugin-api.md rename to content/shared/extended-plugin-api.md diff --git a/content/shared/v3-core-plugins/_index.md b/content/shared/v3-core-plugins/_index.md index b1965ee86f..006e89d29b 100644 --- a/content/shared/v3-core-plugins/_index.md +++ b/content/shared/v3-core-plugins/_index.md @@ -1,5 +1,3 @@ -# Get Started with the Processing Engine and Plugins - Extend InfluxDB 3 with custom Python code that responds to database events. The Processing Engine lets you automate workflows, transform data, and create API endpoints directly within your {{% product-name %}}. ## What is the Processing Engine? From 26084afe4b89fb0bf8f391666b561da199589d09 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:13:19 -0700 Subject: [PATCH 36/51] Update content/influxdb3/core/extended-plugin.md Co-authored-by: Jason Stirnaman --- content/influxdb3/core/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extended-plugin.md index 3e7ed886d2..8f8516f9a8 100644 --- a/content/influxdb3/core/extended-plugin.md +++ b/content/influxdb3/core/extended-plugin.md @@ -1,7 +1,7 @@ --- title: Extend plugins with API features and state management description: | - The Processing engine includes API capabilities that allow your plugins to interact with your data and maintain state between executions. + The Processing engine includes an API that allows your plugins to interact with your data, build and write line protocol, and maintain state between executions. menu: influxdb3_core: name: Extended plugins From 39722d5bf326413327d0c3bfeff13757ce17b366 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:13:33 -0700 Subject: [PATCH 37/51] Update content/influxdb3/enterprise/extended-plugin.md Co-authored-by: Jason Stirnaman --- content/influxdb3/enterprise/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md index 08fce6e602..35f3a0d49e 100644 --- a/content/influxdb3/enterprise/extended-plugin.md +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -1,7 +1,7 @@ --- title: Extend plugins with API features and state management description: | - The Processing engine includes API capabilities that allow your plugins to interact with your data and maintain state between executions. + The Processing engine includes an API that allows your plugins to interact with your data, build and write line protocol, and maintain state between executions. menu: influxdb3_enterprise: name: Extended plugins From 391538fafb6744831480fb0806be3f9ecd9045a0 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:13:47 -0700 Subject: [PATCH 38/51] Update content/influxdb3/enterprise/extended-plugin.md Co-authored-by: Jason Stirnaman --- content/influxdb3/enterprise/extended-plugin.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extended-plugin.md index 35f3a0d49e..c4752fa97f 100644 --- a/content/influxdb3/enterprise/extended-plugin.md +++ b/content/influxdb3/enterprise/extended-plugin.md @@ -4,7 +4,7 @@ description: | The Processing engine includes an API that allows your plugins to interact with your data, build and write line protocol, and maintain state between executions. menu: influxdb3_enterprise: - name: Extended plugins + name: Extend plugins parent: Processing engine and Python plugins weight: 4 influxdb3/enterprise/tags: [processing engine, plugins, API, python] From 88c7cb0af7abc3f61c63669c93f2a24f1aecdbfb Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:14:05 -0700 Subject: [PATCH 39/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index 90451369dc..fb7fba1471 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -1,4 +1,4 @@ -The Processing engine includes API capabilities that allow your plugins to +The Processing engine includes an API that allows your plugins to interact with your data and maintain state between executions. These features let you build plugins that can transform, analyze, and respond to data. From 63aeab7fe44f7431655fe7dc80cb63da306c702f Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:14:16 -0700 Subject: [PATCH 40/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index fb7fba1471..975ac3398e 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -1,5 +1,5 @@ The Processing engine includes an API that allows your plugins to -interact with your data and maintain state between executions. +interact with your data, build and write data in line protocol format, and maintain state between executions. These features let you build plugins that can transform, analyze, and respond to data. The plugin API lets you: From 377604e308d8d8ca4b5069dcdbeaa18aefcddb34 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:14:40 -0700 Subject: [PATCH 41/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index 975ac3398e..0ec0f44500 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -9,7 +9,6 @@ The plugin API lets you: - Log messages for monitoring and debugging - Build data processing workflows -Let's explore how to use these features in your plugins. ### Getting started with the shared API Every plugin has access to the shared API through the `influxdb3_local` object. You don't need to import any libraries. This API is available as soon as your plugin runs. From a775edd612fd24f68721b17048e416abbd2b4480 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:14:53 -0700 Subject: [PATCH 42/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index 0ec0f44500..a58a5f5b0c 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -9,7 +9,7 @@ The plugin API lets you: - Log messages for monitoring and debugging - Build data processing workflows -### Getting started with the shared API +### Get started with the shared API Every plugin has access to the shared API through the `influxdb3_local` object. You don't need to import any libraries. This API is available as soon as your plugin runs. From d3af0d4dc824a21365320197ac923b557512dacb Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:15:04 -0700 Subject: [PATCH 43/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index a58a5f5b0c..2f50458967 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -11,7 +11,7 @@ The plugin API lets you: ### Get started with the shared API -Every plugin has access to the shared API through the `influxdb3_local` object. You don't need to import any libraries. This API is available as soon as your plugin runs. +Every plugin has access to the shared API through the `influxdb3_local` object. You don't need to import any libraries to use the API. It's available as soon as your plugin runs. #### Write data From 4825b91c8ac9f25a32d6c86568c183a806aa511d Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:15:18 -0700 Subject: [PATCH 44/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index 2f50458967..d1dd691f99 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -162,7 +162,7 @@ params = {"table": "metrics", "threshold": 90} results = influxdb3_local.query("SELECT * FROM $table WHERE value > $threshold", params) ``` -Query results come back as a `List` of `Dict[String, Any]`, where each dictionary represents a row with column names as keys and column values as values. This makes it easy to process the results in your plugin code. +Query results are a `List` of `Dict[String, Any]`, where each dictionary represents a row with column names as keys and column values as values. #### Log information From 740dd2b4a4912fd16d7ee2c1b005e81bce200eaa Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:15:29 -0700 Subject: [PATCH 45/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index d1dd691f99..dc39b83e7c 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -182,7 +182,7 @@ influxdb3_local.info("Processing complete", obj_to_log) ``` All log messages appear in the server logs and are stored in system tables that you can query using SQL. -#### Maintaining state with the in-memory cache +#### Maintain state with the in-memory cache The Processing engine provides an in-memory cache system that enables plugins to persist and retrieve data between executions. From 2f77fca66945de02a2e1c1d2dc1ef80a71f11f3f Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:15:40 -0700 Subject: [PATCH 46/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index dc39b83e7c..9ccf4d1b08 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -192,7 +192,8 @@ You can access the cache through the `cache` property of the shared API: # Basic usage pattern influxdb3_local.cache.METHOD(PARAMETERS) ``` -Here are the available methods: + +`cache` provides the following methods to retrieve and manage cached values: | Method | Parameters | Returns | Description | |--------|------------|---------|-------------| From 571c3e5f37f0d370b3e0fc6206281fe7e242d3e8 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 14:15:57 -0700 Subject: [PATCH 47/51] Update content/shared/extended-plugin-api.md Co-authored-by: Jason Stirnaman --- content/shared/extended-plugin-api.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index 9ccf4d1b08..2272a0fc54 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -309,11 +309,4 @@ if not influxdb3_local.cache.get("lookup_table"): ### Next Steps -Now that you understand the Extended Plugin API, you're ready to build data processing workflows that can transform, analyze, and respond to your time series data. - -Try combining these features to create plugins that: - -- Process incoming data and write transformed results -- Maintain state between executions using the cache -- Log detailed information about their operation -- Share configuration through the global cache \ No newline at end of file +With an understanding of the InfluxDB 3 Shared Plugin API, you're ready to build data processing workflows that can transform, analyze, and respond to your time series data or extend example plugins from the [plugin repo]() on GitHub. \ No newline at end of file From 9ac78d95cc4bef477b030d8062060b0aa49238d8 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 28 Apr 2025 14:20:50 -0700 Subject: [PATCH 48/51] updating file names --- content/influxdb3/core/{extended-plugin.md => extend-plugin.md} | 0 .../influxdb3/enterprise/{extended-plugin.md => extend-plugin.md} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename content/influxdb3/core/{extended-plugin.md => extend-plugin.md} (100%) rename content/influxdb3/enterprise/{extended-plugin.md => extend-plugin.md} (100%) diff --git a/content/influxdb3/core/extended-plugin.md b/content/influxdb3/core/extend-plugin.md similarity index 100% rename from content/influxdb3/core/extended-plugin.md rename to content/influxdb3/core/extend-plugin.md diff --git a/content/influxdb3/enterprise/extended-plugin.md b/content/influxdb3/enterprise/extend-plugin.md similarity index 100% rename from content/influxdb3/enterprise/extended-plugin.md rename to content/influxdb3/enterprise/extend-plugin.md From 0ad66f563713e0bf7c053165f17600e6bf55171d Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 28 Apr 2025 14:59:57 -0700 Subject: [PATCH 49/51] creating a linked TOC --- content/shared/extended-plugin-api.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index 2272a0fc54..b8f5244e63 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -4,15 +4,17 @@ These features let you build plugins that can transform, analyze, and respond to The plugin API lets you: -- Write and query data directly from your Python code -- Track information between plugin executions with the in-memory cache -- Log messages for monitoring and debugging -- Build data processing workflows +- [Write and query data](#write-and-query-data) +- [Log messages for monitoring and debugging](#log-messages-for-monitoring-and-debugging) +- [Maintain state with in-memory cache](#maintain-state-with-in-memory-cache) +- [Guidelines for in-memory caching](#guidelines-for-in-memory-caching) ### Get started with the shared API Every plugin has access to the shared API through the `influxdb3_local` object. You don't need to import any libraries to use the API. It's available as soon as your plugin runs. +### Write and query data + #### Write data To write data into your database use the `LineBuilder` API to create line protocol data: @@ -164,7 +166,7 @@ results = influxdb3_local.query("SELECT * FROM $table WHERE value > $threshold", Query results are a `List` of `Dict[String, Any]`, where each dictionary represents a row with column names as keys and column values as values. -#### Log information +### Log messages for monitoring and debugging The shared API `info`, `warn`, and `error` functions accept multiple arguments, convert them to strings, and log them as a space-separated message to the database log. @@ -182,7 +184,7 @@ influxdb3_local.info("Processing complete", obj_to_log) ``` All log messages appear in the server logs and are stored in system tables that you can query using SQL. -#### Maintain state with the in-memory cache +### Maintain state with in-memory cache The Processing engine provides an in-memory cache system that enables plugins to persist and retrieve data between executions. @@ -259,7 +261,7 @@ influxdb3_local.cache.put("execution_count", counter) influxdb3_local.info(f"This plugin has run {counter} times") ``` -#### Best practices for in-memory caching +### Guidelines for in-memory caching To get the most out of the in-memory cache, follow these guidelines: From 13554450b26e63526c4906e5d449663a7cbba879 Mon Sep 17 00:00:00 2001 From: meelahme Date: Mon, 28 Apr 2025 15:37:57 -0700 Subject: [PATCH 50/51] Sharing a link to system table information --- content/shared/extended-plugin-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index b8f5244e63..e018926f21 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -182,7 +182,7 @@ influxdb3_local.error("Failed to connect to external API") obj_to_log = {"records": 157, "errors": 3} influxdb3_local.info("Processing complete", obj_to_log) ``` -All log messages appear in the server logs and are stored in system tables that you can query using SQL. +All log messages are written to the server logs and stored in [system tables](/influxdb3/core/reference/cli/influxdb3/show/system/summary/), where you can query then using SQL. ### Maintain state with in-memory cache From f742d520a6703e22b05490045d5a52c4c7b2e7ea Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Mon, 28 Apr 2025 15:49:24 -0700 Subject: [PATCH 51/51] Update content/shared/extended-plugin-api.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- content/shared/extended-plugin-api.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/shared/extended-plugin-api.md b/content/shared/extended-plugin-api.md index e018926f21..8e65ec07a0 100644 --- a/content/shared/extended-plugin-api.md +++ b/content/shared/extended-plugin-api.md @@ -182,8 +182,7 @@ influxdb3_local.error("Failed to connect to external API") obj_to_log = {"records": 157, "errors": 3} influxdb3_local.info("Processing complete", obj_to_log) ``` -All log messages are written to the server logs and stored in [system tables](/influxdb3/core/reference/cli/influxdb3/show/system/summary/), where you can query then using SQL. - +All log messages are written to the server logs and stored in [system tables](/influxdb3/core/reference/cli/influxdb3/show/system/summary/), where you can query them using SQL. ### Maintain state with in-memory cache The Processing engine provides an in-memory cache system that enables plugins to persist and retrieve data between executions.