add dynamic document pipeline support for logs #37860
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carsonip
reviewed
Feb 12, 2025
| @@ -145,6 +145,9 @@ This can be customised through the following settings: | |||
| - `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute. | |||
| - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. See [Setting a document id dynamically](#setting-a-document-id-dynamically). | |||
|
|
|||
| - `logs_dynamic_pipeline` (optional): Dynamically determines the ingest pipeline to be used in Elasticsearch based on a log record attribute. | |||
| - `enabled`(default=false): Enable/Disable dynamic pipeline for log records. If `elasticsearch.document_pipeline` exists and is not an empty string in the log record attributes, it will be used as the Elasticsearch ingest pipeline. The attribute `elasticsearch.document_pipeline` is removed from the final document. | |||
There was a problem hiding this comment.
The attribute
elasticsearch.document_pipelineis removed from the final document.
nit: While this is true for OTel mode, it doesn't seem to be the case for other modes. I'm fine with that, but we should state that in the documentation. (I think I either missed this for elasticsearch.document_id in its PR review or something has changed since then)
carsonip
approved these changes
Feb 13, 2025
- Added caveat that the attributes are only removed when `otel` mapping mode is used
leehinman
force-pushed
the
37419_elasticsearch_pipeline_id
branch
from
642ff7f to
8ef035c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new config option
logs_dynamic_pipelinethat when set totrue reads the
elasticsearch.document_pipelineattribute from each logrecord and uses it as the ingest pipeline in Elasticsearch. This is
only implemented for logs, but a subsequent PR supporting metrics and
traces could be opened.
Link to tracking issue
Fixes #37419
Testing
Added tests to verify that the document pipeline attribute can be read
from the log record and that the pipeline is properly forwarded to
Elasticsearch. Also asserted that when there is no document pipeline
attribute the current behavior is retained.
Manual testing that setting the option resulted in the named ingest
pipeline being run on the document in Elasticsearch.
Documentation
Updated the readme to mention the new
logs_dynamic_pipelineconfig option.