Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .chloggen/dylan_azureeventhub-beta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog)
component: receiver/azure_event_hub

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Promote the Azure Event Hub receiver to beta stability.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [41661]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: []
165 changes: 93 additions & 72 deletions receiver/azureeventhubreceiver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Stability | [alpha]: metrics, logs, traces |
| Stability | [beta]: metrics, logs, traces |
| Distributions | [contrib] |
| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fazureeventhub%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fazureeventhub) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fazureeventhub%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fazureeventhub) |
| Code coverage | [![codecov](https://codecov.io/github/open-telemetry/opentelemetry-collector-contrib/graph/main/badge.svg?component=receiver_azure_event_hub)](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=receiver_azure_event_hub&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@atoulme](https://www.github.com/atoulme), [@cparkins](https://www.github.com/cparkins), [@dyl10s](https://www.github.com/dyl10s) \| Seeking more code owners! |
| Emeritus | [@djaglowski](https://www.github.com/djaglowski) |

[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
<!-- end autogenerated section -->

Expand All @@ -28,108 +28,129 @@ Event Hub, transforms them, and pushes them through the collector pipeline.

## Configuration

### connection (Required if auth is not used)
A string describing the connection to an Azure event hub. Ignored if `auth` is specified.
### Connection Settings

### event_hub
This section is required when using `auth`.
Either `connection` or `auth` must be specified.

#### name (Required when using auth)
The name of the Event Hub.
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `connection` | string | Yes* | | Connection string from the Azure portal. Ignored if `auth` is set. |
| `auth` | string | Yes* | | ID of an authentication extension (AAD, managed identity, or service principal). |
| `event_hub.name` | string | ** | | Event Hub name. Required when using `auth`. |
| `event_hub.namespace` | string | ** | | Fully qualified namespace (e.g., `namespace.servicebus.windows.net`). Required when using `auth`. |

#### namespace (Required when using auth)
The fully qualified namespace (e.g., `namespace.servicebus.windows.net`).
\* One of `connection` or `auth` is required.
\** Required when using `auth`.

### auth (Optional)
The ID of an authentication extension to use. This can be used to authenticate using Azure Active Directory (AAD) pod identity,
managed identity, or service principal.
When this field is set, `connection` is ignored and `event_hub` section is required.
### Consumer Settings

### group (Optional)
The Consumer Group to read from. If empty will default to the default Consumer Group $Default
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `group` | string | No | `$Default` | Consumer group to read from. |
| `partition` | string | No | `""` | Specific partition to watch. If empty, watches all partitions. |
| `offset` | string | No | `""` | Starting offset. Only applicable when `partition` is set. |
| `storage` | string | No | | ID of a [storage extension] for checkpoint persistence. |

### partition (Optional)
The partition to watch. If empty, it will watch explicitly all partitions.
### Polling Settings

Default: ""
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `max_poll_events` | int | No | `100` | Maximum events to retrieve per poll. |
| `poll_rate` | int | No | `5` | Maximum seconds to wait before returning fewer than `max_poll_events`. |

### offset (Optional)
The offset at which to start watching the event hub. If empty, it starts with the latest offset.
Only applicable when `partition` is set.
### Data Transformation Settings

Default: ""
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `format` | string | No | `azure` | Message format: `azure`, `raw`, or `""`. See [Format](#format) section. |
| `apply_semantic_conventions` | bool | No | `false` | Translate Azure Resource Logs using OpenTelemetry semantic convention attribute names. |
| `time_formats.logs` | []string | No | | Custom time formats for logs. Uses [Go time layout](https://pkg.go.dev/time#Layout). Falls back to ISO8601. |
| `time_formats.metrics` | []string | No | | Custom time formats for metrics. |
| `time_formats.traces` | []string | No | | Custom time formats for traces. |
| `metric_aggregation` | string | No | | Set to `average` to aggregate datapoints as `sum/count`. By default, creates separate metrics with suffixes (`_TOTAL`, `_MIN`, etc.). |

### format (Optional)
Determines how to transform the Event Hub messages into OpenTelemetry logs. See the "Format"
section below for details.

Default: "azure"

### apply_semantic_conventions (optional)
Determines whether Azure Resource Logs are translated into OpenTelemetry Logs using semantic
convention attribute names or not. When not applying semantic conventions, the log entry
attribute names are copied without any changes.

Default: `false` (semantic conventions are not applied)
### Example Configuration

### time_formats (optional)
The simplest configuration uses a connection string from the Azure portal:

All supported time format for logs, metrics and traces. Default is `nil` (unset), which means using the current iso8601 parser. The format is based on https://pkg.go.dev/time#Layout. If no time-zone info, will use UTC time. If all failed, it will use iso8601 format to parse.
```yaml
receivers:
azure_event_hub:
connection: Endpoint=sb://<namespace>.servicebus.windows.net/;SharedAccessKeyName=<key-name>;SharedAccessKey=<key>;EntityPath=<hub-name>
group: $Default # optional, defaults to $Default
```

Default: `nil`
The full list of settings exposed for this receiver are documented above with detailed sample configurations in [testdata/config.yaml](./testdata/config.yaml).

### metric_aggregation (optional)
### Advanced Configuration

Metric records received from an Azure Event Hub will contain multiple aggregated datapoints. Depending on the
[type of metric](https://learn.microsoft.com/en-us/azure/azure-monitor/metrics/data-platform-metrics#types-of-metrics),
these datapoints will be slightly different, but all contain a total/sum, min, max and count. This setting will manage
these datapoints.
#### Azure Auth Extension with Service Principal

#### Possible values:
```yaml
receivers:
azure_event_hub:
event_hub:
name: hubName
namespace: namespace.servicebus.windows.net
auth: azureauth

* By default, these datapoints will be mapped to metrics named with a corresponding suffix, like `_total`, `_min`, etc.
See [azure format](#azure).
* With `average`, datapoints will be aggregated into an average value (`sum/count`), and keep the original metric name.
extensions:
azureauth:
service_principal:
client_id: ${env:AZURE_CLIENT_ID}
client_secret: ${env:AZURE_CLIENT_SECRET}
tenant_id: ${env:AZURE_TENANT_ID}
```

Default: `nil`
#### Checkpoint Persistence with Storage Extension

### max_poll_events (optional)
Specifies the maximum number of events to retrieve in a single poll from the Event Hub.
This component can persist its state (checkpoint offsets) using the [storage extension]. This is **strongly recommended** for production deployments to prevent message reprocessing after collector restarts.

Default: `100`
Without a storage extension configured, the receiver will start from the latest offset on each restart, potentially missing messages that arrived while the collector was down.

### poll_rate (optional)
Specifies the maximum number of seconds to wait for additional events before returning fewer than `max_poll_events`.
```yaml
receivers:
azure_event_hub:
connection: Endpoint=sb://namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=superSecret1234=;EntityPath=hubName
storage: file_storage

Default: `5`
extensions:
file_storage:
directory: /var/lib/otelcol/eventhub
```

### Example Configuration
#### Custom Time Formats and Partitioning

```yaml
receivers:
azure_event_hub:
connection: Endpoint=sb://namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=superSecret1234=;EntityPath=hubName
partition: foo
group: bar
partition: "0"
group: my-consumer-group
offset: "1234-5566"
format: "azure"
# optional
time_formats:
# All supported time format. Default is empty string array, which means using the current iso8601 parser. The format is based on https://pkg.go.dev/time#Layout. If no time-zone info, will use UTC time.
logs: ["01/02/2006 15:04:05","2006-01-02 15:04:05","2006-01-02T15:04:05Z07:00"]
logs: ["01/02/2006 15:04:05", "2006-01-02 15:04:05"]
metrics: ["01/02/2006 15:04:05"]

# Example with Auth
azure_event_hub/auth:
event_hub:
name: hubName
namespace: namespace.servicebus.windows.net
auth: azureauth
partition: foo
group: bar
```

This component can persist its state using the [storage extension].
## Feature Gates

The following feature gates are available for this receiver:

### `receiver.azureeventhubreceiver.UseAzeventhubs`

| Status | Default | From Version | To Version |
|--------|---------|--------------|------------|
| Stable | `true` | v0.129.0 | v0.144.0 |

## Known Limitations

Before using this receiver, be aware of the following limitations:

### Checkpoint Persistence

- Without a configured [storage extension], checkpoint state is stored only in memory. If the collector restarts, the receiver will resume from the latest offset, potentially missing messages or reprocessing already-consumed messages.

## Format

Expand All @@ -139,8 +160,8 @@ The "raw" format maps the AMQP properties and data into the
attributes and body of an OpenTelemetry LogRecord, respectively.
The body is represented as a raw byte array.

> [!WARN]
> This format is not supported for Metrics.
> [!WARNING]
> This format is **only supported for Logs**. Using `raw` format with Metrics or Traces pipelines will result in an error.

### azure

Expand Down
2 changes: 1 addition & 1 deletion receiver/azureeventhubreceiver/feature_flags_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ func TestAzEventHubFeatureGateRegistration(t *testing.T) {
t.Errorf("expected gate ID %q, got %q", azEventHubFeatureGateName, azEventHubFeatureGate.ID())
}
if azEventHubFeatureGate.Stage() != featuregate.StageStable {
t.Errorf("expected stage %q, got %q", featuregate.StageBeta, azEventHubFeatureGate.Stage())
t.Errorf("expected stage %q, got %q", featuregate.StageStable, azEventHubFeatureGate.Stage())
}
expectedDesc := "When enabled, the Azure Event Hubs receiver will use the azeventhub library."
if azEventHubFeatureGate.Description() != expectedDesc {
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion receiver/azureeventhubreceiver/metadata.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ type: azure_event_hub
status:
class: receiver
stability:
alpha: [metrics, logs, traces]
beta: [metrics, logs, traces]
distributions: [contrib]
codeowners:
active: [atoulme, cparkins, dyl10s]
Expand Down