Skip to content

Metric event type handling support #282

Open
@cosmo0920

Description

@cosmo0920

I'm currently investigating metric event type with this connector with the following config and event format via custom forwarder.

connect-distributed.properties for connect

# A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
bootstrap.servers=localhost:9092

group.id=kafka-connect-splunk-hec-sink

or stored into Kafka
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=false
value.converter.schemas.enable=false

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter

internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

offset.storage.topic=connect-offsets
offset.storage.replication.factor=1

config.storage.topic=connect-configs
config.storage.replication.factor=1

status.storage.topic=connect-status
status.storage.replication.factor=1
#status.storage.partitions=5

# Flush much faster than normal, which is useful for testing/debugging
offset.flush.interval.ms=10000

plugin.path=connectors/

Record format

JSON string with the following format:

{"host":"development-box", "time":"1629859258.5508862","event":"metric", "fields":{"metric_name":"network_device_eth0_transmit_bytes_total", "_value":36589580.0}}

as mentioned in

Created Connector Task

{
  "name": "kafka-connect-splunk",
    "config": {
     "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector",
     "tasks.max": "10",
     "topics":"myapp.test,t1,t2,t3,t4,t5,t6,t7,t8,t9,t10",
     "splunk.hec.uri":"https://localhost:8088",
     "splunk.hec.token": "<Splunk HEC Token>",
     "splunk.hec.ack.enabled" : "true",
     "splunk.hec.raw" : "false",
     "splunk.hec.track.data" : "true",
     "splunk.hec.ssl.trust.store.path":"/etc/ssl/certs/java/cacerts",
     "splunk.hec.ssl.trust.store.password":"changeit",
     "splunk.hec.ssl.validate.certs": "false"
    }
}

And created metric index on Splunk with this instruction: https://docs.splunk.com/Documentation/Splunk/8.2.1/Metrics/GetMetricsInOther#Get_metrics_in_from_clients_over_HTTP_or_HTTPS


But no luck. What am I missing about ingestion of metric record via custom HEC forwarder via this connect? Or, this kafka-connect-splunk doesn't support metric type Splunk HEC events for now?

Additional context

With the above settings, event type indices not for metric type indices, and hec token for normal events, I got succeeded to ingest Splunk HEC events normally.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions