Skip to content

Getting timeout exception for s3 source connector, has to do with tasks.file.status.storage.bootstrap.servers.. #578

Open
@agavili

Description

@agavili

Setup Details
I'm running FilePulse 2.13.0 (https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.13.0/streamthoughts-kafka-connect-file-pulse-2.13.0.zip) on Mac

Concern
I'm trying to create a s3 source connector using filepulse and this connector is linked to a topic in IBM Cloud's Event Streams. I'm not entirely sure what to set tasks.file.status.storage.bootstrap.servers to. When I set it to localhost:9092, I get a timeout exception. The same occrs when I set the field to the cluster's internal listener: development-kafka-bootstrap.cp4i.svc:9093. If I comment out the field, I get a Failed to create shared StateBackingStore for group 'connect-file-pulse-amazon-s3-csv' error. What should I set it to? Is there any other configuration I'm missing?

Here's my connector's yaml:

apiVersion: eventstreams.ibm.com/v1beta2
kind: KafkaConnector
metadata:
  name: connect-file-pulse-amazon-s3-csv
  labels:
    eventstreams.ibm.com/cluster: my-connect-cluster
spec:
  class: io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector
  tasksMax: 1
  config:
    topic: XXXX
    tasks.max: 1
    fs.listing.class: io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing
    fs.listing.interval.ms: 10000
    # fs.listing.filters: io.streamthoughts.kafka.connect.filepulse.scanner.local.filter.IgnoreHiddenFileListFilter
    file.filter.regex.pattern: .*\\.csv$

    fs.cleanup.policy.class: io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy 

    aws.access.key.id: XXXX
    aws.secret.access.key: XXXX
    aws.s3.region: us-east-2
    aws.s3.bucket.name: XXXX

    tasks.reader.class: io.streamthoughts.kafka.connect.filepulse.fs.reader.AmazonS3RowFileInputReader

    skip.headers: 1
    offset.attributes.string: uri

    filters: ParseLine
    filters.ParseLine.type: io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter
    filters.ParseLine.extractColumnName: headers
    filters.ParseLine.trimColumn: true
    filters.ParseLine.separator: ;
    tasks.file.status.storage.bootstrap.servers: localhost:9092
    tasks.file.status.storage.topic: connect-file-pulse-status
    tasks.file.status.storage.topic.partitions: 10
    # tasks.file.status.storage.topic.replication.factor: 1

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requestedwontfixThis will not be worked on

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions