Cross region buckets using the aws_s3 input

[ryanpodonnell1]

I have an input like this, which works fine. The problem I am having is that it appears not to be able to pull from buckets in different regions. Is there some dynamic way I can pull the region of the bucket in?

input:
  label: "${Workspace:dev}_conscan"
  aws_s3:
    bucket: "${BENTHOS_ConscanS3Bucket}"
    scanner:
      lines: {}
    codec: gzip/all-bytes
    sqs:
      url: "${BENTHOS_ConscanSqsQueue}"
      key_path: detail.object.key
      bucket_path: detail.bucket.name
      max_messages: 10
      wait_time_seconds: 0

so it can end up like this?

input:
  label: "${Workspace:dev}_conscan"
  aws_s3:
    bucket: "${BENTHOS_ConscanS3Bucket}"
    region: detail.bucket.region <= here so I can force the regional endpoint?
    scanner:
      lines: {}
    codec: gzip/all-bytes
    sqs:
      url: "${BENTHOS_ConscanSqsQueue}"
      key_path: detail.object.key
      bucket_path: detail.bucket.name
      max_messages: 10
      wait_time_seconds: 0

[mihaitodor]

Hey @ryanpodonnell1, I'm not very familiar with regional endpoints for S3 buckets, but this might be related to #2343 which requires some code enhancements unfortunately.

[ryanpodonnell1]

Yes that looks similar. I'll resolve the convo