multilevel failed to obtain docs

[nullhack]

Hi, thank you for the project, I have a question that might be some simple thing, but I have spend some hours trying to figure out what is the issue unsuccessfuly. I am trying to test multilevel cache, but I keep receiving a message failed to obtain docs for cache type multilevel

I'm using the docker version from docker.redpanda.com/redpandadata/connect:latest (Version: v4.33.0).

My config.yaml looks like this:

input:
  kafka:
    addresses: ["ADDR"]
    topics: ["TOPIC"]
    consumer_group: "CG"

pipeline:
  processors:
    - cache:
        resource: leveled
        operator: add
        key: '${! json("message.id") }'
        value: "storeme"
    - mapping: root = if errored() { deleted() }

output:
  kafka:
    addresses: ["ADDR"]
    topic: TOPIC

cache_resources:
  - label: leveled
    multilevel: [ hot, cold ]
  - label: hot
    memory:
      default_ttl: 60s
  - label: cold
    aws_s3:
      (...)

Am I doing anything wrong?

I am trying to find a workaround for the limitation of add in S3, would this work? (i.e. do not create new versions of the cache in S3 if it already exists)

Thank you!

[mihaitodor]

Hey @nullhack 👋 I'm really struggling to reproduce the error you're seeing. I tried with this Docker container:

> docker run --rm -it -v $(pwd):/workspace docker.redpanda.com/redpandadata/connect:4.33.0 run /workspace/test.yaml

and I have this test.yaml:

input:
  stdin: {}

pipeline:
  processors:
    - cache:
        resource: leveled
        operator: add
        key: '${! json("message.id") }'
        value: "storeme"
    - mapping: root = if errored() { deleted() }

output:
  stdout: {}

cache_resources:
  - label: leveled
    multilevel: [ hot, cold ]
  - label: hot
    memory:
      default_ttl: 60s
  - label: cold
    aws_s3:
      bucket: test

and it seems to at least initialise fine. I can also replace that aws_s3 with another memory cache and it doesn't raise any error...

I can see which code generates that error, but I'm really not sure how to trigger it.

I am trying to find a workaround for the limitation of add in S3, would this work? (i.e. do not create new versions of the cache in S3 if it already exists)

Not sure I follow... It's implemented here.

[nullhack]

Hi @mihaitodor thank you for your reply. I actually used the latest tag, I'm not sure if there's any difference, but here is a screenshot:

I tried with the tag 4.33.0 and it works just fine, thank you!

About the S3, I will create a new discussion topic on that.

Thank you!

[mihaitodor]

Thanks for that! I was able to reproduce it with the latest tag. Please use 4.33.0 while we get this sorted.