Examples of how to convert yaml complex types to CLI equivalent

[nullhack]

Hi, thank you for the awesome tool, I'm really impressed by how much power I have in such small amount of config. But one particular thing that I am struggling is the command line version of the config file.

I tried some combinations, but I could not get it to work. I have a config.yaml (that works) similar to this:

input:
  kafka:
    addresses: ["ADDRESS"]
    topics: ["TOPIC"]
    consumer_group: "CG"
output:
  aws_s3:
    bucket: BUCKET
    path: OUT_PATH
    force_path_style_urls: true
    region: us-east-1
    endpoint: S3_ENDPOINT
    credentials:
      id: CREDENTIAL_ID
      secret: CREDENTIAL_SECRET
    batching:
      count: 1000
      processors:
        - parquet_encode:
            schema:
              - name: NAME
                type: UTF8
              - name: ADDRESS
                type: UTF8

Due to not being able to perform writes in this particular server, I am trying to run the CLI equivalent of the config.

Something like:

connect run \
  -s "input.type=http_server" \
  -s "input.type=kafka" \
  -s "input.kafka.addresses=ADDRESS" \
  -s "input.kafka.topics=TOPIC" \
  -s "input.kafka.consumer_group=CG" \
  -s "output.aws_s3.bucket=BUCKET" \
  -s 'output.aws_s3.path=OUT_PATH' \
  -s "output.aws_s3.force_path_style_urls=true" \
  -s "output.aws_s3.region=us-east-1" \
  -s "output.aws_s3.endpoint=S3_ENDPOINT" \
  -s "output.aws_s3.credentials.id=CREDENTIAL_ID" \
  -s "output.aws_s3.credentials.secret=CREDENTIAL_SECRET" \
  -s "output.aws_s3.batching.count=1000" \
  -s "output.aws_s3.batching.processors ...

My question is: How to set up the processors properly to get the parquet encode to work (or any other complex mapping involving yaml lists and dictionaries) ?

Is it possible to add examples somewhere in the documentation as well?

Thank you!

[mihaitodor]

Hey @nullhack 👋 Unfortunately, it's a bit awkward because the underlying CLI library doesn't allow us to set DisableSliceFlagSeparator on individual flags. Because of this, setting an object or an array which contain commas doesn't work. Not sure if that was intentional or just an oversight in urfave/cli#1671. However, it is possible to initialise and append to arrays and to initialise objects and update / add extra keys to them if done step by step. Here's an example:

test.yaml:

input:
  generate:
    count: 1
    mapping: root = ""

CLI command:

>  redpanda-connect run -s 'input.generate.mapping="root = \"blobfish are cool\""' -s 'input.processors=[]' -s 'input.processors.-={"log":{"message":"foobar"}}' -s 'input.processors.-={"log":{"message":"test"}}' test.yaml

This changes the generate mapping and injects 2 processors at the output level. Hope this is enough to get you sorted.

PS: Converting this to discussion as per #2026.

[nullhack]

Thank you @mihaitodor