Feature request: decode binary formats

[moorchegue]

Is there any way to add custom decoders for messages in e.g. Protobuf, Avro or MessagePack? Perhaps by calling a configurable external process that would get STDIN and spit JSON back out?

[EdJoPaTo]

Thank you for bringing up the idea!

MessagePack is supported already as it doesn’t require some schema. ProtoBuf requires a schema or doesn’t it? Haven’t heard about Avro before.

As soon as something requires a schema it gets a bit more complex. Not sure how that would work well.

Currently I would think specific implementations (like ProtoBuf) might be more helpful so a specific schema could be supplied. That would be easier to use than providing a full bytes to content logic.

#159 might also be helpful in this case as external tools could then be used on the actual binary data? 🤔

[moorchegue]

Yeah, Protobuf requires a schema. Furthermore, as far as I understand, it requires compilation into a language first, so raw schemas can't be used even with dynamically typed languages.

Avro has both schema and schemaless modes. In the first case schema is basically included into a message. So that could be decoded without any external dependency. Schemas here are JSON-based, so this could be used in a dynamically typed language directly without the compilation step.

For schema-dependent cases I think an external filter process call or some other form of communication would be necessary. This could also be useful for things like doing some custom jq queries for JSON in order to only display relevant data. But I'm not sure what would be the best way to generalize this.

[lukas-mertens]

https://protobuf-decoder.netlify.app/ exists which is actually quite useful already for protobuf. You can then just look into your schema yourself and map fields

[moorchegue]

Still very much interested in this! :D