Glue serde avro to json deserialization includes namespaces and union types

[Ronserruya]

Originally reported in #3224 , split into a separate issue following the discussion in #3235

When the glue serde deserializes to json from avro, it includes the record namespaces and types in the case of union. This is the first time I'm encountering the behaviour since the python deserializer or the one used in kafka-connect don't follow this behavior

Example:

Original msg:

{"name": {"first": "ron", "last": "serruya", "full": "ron serruya"}, "ids1": [5,6], "ids2": ["abc", 123]}

schema used:

{
  "type": "record",
  "name": "generation",
  "namespace": "top_level",
  "fields": [
    {
      "name": "name",
      "type": [
        {
          "type": "record",
          "name": "name",
          "namespace": "top_level.generation",
          "fields": [
            {
              "name": "raw",
              "type": [
                "string",
                "null"
              ]
            },
            {
              "name": "first",
              "type": "string"
            },
            {
              "name": "full",
              "type": "string"
            },
            {
              "name": "last",
              "type": ["string"]
            }
          ]
        },
        "null"
      ]
    },
    {
      "name": "ids1",
      "type": {"type": "array", "items": "int"}
    },
    {
      "name": "ids2",
      "type": {"type": "array", "items": ["string", "int"]}
    }
  ]
}

base64 encoded avro msg (just the msg, without the glue-related bytes at the start)
AAIGcm9uFnJvbiBzZXJydXlhAA5zZXJydXlhBAoMAAQABmFiYwL2AQA=

The current glue deserializer shows this msg as:

{
  "name": {
    "top_level.generation.name": {
      "raw": null,
      "first": "ron",
      "full": "ron serruya",
      "last": {
        "string": "serruya"
      }
    }
  },
  "ids1": [
    5,
    6
  ],
  "ids2": [
    {
      "string": "abc"
    },
    {
      "int": 123
    }
  ]
}

As you can see it adds string, int, or the record namespace top_level.generation.name

I fixed this issue locally by adding this line: encoder.setIncludeNamespace(false); in the avroRecordToJson method

But according to the comment in #3235 , that's not a completely valid fix since it can break other stuff?

Before and after the fix:

[Haarolean]

Hey, thanks, we'll take a look.

[S1M0NM]

This seems to also apply to the default SchemaRegistry Serde.

Is there a way i can fix this for the included SchemaRegistry serde?

[frankgrimes97]

@Haarolean Any update on this? We're also being affected by this odd display behavior and would like to see it fixed.

[Haarolean]

@frankgrimes97 planned for 0.8

[iliax]

Reopening, since it is only fixed for Kafka schema registry, not glue

[frankgrimes97]

@Haarolean Any update on when we might see a fix and 0.8 release?

[Haarolean]

@Haarolean Any update on when we might see a fix and 0.8 release?

@frankgrimes97
#4255
kafbat/kafka-ui#23