[jpalmerr]

KAFKA-21309

A consumer subscribed with SubscriptionPattern requests metadata by
topic ID. When an assigned topic is deleted, or the client may not
describe it, the broker answers its ID with an error and no topic name,
as the schema allows. The client did not handle the null topic name row.

This fix

1. MetadataResponse skips error rows without a name, as opposed to
   throwing IllegalStateException.
2. Now that a null name can reach the response handler, it skips the
   row. On a retriable error (e.g. UNKNOWN_TOPIC_ID) it also requests a
   metadata update, else (e.g. TOPIC_AUTHORIZATION_FAILED) it leaves the
   Fetch response to raise TopicAuthorizationException

Once metadata updates resume, reconciliation revokes the deleted topic's
partitions and resolves newly assigned topics.

[jpalmerr]

@dajac @lianetm hi! heres the reproduction as a failing unit test. I've got a script to observe the full behaviour against a real broker, including the consumer never picking up a new topic afterwards, comparing 4.0.0 with 4.1.0+.

gist with script

[jpalmerr]

This PR now has the fix.

With the reproducer from the gist, a consumer on this branch kept updating metadata after the assigned topic was deleted, consumed the newly created matching topic, and sent no excess heartbeats.