Write tests to verify consumer can handle race conditions between metadata updates/ group rebalances #1251
Description
The root cause of #1237 was a race condition between a metadata update response and a consumer group rebalancing finishing.
The issue only appears when the responses come back in a particular sequence (explained in KAFKA-3949 (#1241)).
It'd be good to have an end-to-end test on this behavior to verify it doesn't happen again.
@tvoinarovskyi / @dpkp let me know if you have any suggestions on how to write this test somewhat cleanly. My day job wants this test in place, so I can spend time on it, but when I looked at it, I'm not coming up with a straightforward way to force the proper response sequence.
I'm thinking it'll be more straightforward after #1230 is merged so that I only need to mock the broker response at the protocol layer and not the network layer... your thoughts?