Remove grpc jars in directories other than connect plugin directories

[Animesh Kumar (akanimesh7)]

On running some of the connectors (spanner sink, pubsub sink and gcp bigtable sink) which package grpc dependencies, we are facing issues starting with CP 7.6.0 release. The detailed stacktrace is

[2024-03-05 12:09:28,047] ERROR Error with getting Connection To Google Cloud Bigtable:  (io.confluent.connect.gcp.bigtable.client.BigtableConnectionProvider:100)
java.lang.IllegalStateException: Could not find an appropriate constructor for com.google.cloud.bigtable.hbase2_x.BigtableConnection
	at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:200)
	at io.confluent.connect.gcp.bigtable.client.BigtableConnectionProvider.newConnection(BigtableConnectionProvider.java:98)
	at io.confluent.connect.gcp.bigtable.client.BigtableConnectionProvider.testConnection(BigtableConnectionProvider.java:134)
	at io.confluent.connect.gcp.bigtable.BigtableSinkConnectorConfig.validateBigtableConnection(BigtableSinkConnectorConfig.java:266)
	at io.confluent.connect.gcp.bigtable.BigtableSinkConnectorConfig.validateMultiConfigs(BigtableSinkConnectorConfig.java:96)
	at io.confluent.connect.utils.validators.all.ConfigValidation.lambda$callValidators$0(ConfigValidation.java:222)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at io.confluent.connect.utils.validators.all.ConfigValidation.callValidators(ConfigValidation.java:222)
	at io.confluent.connect.utils.validators.all.ConfigValidation.validate(ConfigValidation.java:182)
	at io.confluent.connect.gcp.bigtable.BigtableSinkConnector.validate(BigtableSinkConnector.java:51)
	at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:592)
	at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$6(AbstractHerder.java:470)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:197)
	... 17 more
Caused by: java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.NettyChannelProvider not a subtype
	at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:589)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1237)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
	at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)
	at io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:96)
	at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:41)
	at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
	at com.google.cloud.bigtable.grpc.BigtableSession.createNettyChannel(BigtableSession.java:458)
	at com.google.cloud.bigtable.grpc.BigtableSession$3.create(BigtableSession.java:413)
	at com.google.cloud.bigtable.grpc.io.ChannelPool.<init>(ChannelPool.java:248)
	at com.google.cloud.bigtable.grpc.BigtableSession.createRawDataChannelPool(BigtableSession.java:416)
	at com.google.cloud.bigtable.grpc.BigtableSession.<init>(BigtableSession.java:256)
	at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:123)
	at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:88)
	at com.google.cloud.bigtable.hbase2_x.BigtableConnection.<init>(BigtableConnection.java:56)
	... 22 more

The root cause for this is introduction of conflicting JARs in the kafka and confluent-metadata-service directories.

For kafka directory

[appuser@connect ~]$ ls -l /usr/share/java/kafka | grep grpc
-rw-r--r-- 1 root root   262791 Feb  7 13:36 grpc-api-1.54.0.jar
-rw-r--r-- 1 root root    30595 Feb  7 13:36 grpc-context-1.45.1.jar
-rw-r--r-- 1 root root    30586 Feb  7 13:36 grpc-context-1.54.0.jar
-rw-r--r-- 1 root root   737184 Feb  7 13:36 grpc-core-1.54.0.jar
-rw-r--r-- 1 root root   283595 Feb  7 13:36 grpc-netty-1.54.0.jar
-rw-r--r-- 1 root root     3586 Feb  7 13:36 grpc-netty-linux-0.8.4.jar
-rw-r--r-- 1 root root     3543 Feb  7 13:36 grpc-netty-macos-0.8.4.jar
-rw-r--r-- 1 root root  9393652 Feb  7 13:36 grpc-netty-shaded-1.54.0.jar
-rw-r--r-- 1 root root     5105 Feb  7 13:36 grpc-protobuf-1.54.0.jar
-rw-r--r-- 1 root root     7555 Feb  7 13:36 grpc-protobuf-lite-1.54.0.jar
-rw-r--r-- 1 root root    50909 Feb  7 13:36 grpc-stub-1.54.0.jar

For confluent-metadata-service directory

[appuser@connect ~]$ ls -l /usr/share/java/confluent-metadata-service | grep grpc
-rw-r--r-- 1 root root  262791 Feb  7 15:08 grpc-api-1.54.0.jar
-rw-r--r-- 1 root root   30595 Feb  7 15:08 grpc-context-1.45.1.jar
-rw-r--r-- 1 root root  737184 Feb  7 15:08 grpc-core-1.54.0.jar
-rw-r--r-- 1 root root  283595 Feb  7 15:08 grpc-netty-1.54.0.jar
-rw-r--r-- 1 root root    3586 Feb  7 15:08 grpc-netty-linux-0.8.4.jar
-rw-r--r-- 1 root root    3543 Feb  7 15:08 grpc-netty-macos-0.8.4.jar
-rw-r--r-- 1 root root 9393652 Feb  7 15:08 grpc-netty-shaded-1.54.0.jar
-rw-r--r-- 1 root root    5105 Feb  7 15:08 grpc-protobuf-1.54.0.jar
-rw-r--r-- 1 root root    7555 Feb  7 15:08 grpc-protobuf-lite-1.54.0.jar
-rw-r--r-- 1 root root   50909 Feb  7 15:08 grpc-stub-1.54.0.jar

These were not present in the previous CP versions. Except for grpc-context-1.45.1.jar present in the kafka directory. But removing it is also seemingly harmless.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Truc Nguyen (trnguyencflt)]

Animesh Kumar (@akanimesh7) why not taking a safer approach of investigating why grpc jars end up there, so that we can exclude those jars from the root.

This change is disruptive because it will remove every grpc jars.