Endpoint URL config is not used in KinesisProducer when writing stream

[CICDamen]

Description
After setting the .option("kinesis.endpointUrl") in writeStream to a local testing endpoint (Localstack), the connector still uses the default Kinesis URL. This means that this configuration is not properly propagated to the KinesisProducerConfiguration class and not used in the KinesisProducer handling the writes.

Setting in writer

writeDf.writeStream
      .format("aws-kinesis")
      .outputMode("append")
      .option("kinesis.endpointUrl", "http://127.0.0.1:56803")

KinesisProducer error

5/03/24 12:54:49 ERROR KinesisProducer: Error in child process java.lang.RuntimeException: EOF reached during read
	at org.apache.spark.sql.connector.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:532)
	at org.apache.spark.sql.connector.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:508)
	at org.apache.spark.sql.connector.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:504)
	at org.apache.spark.sql.connector.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.readSome(Daemon.java:553)
	at org.apache.spark.sql.connector.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.receiveMessage(Daemon.java:243)
	at org.apache.spark.sql.connector.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon.access$500(Daemon.java:61)
	at org.apache.spark.sql.connector.kinesis.shaded.com.amazonaws.services.kinesis.producer.Daemon$3.run(Daemon.java:298)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Producer using default endpoints instead of custom endpointUrl

25/03/24 12:54:49 INFO LogInputStreamReader: [2025-03-24 12:54:49.703784] [info] [kinesis_producer.cc:62] Using default Kinesis endpoint
25/03/24 12:54:49 INFO LogInputStreamReader: [2025-03-24 12:54:49.705050] [info] [kinesis_producer.cc:62] Using default CloudWatch endpoint
25/03/24 12:54:49 INFO LogInputStreamReader: [2025-03-24 12:54:49.705856] [info] [kinesis_producer.cc:62] Using default STS endpoint

Error when trying to reach

25/03/24 12:54:49 WARN LogInputStreamReader: [2025-03-24 12:54:49.701438] [warning] [AWS Log: ERROR] Curl returned error code 7 - Couldn't connect to server
25/03/24 12:54:49 WARN LogInputStreamReader: [2025-03-24 12:54:49.701500] [warning] [AWS Log: ERROR] Http request to retrieve credentials failed
25/03/24 12:54:49 WARN LogInputStreamReader: [2025-03-24 12:54:49.701968] [warning] [AWS Log: ERROR] Can not retrieve resource from http://169.254.169.254/latest/api/token
25/03/24 12:54:49 WARN LogInputStreamReader: [2025-03-24 12:54:49.702818] [warning] [AWS Log: ERROR] Can not retrieve resource from http://169.254.169.254/latest/meta-data/iam/security-credentials
25/03/24 12:54:49 WARN LogInputStreamReader: [2025-03-24 12:54:49.703751] [warning] [AWS Log: ERROR] Can not retrieve resource from http://169.254.169.254/latest/meta-data/placement/availability-zone