Make and keep production and test code working with "up to date" Java versions

[snazy]

Describe the bug

Polaris, as a server, should be able run run recent Java versions for various reasons (GC improvements, heap usage improvements, OS integration improvements, etc etc etc).

"Client" libraries (think: Iceberg, Spark, Hadoop) however are known to cause problems with "newest" Java versions and/or recent library versions for various reasons.

Polaris relies on Hadoop and Spark for its tests and we are therefore limited to only run on Java versions that those libraries support / work with. It usually takes a lot of time until all these libraries are all compatible with "newer" Java versions - often lagging behind a few Java versions.

Goals

1. Tests should not rely on Hadoop or Spark, at best not even Iceberg, but only on Polaris code
2. If tests (think: integration tests) really need Hadoop and/or Spark, those should run separately using a "compatible" Java version to allow Polaris to leverage modern Java versions.

Polaris tests using Hadoop fail with Java 23

Background: The whole Java SecurityManager stuff was deprecated in Java 17 via JEP 411 (April 2021). Alternatives (Subject.getSubject() --> Subject.current()) have been introduced with Java 18.

JEP 486 is about to permanently disable the security manager, even the workaround to set java.security.manager=allow will not work beginning with Java 24 (the JVM won't start).

There's HADOOP-19212 (opened June 2024) with sadly rather stale PR.

Workaround for Java 23: #723