Clear all 48 oss-canary CVEs blocking ELR and bump to 2.45.61

[cirquare]

Summary

ELR 367778 for com.linkedin.beam:beam-runners-flink-1.18:2.45.60 failed oss-canary validation with 48 flagged Maven coordinates, blocking publication. This clears all 48 and shrinks the runtime dependency graph from 212 to 70 nodes.

Why the previous tickets did not unblock it

ELR runs two independent gates. INFOSEC-143688 and TOOLS-496440 both address the 7-day cooldown gate — INFOSEC-143688 explicitly says "from the age requirement only." ELR 367778 failed the vulnerability gate, which is separate.

Also worth noting: oss-canary walks the unresolved POM graph (the report lists 10 guava versions, 9 protobuf-java, 6 jackson-core). Gradle force/resolutionStrategy cannot help, because losing versions remain as nodes. Only edge removal, exclusion, or raising a version we declare ourselves works.

The 2.45.58 → 2.45.60 diff touched only DoFnOperator and the version bump — no dependency changes. The artifact did not regress; the CVE feed got stricter.

Changes

1. Drop google-cloud-platform-core from the Flink runner — clears 31 coordinates

It was a compile-scope dependency reaching the published POM and pulling the entire Google Cloud stack, but its only use in main source was one line in FlinkJobServerDriver setting a GCS upload buffer for the portability job server. This removes guava (10 versions), protobuf-java (9), grpc-netty-shaded (4), gson (3), google-oauth-client (2), protobuf-javalite (2), bouncycastle (2), httpclient, protobuf-parent, old jackson-core 2.9.6–2.13.4, and commons-lang3:3.5.

Safe downstream: beam-runner already excludes the GCP artifacts (exclude group: 'com.google.cloud' in build.gradle, plus a gcp-core exclusion in runtime-spark), and runtime-flink never references gcp-core. The testImplementation on sdks:java:io:google-cloud-platform is untouched.

2. Version bumps in BeamModulePlugin.groovy

Library	Before	After	CVE
jackson	2.14.1	2.19.2	unbounded exception message via createParser(DataInput)
classgraph	4.8.104	4.8.179	XXE
commons-compress	1.21	1.27.1	resource allocation + infinite loop
commons-lang3	3.9	3.18.0	ClassUtils.getClass uncontrolled recursion
snappy-java	1.1.8.4	1.1.10.7	SnappyInputStream DoS
avro	1.8.2	1.11.4	schema-parsing RCE + OOM
commons-io	2.7	2.16.1	required floor for commons-compress 1.26+
commons-codec	1.15	1.17.1	same

snakeyaml 1.33 → 2.4 comes free via jackson-dataformat-yaml, fixing CVE-2022-1471.

Note: commons-compress 1.26+ calls BoundedInputStream.builder(), which requires commons-io ≥ 2.16. Bumping it alone fails 32 tests with NoSuchMethodError.

3. Exclude commons-compress from the Flink modules — clears 1

flink-core:1.18.0 declares 1.21 in its own POM, so bumping ours does not remove it from the scan. Verified flink-runtime references it in 0 classes and flink-core in exactly 3 (Bzip2/XZ/ZStandard InputStreamFactory — the compressed FileInputFormat path Beam pipelines never use). Beam still supplies the patched 1.27.1 via sdks:java:core → avro.

Per-dependency exclusions are used deliberately; a blanket configurations.all { exclude ... } was tried and rejected because it stripped commons-compress entirely, including the copy Avro needs for its bzip2/xz codecs.

4. Avro 1.8.2 → 1.11.4 — clears the last 2 (avro:1.8.2, jackson-mapper-asl:1.9.13)

• Codegen plugin → com.github.davidmc24.gradle.plugin:gradle-avro-plugin:1.9.1 (bundles avro-compiler:1.11.3). The old 0.16.0 pinned avro-compiler:1.8.2, which emits TimeConversions.TimeConversion/TimestampConversion — deleted in Avro 1.9 when Joda was dropped for java.time. The now-unused JitPack repository entry is removed.
• org.tukaani:xz declared explicitly — Avro 1.11 marks it optional, so it silently left the classpath and broke the xz codec.
• org.apache.avro.RandomData → org.apache.avro.util.RandomData.
• java.time branches added to the AvroUtils ByteBuddy converters (AvroConvertType, ...ForGetter, ...ForSetter) and to the runtime LogicalTypes.TimestampMillis path. Purely additive — these types previously threw ClassCastException.
• Kryo fix ported from upstream Beam 2.75: addDefaultSerializer(SerializableSchemaSupplier.class, JavaSerializer.class) plus widening that class from private to package-private, exactly as upstream. Root cause is Avro 1.9+ giving JsonProperties an instance-level immutable reserved Set that Kryo cannot rebuild.

5. Version bump to 2.45.61

BeamModulePlugin.groovy is the publish trigger path and holds project.version (line 401), so changing it without a bump would republish 2.45.60. Applied via ./update_version.sh 2.45.61.

On the Avro ReflectData field-order change — no production impact

Avro's ReflectData switched from declaration order to alphabetical. Measured directly across versions:

declaration order: boolNonNullable,anInt,aLong,string,bytes

avro 1.8.2    -> boolNonNullable,anInt,aLong,string,bytes   (declaration order)
avro 1.9.2    -> boolNonNullable,anInt,aLong,string,bytes   (declaration order)
avro 1.10.2   -> aLong,anInt,boolNonNullable,bytes,string   <-- changed here
avro 1.11.3   -> aLong,anInt,boolNonNullable,bytes,string   <-- what beam-runner pins today
avro 1.11.4   -> aLong,anInt,boolNonNullable,bytes,string   <-- this PR

The change landed in Avro 1.10, and 1.11.3 is behaviorally identical to 1.11.4. ReflectData resolves at runtime from the classpath, and beam-runner's product-spec.json already pins 1.11.3 — so LinkedIn jobs have been getting alphabetical ordering all along. Beam declaring 1.8.2 only affected what Beam compiled and tested against.

The updated AvroSchemaTest expectations are therefore Beam's tests catching up to what already runs, not a behavior change. This PR removes a version skew rather than introducing one. No repo in the SP workspace pins Avro below 1.10 (beam-runner 1.11.3, ctc 1.11.4).

Testing Done

Local, JDK 11 (CI uses JDK 8 — see caveat below).

Suite	Result
:runners:flink:build + :runners:flink:1.18:build	BUILD SUCCESSFUL
:runners:flink:1.18:test	360 tests, 0 failures
:runners:flink:1.20:test	357 tests, 0 failures
:runners:spark:3:build	BUILD SUCCESSFUL
:runners:samza:build	114 run, 21 failures — all pre-existing
:sdks:java:core:test	3452 run, 1 failure — pre-existing
:runners:core-java:test	342 run, 7 failures — all pre-existing
spotlessApply + checkstyle	clean on all changed files

Every failure was baselined against clean li_trunk in a separate worktree before being attributed. For Samza the failing-test name sets were compared and are identical (21 vs 21, zero new). Samza failures are an arm64 macOS issue (NoClassDefFoundError: org.rocksdb.Options — no Apple Silicon native in the bundled RocksDB JNI) and should pass on CI's ubuntu-latest.

Pre-existing failures carried over unchanged: ProxyInvocationHandlerTest.testConcurrency, DistributionCellTest.{testEquals,testReset}, MetricsContainerImplTest.* (3), MetricsContainerStepMapTest.*, SplittableParDoProcessFnTest.testCheckpointsAfterDuration.

CVE verification

Re-resolved :runners:flink:1.18:runtimeClasspath and checked each of the 48 reported coordinates:

CLEARED: 48/48   REMAINING: 0
graph nodes: 212 -> 70

Published POM verified: gcp-core absent, jackson-databind at 2.19.2, <exclusion> entries present for commons-compress.

Caveats for reviewers

• JDK mismatch — CI uses JDK 8 (javaVersion=1.8); all local validation ran on JDK 11 (no JDK 8 available locally).
• validatesRunner not run locally for Flink, Spark, or Samza. CI runs all three.
• This PR fans out to four gates. Touching BeamModulePlugin.groovy matches the trigger path for the Flink, Spark, and both Samza workflows, so all four will run even though only Flink files changed.
• Scope is the Flink runners. auto-elr publishes 22 modules, each with its own ELR. The version bumps and Avro migration help all of them, but the gcp-core removal only helps Flink — sdks:java:extensions:google-cloud-platform-core and sdks:java:io:google-cloud-platform are GCP by nature and will still carry that stack.
• sdks:java:core has pre-existing spotless and checkstyle violations in unrelated files (DLQ / WindowedValue*), left untouched here.