Skip to content

Relocate org.apache.datasketches in beam-sdks-java-harness - #166

Merged
yananhao12 merged 1 commit into
linkedin:li_trunkfrom
yehaolan:fix-datasketches-shading-2
May 5, 2026
Merged

yananhao12 merged 1 commit into
linkedin:li_trunkfrom
yehaolan:fix-datasketches-shading-2

Conversation

@yehaolan

@yehaolan yehaolan commented May 5, 2026 •

Copy link
Copy Markdown

Problem

beam-sdks-java-harness bundled ~70 unrelocated org.apache.datasketches.memory.* classes frozen at v2.0.0 with no shade relocation. These conflict with datasketches-memory:3.0.2 pulled in
transitively via datasketches-java:3.0.0 in consumer MPs, causing linkage checker "missing method" errors.

Root cause: BaseWritableMemoryImpl in v2.0.0 extends BaseStateImpl (no AutoCloseable), but in v3.0.2 it extends ResourceImpl implements Resource (adds close()). Both versions share the same
FQCNs, causing binary incompatibility.

Fix

Add a shade relocation rule in sdks/java/harness/build.gradle to relocate org.apache.datasketches to org.apache.beam.repackaged.harness.org.apache.datasketches, following the same pattern already
used in sdks/java/core for commons-compress, commons-lang3, and antlr.

Verification

Before fix — ~70 unrelocated classes visible:
jar tf beam-sdks-java-harness-2.45.54-SNAPSHOT.jar | grep '^org/apache/datasketches/memory/'
returns ~70 classes

After fix — no unrelocated classes, all relocated to shaded path:
jar tf beam-sdks-java-harness-2.45.55-SNAPSHOT.jar | grep '^org/apache/datasketches/memory/'
returns nothing

jar tf beam-sdks-java-harness-2.45.55-SNAPSHOT.jar | grep 'datasketches'
returns org/apache/beam/repackaged/harness/org/apache/datasketches/...

Changes

  • sdks/java/harness/build.gradle: add relocation rule for org.apache.datasketches
  • buildSrc/.../BeamModulePlugin.groovy: bump version to 2.45.55
  • gradle.properties: bump version to 2.45.55

@yananhao12
yananhao12 merged commit 42f92ea into linkedin:li_trunk May 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants