Merge upstream changes#20
Merged
Merged
Conversation
* REST: Promote the S3 signing endpoint to the main spec Dev ML discussion: https://lists.apache.org/thread/2kqdqb46j7jww36wwg4txv6pl2hqq9w7 This commit promotes the S3 remote signing endpoint from an AWS-specific implementation to a first-class REST catalog API endpoint. This enables other storage providers (GCS, Azure, etc.) to eventually reuse the same signing endpoint pattern without duplicating the API definition. Summary of changes: - Added `/v1/{prefix}/namespaces/{namespace}/tables/{table}/sign/{provider}` endpoint to the main REST catalog OpenAPI spec. - Defined `RemoteSignRequest`, `RemoteSignResult` and `RemoteSignResponse` schemas. - Defined a new `provider` request body parameter in order to disambiguate requests from different storage providers. - Deprecated the separate `s3-signer-open-api.yaml` spec from the AWS module (for removal). - Updated the Python client.
…he#15049) Introduces foundational types for V4 manifest support These types follow the https://s.apache.org/iceberg-single-file-commit and will be used by subsequent PRs for manifest reading/writing. For now, we are adding these as package-private interfaces in core, and eventually we will move them into api.
…5734) Signal to the TableOperations that there is no retry needed for files which do not exist.
…5825) * Spark: Control merge schema evolution by table property Add a new table property write.spark.auto-schema-evolution (default true) that controls whether the AUTOMATIC_SCHEMA_EVOLUTION capability is reported to Spark. When set to false, Spark's MERGE WITH SCHEMA EVOLUTION no longer evolves the target table schema. Also add a guard in SparkWriteBuilder to reject mergeSchema write option when the property is disabled. * Remove unnecessary validation from SparkWriteBuilder The capability removal in SparkTable is sufficient to control schema evolution. The mergeSchema write option path already requires accept-any-schema, making a second gate redundant. * Address review comments - Rename property to write.spark.auto-schema-evolution.enabled - Rename caps to tableCapabilities in computeCapabilities - Add explicit = in ALTER TABLE SET TBLPROPERTIES test SQL
This fixes Russell's feedback on apache#15049 to avoid version-specific language that will go stale.
Adds a build-time check that prevents accidental transitive dependency leaks into shipped shadow JARs and distribution archives. A checked-in runtime-deps.txt baseline lists every dependency resolved into each bundled artifact. checkRuntimeDeps compares resolved deps against the baseline and fails the build with a clear diff on mismatch, wired into the check lifecycle so it runs in CI automatically. This guards all 11 bundled modules: Spark runtime (3.4, 3.5, 4.0, 4.1), Flink runtime (1.20, 2.0, 2.1), cloud bundles (AWS, Azure, GCP), and Kafka Connect runtime.
Co-authored-by: manuzhang <owenzhang1990@gmail.com>
Bumps [mkdocs-redirects](https://github.com/ProperDocs/properdocs-redirects) from 1.2.2 to 1.2.3. - [Release notes](https://github.com/ProperDocs/properdocs-redirects/releases) - [Commits](ProperDocs/mkdocs-redirects@v1.2.2...v1.2.3) --- updated-dependencies: - dependency-name: mkdocs-redirects dependency-version: 1.2.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 7.6.0 to 8.0.0. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@37802ad...cec2083) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…15886) Bumps [org.openapitools:openapi-generator-gradle-plugin](https://github.com/OpenAPITools/openapi-generator) from 7.20.0 to 7.21.0. - [Release notes](https://github.com/OpenAPITools/openapi-generator/releases) - [Changelog](https://github.com/OpenAPITools/openapi-generator/blob/master/docs/release-summary.md) - [Commits](OpenAPITools/openapi-generator@v7.20.0...v7.21.0) --- updated-dependencies: - dependency-name: org.openapitools:openapi-generator-gradle-plugin dependency-version: 7.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#15889) Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.78.0 to 26.79.0. - [Release notes](https://github.com/googleapis/java-cloud-bom/releases) - [Commits](googleapis/java-cloud-bom@v26.78.0...v26.79.0) --- updated-dependencies: - dependency-name: com.google.cloud:libraries-bom dependency-version: 26.79.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…e#15890) Bumps software.amazon.awssdk:bom from 2.42.18 to 2.42.23. --- updated-dependencies: - dependency-name: software.amazon.awssdk:bom dependency-version: 2.42.23 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `jetty` from 12.1.5 to 12.1.7. Updates `org.eclipse.jetty:jetty-server` from 12.1.5 to 12.1.7 Updates `org.eclipse.jetty.ee10:jetty-ee10-servlet` from 12.1.5 to 12.1.7 --- updated-dependencies: - dependency-name: org.eclipse.jetty:jetty-server dependency-version: 12.1.7 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.eclipse.jetty.ee10:jetty-ee10-servlet dependency-version: 12.1.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#15891) Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.2.10.Final to 4.2.12.Final. - [Release notes](https://github.com/netty/netty/releases) - [Commits](netty/netty@netty-4.2.10.Final...netty-4.2.12.Final) --- updated-dependencies: - dependency-name: io.netty:netty-buffer dependency-version: 4.2.12.Final dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* AWS: Add chunked encoding configuration for S3 requests * add testMultipartUploadWithChunkedEncodingDisabled * update open api define * update * update default value * update case * assert file contents in testMultipartUploadWithChunkedEncoding * Remove s3.chunked-encoding-enabled config entry from REST catalog open API spec * Use IOUtil.readFully for reliable reads in TestS3MultipartUpload * ensure testIo is properly closed * retrigger CI * Change chunked encoding default to true to match AWS SDK behavior * Fix test to verify explicit disable of chunked encoding instead of duplicating default
* Make MAX_WAIT_TIME_MS configurable for RESTTableScan * fix style * fix checkstyle: add hasMessage check to assertThatThrownBy Co-authored-by: Isaac * Address Amogh's comments * address comments
…ion with Hive catalog (apache#15814) * Docs: Add Hive Metastore schema validation warnings for DROP COLUMN and REORDER When using a Hive catalog, ALTER TABLE DROP COLUMN (non-last column) and ALTER COLUMN REORDER fail because the Hive Metastore validates schema changes by comparing column types positionally. Dropping a middle column shifts subsequent columns, causing HMS to reject the change as an incompatible type change via MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange. Add warning admonitions to spark-ddl.md (DROP COLUMN and REORDER sections) and flink-ddl.md (Hive catalog section) documenting the limitation, workaround (hive.metastore.disallow.incompatible.col.type.changes=false), and trade-off (Hive engine can no longer read the table). * Docs: Clarify HMS workaround for embedded vs remote deployment * Docs: add more warning for spark-ddl.md * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Manu Zhang <OwenZhang1990@gmail.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Manu Zhang <OwenZhang1990@gmail.com>
Fix zizmor ref-version-mismatch audit failure caused by the rolling v7 tag moving to v7.0.1 while workflows pinned the v7.0.0 hash. Regenerate Spark 4.1 runtime-deps.txt to reflect dependency changes from recent dependabot bumps. Made-with: Cursor Co-authored-by: Neelesh Salian <n_salian@apple.com>
…ache#15941) This reverts commit 9a939d6.
Bumps [joda-time:joda-time](https://github.com/JodaOrg/joda-time) from 2.5 to 2.14.2. - [Release notes](https://github.com/JodaOrg/joda-time/releases) - [Changelog](https://github.com/JodaOrg/joda-time/blob/main/RELEASE-NOTES.txt) - [Commits](JodaOrg/joda-time@v2.5...v2.14.2) --- updated-dependencies: - dependency-name: joda-time:joda-time dependency-version: 2.14.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `junit-platform` from 1.14.3 to 1.14.4. Updates `org.junit.platform:junit-platform-launcher` from 1.14.3 to 1.14.4 - [Release notes](https://github.com/junit-team/junit-framework/releases) - [Commits](https://github.com/junit-team/junit-framework/commits) Updates `org.junit.platform:junit-platform-suite-api` from 1.14.3 to 1.14.4 - [Release notes](https://github.com/junit-team/junit-framework/releases) - [Commits](https://github.com/junit-team/junit-framework/commits) Updates `org.junit.platform:junit-platform-suite-engine` from 1.14.3 to 1.14.4 - [Release notes](https://github.com/junit-team/junit-framework/releases) - [Commits](https://github.com/junit-team/junit-framework/commits) --- updated-dependencies: - dependency-name: org.junit.platform:junit-platform-launcher dependency-version: 1.14.4 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.junit.platform:junit-platform-suite-api dependency-version: 1.14.4 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.junit.platform:junit-platform-suite-engine dependency-version: 1.14.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.35.2 to 4.35.3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@95e58e9...e46ed2c) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.35.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `junit` from 5.14.3 to 5.14.4. Updates `org.junit.jupiter:junit-jupiter` from 5.14.3 to 5.14.4 - [Release notes](https://github.com/junit-team/junit-framework/releases) - [Commits](junit-team/junit-framework@r5.14.3...r5.14.4) Updates `org.junit.jupiter:junit-jupiter-engine` from 5.14.3 to 5.14.4 - [Release notes](https://github.com/junit-team/junit-framework/releases) - [Commits](junit-team/junit-framework@r5.14.3...r5.14.4) --- updated-dependencies: - dependency-name: org.junit.jupiter:junit-jupiter dependency-version: 5.14.4 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.junit.jupiter:junit-jupiter-engine dependency-version: 5.14.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…6277) Co-authored-by: Cursor <cursoragent@cursor.com>
…1.0 to 7.22.0 (apache#16278) Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.