Move non-dill specific code out of dill_pickler.py #35139
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AdrS
reviewed
Jun 4, 2025
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Run Python_Coverage PreCommit |
|
Run Python_Runners PreCommit 3.9 |
|
Run Python_Coverage PreCommit |
|
Run Python_Coverage PreCommit |
|
assign set of reviewers |
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
shunping
requested changes
Jun 6, 2025
kristynsmith
commented
Jun 6, 2025
|
Run Python_Dataframes PreCommit 3.11 |
|
Run Python_Coverage PreCommit |
|
LGTM. Thanks! |
shunping
approved these changes
Jun 6, 2025
razvanculea
pushed a commit
to razvanculea/beam
that referenced
this pull request
Jun 17, 2025
* Create consistent_pickle.py * Update dill_pickler.py * Update consistent_pickle.py * Add Apache license to consistent_pickle.py * Update dill_pickler.py * Update and rename consistent_pickle.py to code_object_pickler.py * Update dill_pickler.py to change consistent_pickle to code_object_pickler * Fix formatting issue in code_object_pickler.py * Fix formatting issue in dill_pickler.py * Remove apache license from code_object_pickler.py * Add apache license to code_object_pickler.py
shunping
added a commit
that referenced
this pull request
Jul 30, 2025
* Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per #35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in #35137 * keep in sync with iobase changes in #35137 * [Website] add akvelon case study (#34943) * feat: add akvelon logo * feat: add akvelon case study * fix: remove white space * feat: add akvelon to main page * feat: use new images * fix: typos * fix: change order of akvelon case-study * fix: update text * fix: update mainPage text * fix: update images * fix: about akvelon section update * fix: update akvelon card * fix: update akvelon header * fix: update code tag * fix: update about akvelon * fix: update date and order * fix: add link and change img * fix: change CDAP text * fix: add bold weight * fix: solve conflicts * fix: remove unused code * fix: delete whitespace * fix: indents format * fix: add bold text --------- Co-authored-by: Bulat Safiullin <[email protected]> * added the rate test for GenerateSequence (#35108) * added the rate test for GenerateSequence * keep the master yaml * Re-enable logging after importing vllm. (#35103) Co-authored-by: Claude <[email protected]> * Deprecate Java 8 (#35064) * Deprecate Java 8 * Java 8 client now using Java 11 SDK container * adjust non LTS fallback to use the next LTS instead of the nearest LTS. Previously Java18 falls back to Java17, which won't work * Emit warning when Java 8 is used. Java8 is still supported until Beam 3.0 * Clean up subproject build file requiring Java9+ * Require java 11 to build SDK container * fix workflow * Simplify XVR test workflow * Fix Samza PVR * Remove beam college banners (#35123) * feat: change text (#35130) Co-authored-by: Bulat Safiullin <[email protected]> * Update Custom Remote Inference example to use RemoteModelHandler (#35066) * Update Custom Remote Inference example to use RemoteModelHandler * restore old kernel metadata * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb Co-authored-by: Danny McCormick <[email protected]> * Add DLQ --------- Co-authored-by: Danny McCormick <[email protected]> * Remove Java 8 container (#35125) * add extra_transforms block documentation to chain transform documentation (#35101) * add note about testing (#35075) * fix whitespaces/newlines * fixes per #35137 (review) * implement pre_finalize_windowed and finalize_windowed_write in FileBasedSink handle file naming based on windowing for streaming in internal methods TextIO - expose the new filebasedsink capabilities to write in streaming * update changes.md * fix formatting * space formatting * space nagging * keep in sync with refactor in #35137 * keep in sync with iobase changes in #35137 * [Website] update akvelon case study: update text and fix landing page (#35133) * feat: change text * fix: add learn more to quotes Akvelon --------- Co-authored-by: Bulat Safiullin <[email protected]> * Fix PostCommit Python Xlang IO Dataflow job (#35131) * Add support for iterable type * Fix formatting * Bump google.golang.org/grpc from 1.72.0 to 1.72.2 in /sdks (#35113) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.0 to 1.72.2. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.0...v1.72.2) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.72.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cloud.google.com/go/bigquery from 1.67.0 to 1.69.0 in /sdks (#35061) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.67.0 to 1.69.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.67.0...spanner/v1.69.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-version: 1.69.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add known issues. (#35138) * Add known issues. * Add fixes notes. --------- Co-authored-by: Claude <[email protected]> * Bump @octokit/plugin-paginate-rest and @octokit/rest (#34167) Bumps [@octokit/plugin-paginate-rest](https://github.com/octokit/plugin-paginate-rest.js) to 11.4.3 and updates ancestor dependency [@octokit/rest](https://github.com/octokit/rest.js). These dependencies need to be updated together. Updates `@octokit/plugin-paginate-rest` from 2.17.0 to 11.4.3 - [Release notes](https://github.com/octokit/plugin-paginate-rest.js/releases) - [Commits](octokit/plugin-paginate-rest.js@v2.17.0...v11.4.3) Updates `@octokit/rest` from 18.12.0 to 21.1.1 - [Release notes](https://github.com/octokit/rest.js/releases) - [Commits](octokit/rest.js@v18.12.0...v21.1.1) --- updated-dependencies: - dependency-name: "@octokit/plugin-paginate-rest" dependency-type: indirect - dependency-name: "@octokit/rest" dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Explicitly handle singleton iterators instead of using helper and catching exceptions which may be from generating iterable (#35124) * Build last snapshot against RC00 tag instead of release branch (#35142) The last snapshot Beam Java SDK (aka "RC00" release) build is triggered manually, to verify a RC1 build will be successful. It has been built against release-2.xx branch, where the Dataflow container tag replaced from beam-master to the 2.xx.0 However, the versioned containers are not yet released, causing a timing gap that Beam 2.xx.0-SNAPSHOT won't work on Dataflow between release branch cut and RC1 rolled out to Dataflow Since we now have a v2.xx.0-RC00 tag, build RC00 against this tag resolves the issue. * Bump nodemailer from 6.7.5 to 6.9.9 in /scripts/ci/issue-report (#35143) Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.7.5 to 6.9.9. - [Release notes](https://github.com/nodemailer/nodemailer/releases) - [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md) - [Commits](nodemailer/nodemailer@v6.7.5...v6.9.9) --- updated-dependencies: - dependency-name: nodemailer dependency-version: 6.9.9 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix tests affected by Java 8 container turn down (#35145) * Fix tests affected by Java 8 container turn down * still use Java 8 for Samza runner * Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#35146) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.79.3 to 1.80.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.79.3...service/s3/v1.80.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.80.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix jdbc transform validation (#35141) * fix jdbc transform validation * add test * annotations * spotless * Fix Java Example ARM PostCommit * fix: add missed word (#35163) Co-authored-by: Bulat Safiullin <[email protected]> * Add postcommit yaml xlang workflow and split tests accordingly (#35119) * move precommit yaml xlang to postcommit * add json file to trigger path * update pull request targets * add readme workflow changes * add cloud sdk setup * switch out to ubuntu-latest * add back precommit workflow * switch names and add postcommit * switch out to postCommitYamlIT * add test_files_dir flag * add conftest.py file for capturing directory flag * shift yaml tests around to appropriate locations * add back precommit to readme * add license for conftest.py * revert precommit to previous name * remove github.event.comment.body trigger * Replace usages of deprecated pkg_resources package (#35153) * Remove usages of deprecated pkg_resources package * use stdlib importlib.resources * remove extra comma * linting * import order * Improve error message when accidentally using PBegin/Pipeline (#35156) * Create test * Implement new error message * Add beam.Create into unit test pipeline * add friendly error message for when transform is applied to no output (#35160) * add friendly error message for when transform is applied to no output * update test name * Fix pubsub unit tests that depend on old behavior * Add warning if temp location bucket has soft delete enabled for Go SD… (#34996) * Add warning if temp location bucket has soft delete enabled for Go SDK (resolves #31606) * Corrected Formatting * Applied suggested changes * Constrain DequeCoder type correctly, as it does not support nulls The DequeCoder uses ArrayDeque internally, which disallows null elements. We could switch Deque implementations, but this change is better. To quote the JDK docs: "While Deque implementations are not strictly required to prohibit the insertion of null elements, they are strongly encouraged to do so. Users of any Deque implementations that do allow null elements are strongly encouraged not to take advantage of the ability to insert nulls. This is so because null is used as a special return value by various methods to indicated that the deque is empty." * Do not overwrite class states if a cached dynamic class is returned in cloudpickle.load (#35063) * Fix class states overwritten after cloudpickle.load * Fix further * Fix lint * Make SDK harness change effective on Iceberg Dataflow test (#35173) * Fix beam_PostCommit_Java_Examples_Dataflow_V2 (#35172) * [YAML]: Update postgres IT test and readme (#35169) * update postgres test without driver_class_name * update readme on how to run integration tests * fix misspelling * fix trailing whitespace * Bump Java beam-master container (#35170) * Make WindowedValue a public interface The following mostly-automated changes: - Moved WindowedValue from util to values package - Make WindowedValue an interface with companion class WindowedValues * Run integration tests for moving WindowedValue and making public * Add timer tests to make sure event-time timer firing at the right time. (#35109) * Add timer tests to make sure event-time timer firing at the right time. * Add more tests. * Disable the failed event-time timer tests for FnApiRunner. * Fix lints and reformat. * Disable another new test in FnApiRunnerTest and PortableRunnerTest due to flakiness. * Disable a new test in FlinkRunnerTest * Take out the early firing test case because it depends on bundle size. * change to ubuntu-20.04 (#35182) * Fix IntelliJ sync project failure due to circular Beam dependency (#35167) * Fix IntelliJ sync project failure due to circular Beam dependency * address comments * Update workflows categories (#35162) * Add cloudpickle coder. (#35166) * Move examples from sql package (#35183) * Fix the beam interactive install problem when on Google Colab (#35148) * Fix Google Colab Issue * Update CHANGES.md * Bump github.com/docker/docker in /sdks (#35112) Bumps [github.com/docker/docker](https://github.com/docker/docker) from 28.0.4+incompatible to 28.2.2+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v28.0.4...v28.2.2) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-version: 28.2.2+incompatible dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats-server/v2 from 2.11.3 to 2.11.4 in /sdks (#35161) Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.11.3 to 2.11.4. - [Release notes](https://github.com/nats-io/nats-server/releases) - [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml) - [Commits](nats-io/nats-server@v2.11.3...v2.11.4) --- updated-dependencies: - dependency-name: github.com/nats-io/nats-server/v2 dependency-version: 2.11.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Touch JPMS test trigger file * Use staged SDK harness & Dataflow worker jar in JPMS tests * Bump cloud.google.com/go/storage from 1.52.0 to 1.55.0 in /sdks (#35114) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.52.0 to 1.55.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.52.0...spanner/v1.55.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.55.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/nats-io/nats.go from 1.42.0 to 1.43.0 in /sdks (#35147) Bumps [github.com/nats-io/nats.go](https://github.com/nats-io/nats.go) from 1.42.0 to 1.43.0. - [Release notes](https://github.com/nats-io/nats.go/releases) - [Commits](nats-io/nats.go@v1.42.0...v1.43.0) --- updated-dependencies: - dependency-name: github.com/nats-io/nats.go dependency-version: 1.43.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move non-dill specific code out of dill_pickler.py (#35139) * Create consistent_pickle.py * Update dill_pickler.py * Update consistent_pickle.py * Add Apache license to consistent_pickle.py * Update dill_pickler.py * Update and rename consistent_pickle.py to code_object_pickler.py * Update dill_pickler.py to change consistent_pickle to code_object_pickler * Fix formatting issue in code_object_pickler.py * Fix formatting issue in dill_pickler.py * Remove apache license from code_object_pickler.py * Add apache license to code_object_pickler.py * Fix some lints introduced in a recent PR. (#35193) * small filesystem fixes (#34956) * Fix typehint for get_filesystem * Fix bug with creating files in cwd with LocalFileSystem * lint * revert precommit config change * isort * add file to start PR builder * Revert "add file to start PR builder" This reverts commit f80b361. * fix isort again * [YAML] Add a spec provider for transforms taking specifiable arguments (#35187) * Add a test provider for specifiable and try it on AnomalyDetection. Also add support on callable in spec. * Minor renaming * Fix lints. * Touch trigger files to test WindowedValueReceiver in runners * Introduce WindowedValue receivers and consolidate runner code to them We need a receiver for WindowedValue to create an OutputBuilder. This change introduces the interface and uses it in many places where it is appropriate, replacing and simplifying internal runner code. * Eliminate nullness errors from ByteBuddyDoFnInvokerFactory and DoFnOutputReceivers * Fix null check when fetching driverJars from value provider * Fix PostCommit Python ValidatesRunner Samza / Spark jobs (#35210) * Skip Samza and Spark runner tests * Fix formatting * Update pypi documentation 30145 (#34329) * Add Pypi friendly documentation * provided full link path * Add more YAML examples involving Kafka and Iceberg (#35151) * Add more YAML examples involving Kafka and Iceberg * Fix some missed details from rebasing * Adding unit tests for YAML examples * Clean up and address PR comments * Formatting * Formatting * Evaluate namedTuples as equivalent to rows (#35218) * Evaluate namedTuples as equivalent to rows * lint * Add a new experiment flag to enable real-time clock as processing time. (#35202) * Touch trigger files for lightweight runners * Eliminate WindowedValue.getPane() in preparation for making it a user-facing interface * Do not fail if there were failed deletes (#35222) * Force FnApiRunner in cases where prism can't handle use case (#35219) * Force FnApiRunner in cases where prism can't handle use case * yapf * Bump golang.org/x/net from 0.40.0 to 0.41.0 in /sdks (#35206) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.40.0 to 0.41.0. - [Commits](golang/net@v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.41.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix incorrect typehints generated by FlatMap with default identity function (#35164) * create unit test * add some unit tests * fix bug where T is considered iterable * update strip_iterable to return Any for "stripped iterable" type of TypeVariable * remove typehint from identity function and add a test to test for proper typechecking * Move callablewrapp typehint test * Remove print * isort * isort * return any for yielded type of T * Parse values returned from Dataflow API to BoundedTrieData (#34738) * Parse struct returned from Dataflow API to BoundedTrieData fix checkstyle Use getBoundedTrie add debug log adapt to ArrayMap * Add test, clean up * Fix test: setTrie -> setBoundedTrie * Fix comments * Remove breaking PDone change (#35224) * Remove breaking PDone change * Update pipeline.py * Update pipeline.py * Update pipeline_test.py * Move none check * yapf * fmt * Generic Postgres + Cloudsql postgres embeddings. (#35215) * Add base Postgres vector writer, CloudSQL vector writer and refactor. * Trigger tests. * Linter fixes. * Fix test * Add back tests. Update changes.md. Fix unrelated lint. * Drop test tables. * Fix test * Fix tests. * Fix test. * Fix tests. --------- Co-authored-by: Claude <[email protected]> * Allow only one thread at a time to start the VLLM server. (#35234) * [IcebergIO] Create namespaces if needed (#35228) * create namespaces dynamically * cleanup namespaces in ITs * optimization * Support configuring flush_count and max_row_bytes of WriteToBigTable (#34761) * Update CHANGES.md (#35242) Highlight improvements to the vllm model handler. * [Beam SQL] Implement Catalog and CatalogManager (#35223) * beam sql catalog * api adjustment * cover more naming syntax (quotes, backticks, none) * spotless * fix * add documentation and cleanup * rename to dropCatalog; mark BeamSqlCli @internal; rename to EmptyCatalogManager * use registrars instead; remove initialize() method from Catalog * cleanupo * [IcebergIO] Fix conversion logic for arrays of structs and maps of structs; fix output Schema resolution with column pruning (#35230) * fix complex types; filx filter schema logic * update Changes * add missing changes from other PRs * trigger ITs * make separate impl for iterable * fix * fix long_description when the md file cannot be found (#35246) * fix long_description when the md file cannot be found * yapf * [IcebergIO] Create tables with a partition spec (#34966) * create table with partition spec * add description and regenerate config doc; trigger tests * spotless * log partition spec * fixes * Fix typo java-dependencies.md (#35251) * Adding project and database support in write transform for firestoreIO (#35017) * Adding project and database support in write transform for firestoreIO * Spotless Apply * Resolving comments * Removing useless comments * public to private * Spotless apply * Fix a logical type issue about JdbcDateType and JdbcTimeType (#35243) * Fix a logical type issue about JdbcDateType * Fix typo and also fix the logical class for java time. * Get rid of the workaround on logical type registration. Trigger tests. * Fix lints. * Remove internal code. (#35239) Co-authored-by: Claude <[email protected]> * enable setting max_writer_per_bundle for avroIO and other IO (#35092) * enable setting max_writer_per_bundle for avroIO and other IO * enable setting max_writer_per_bundle for avroIO and other IO * enable for TFRecordIO and corrections * Updated standard_external_transforms.yaml * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. (#35252) * [Java FnApi] Fix race in BeamFnStatusClient by initializing all fields before starting rpc. * spotless * [IcebergIO] Integrate with Beam SQL (#34799) * add Iceberg table provider and tests * properties go in the tableprovider initialization * tobuilder * streaming integration test * spotless * extend test to include multi nested types; fix iceberg <-> conversion logic * add to changes.md * spotless * fix tests * clean * update CHANGES * add projection pushdown and column pruning * spotless * fixes * fixes * updates * sync with HEAD and use new Catalog implementation * mark new interfaces @internal * spotless * fix unparse method * update changes (#35256) * [YAML]: fix postcommit oracle bug and reorganize postcommit tests (#35191) * fix postcommit oracle test * add revision * switch to hosted runner to try with kafka test * add extended timeout * upgrade to 4.10 testcontainers * switch out to redpanda for kafka * remove redpandacontainer * tmp comment * add postgres comment * revert to old kafkaContainer * revert commented out code and revert testcontainer version change * change mysql image version * revert image change * revert image change again :) * add comments to mysql again * shift post commit files to different folders * rename to Data version * add databases version * add messaging version * update readme for three post commits * update gradle with new post commits * fix job names * uncomment fixture on mysql * switch back to one workflow and update readme as such * remove old workflow files * update order and remove comment * update gradle with parameterized options * update gradle command calls to correct location * update workflow to three jobs explicit * add back Bigquery deselect * fix mysql teardown error * Simplify down to one from three explicit jobs * remove tab * remove Data * fix parsing parameters * update hadoop prefix (#35257) * Set go version 1.24.4 (#35261) * Set go version 1.24.4 * Set go version 1.24.4 * upgrade org.apache.parquet:parquet-avro to to 1.15.2 (#35037) * upgrade org.apache.parquet:parquet-avro to to 1.15.2 * keep iceberg to 1.6.1 * use 1.9.0 for iceberg * run all iceberg tests * require Java 11 * fixed java11 for iceberg * switched back to iceberg_version = "1.6.1" * merged master --------- Co-authored-by: Vitaly Terentyev <[email protected]> * Bump google.golang.org/grpc from 1.72.2 to 1.73.0 in /sdks (#35236) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.72.2 to 1.73.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.72.2...v1.73.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-version: 1.73.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add more doc strings for integration tests (#35171) * [IcebergIO] extend table partitioning to Beam SQL (#35268) * add sql partitioning * trigger ITs and add to CHANGES * simplify test * include emoty namespace fix * add error when using partitioning for unsupported tables * fix test * Moving to 2.67.0-SNAPSHOT on master branch. * Update CHANGES.md to have fields for 2.67.0 release * add free disk space step (#35260) * [yaml]: Fix post commit yaml once more (#35273) * switch back to self hosted runner and comment out most of kafka test for now * add git issue to track * revision * remove free space step - seems to be causing more issues than helping * Polish anomaly detection notebook and get ready to be imported in public devsite. (#35278) * Polish anomaly detection zscore notebook for public doc. * Adjust formatting. * Adjust formatting. * Suppress Findbugs (#35274) * Support Java 17+ compiled Beam components for build, release, and xlang tests (#35232) * Install JDK 21 in release build Support Beam components requiring Java17+ for release workflows They will be compiled with JDK21 with byte code compatibility configured by applyJavaNature(requireJavaVersion) * Use Java 21 for Python PostCommit * Honor JAVA_HOME in JavaJarServer checks and tests * Disable Debezium test on Java17- * add example line * [IO] Update Debezium in DebeziumIO to 3.1.1 (#34763) * Updating Debezium IO to 3.1.1 Enforce JDK17 in build.gradle of Debezium IO * adjust to review comments by @Abacn * cleanup * mention which PR needs to merge before unpinning * Mention Beam version 2.66 in the README for Debezium --------- Co-authored-by: Shunping Huang <[email protected]> * Document BQ Storage API pipeline options (#35259) * Document BQ Storage API pipeline options * Fix whitespace * Fix whitespace * Speed up StreamingDataflowWorkerTest by removing 10 second wait from shutdown path. (#35275) Takes test time from 5m18s to 2m40s. The previous shutdown() call doesn't do anything since that just means future scheduling won't trigger but we only schedule on the executor once. Also cleanup test logs by making sure to stop all workers we start so they don't continue to run in the background and log. This shutdown paths is only used in testing. * Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301) Bumps [org.ajoberstar.grgit:grgit-gradle](https://github.com/ajoberstar/grgit) from 4.1.1 to 5.3.2. - [Release notes](https://github.com/ajoberstar/grgit/releases) - [Commits](ajoberstar/grgit@4.1.1...5.3.2) --- updated-dependencies: - dependency-name: org.ajoberstar.grgit:grgit-gradle dependency-version: 5.3.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google.golang.org/api from 0.235.0 to 0.237.0 in /sdks (#35302) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.235.0 to 0.237.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.235.0...v0.237.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-version: 0.237.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump github.com/linkedin/goavro/v2 from 2.13.1 to 2.14.0 in /sdks (#35205) Bumps [github.com/linkedin/goavro/v2](https://github.com/linkedin/goavro) from 2.13.1 to 2.14.0. - [Release notes](https://github.com/linkedin/goavro/releases) - [Changelog](https://github.com/linkedin/goavro/blob/master/debug_release.go) - [Commits](linkedin/goavro@v2.13.1...v2.14.0) --- updated-dependencies: - dependency-name: github.com/linkedin/goavro/v2 dependency-version: 2.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update spotbugs version, fix runner ubuntu version, fix found spotbugs issues (#35303) * feat: Add option to control resource cleanup failure for IT (#35287) * Revert "Bump org.ajoberstar.grgit:grgit-gradle from 4.1.1 to 5.3.2 (#35301)" (#35305) This reverts commit 0fd77ce. * Add PeriodicStream in the new time series folder. (#35300) * Add PeriodicStream in the new time series folder. * Add some more docstrings and minor fix on test name. * Fix lints and docs. * try buildah to replace kaniko (#35289) * try buildah to replace kaniko * trigger post-commit * Adding error handler for SpannerReadSchemaTransformProvider and missi… (#35241) * Adding error handler for SpannerReadSchemaTransformProvider and missing tests for SpannerSchemaTransformProvider * Removed not used logging * Spotless Apply * Spotless Apply * Spotless Apply * Typo correction * requests vulnerability. (#35308) * [IcebergIO] create custom java container image for tests (#35307) * create custom java container image for tests * syntax * eval depends on df * add streaming support to iobase (python) (#35137) * Add support for streaming writes in IOBase (Python) * add triggering_frequency in iobase.Sink * fix whitespaces/newlines * fixes per #35137 (review) * refactor for code redability * refactor _expand_unbounded , default num_shards to 1 , if undef or 0 * fix formatter * space * keep num_shards = 0 the same as before for bounded write * reafactor and code clean --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: bullet03 <[email protected]> Co-authored-by: Bulat Safiullin <[email protected]> Co-authored-by: liferoad <[email protected]> Co-authored-by: claudevdm <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Yi Hu <[email protected]> Co-authored-by: Danny McCormick <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: Derrick Williams <[email protected]> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: scwhittle <[email protected]> Co-authored-by: Radosław Stankiewicz <[email protected]> Co-authored-by: Hai Joey Tran <[email protected]> Co-authored-by: Tanu Sharma <[email protected]> Co-authored-by: Kenneth Knowles <[email protected]> Co-authored-by: Minbo Bae <[email protected]> Co-authored-by: Shunping Huang <[email protected]> Co-authored-by: Chenzo <[email protected]> Co-authored-by: kristynsmith <[email protected]> Co-authored-by: Rakesh Kumar <[email protected]> Co-authored-by: Charles Nguyen <[email protected]> Co-authored-by: tvalentyn <[email protected]> Co-authored-by: Ahmed Abualsaud <[email protected]> Co-authored-by: Minh Son Nguyen <[email protected]> Co-authored-by: Amar3tto <actions@GitHub Actions 1000279405.local> Co-authored-by: Vitaly Terentyev <[email protected]> Co-authored-by: Tobias Kaymak <[email protected]> Co-authored-by: Veronica Wasson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Move non-dill specific code from dill_pickler.py into new file code_object_pickler.py
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.