Skip to content

Upgrade spark-operator to v2.1.1#6865

Merged
pingsutw merged 21 commits into
flyteorg:masterfrom
kevinliao852:upgrade-spark-operator-2.1.1
Apr 30, 2026
Merged

Upgrade spark-operator to v2.1.1#6865
pingsutw merged 21 commits into
flyteorg:masterfrom
kevinliao852:upgrade-spark-operator-2.1.1

Conversation

@kevinliao852
Copy link
Copy Markdown
Contributor

Tracking issue

Related #6686, #6348

Why are the changes needed?

  • These changes are required to support newer Spark Operator features

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

  • added: For new features.
  • changed: For changes in existing functionality.
  • deprecated: For soon-to-be-removed features.
  • removed: For features being removed.
  • fixed: For any bug fixed.
  • security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Signed-off-by: Kevin Liao <q85292542000@gmail.com>
@welcome
Copy link
Copy Markdown

welcome Bot commented Jan 20, 2026

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

  • Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
  • Sign off your commits (Reference: DCO Guide).

@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 20, 2026

Codecov Report

❌ Patch coverage is 97.72727% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 56.96%. Comparing base (91d6af1) to head (f864c51).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
flyteplugins/go/tasks/plugins/k8s/spark/spark.go 97.72% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6865      +/-   ##
==========================================
- Coverage   56.96%   56.96%   -0.01%     
==========================================
  Files         931      931              
  Lines       58275    58276       +1     
==========================================
- Hits        33199    33198       -1     
- Misses      22017    22019       +2     
  Partials     3059     3059              
Flag Coverage Δ
unittests-datacatalog 53.51% <ø> (ø)
unittests-flyteadmin 53.15% <ø> (ø)
unittests-flytecopilot 43.06% <ø> (ø)
unittests-flytectl 64.09% <ø> (-0.05%) ⬇️
unittests-flyteidl 75.71% <ø> (ø)
unittests-flyteplugins 60.19% <97.72%> (+<0.01%) ⬆️
unittests-flytepropeller 53.71% <ø> (ø)
unittests-flytestdlib 62.62% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Update viper to v1.11.0 in every affected go.mod, resolving previous update that only covered one subproject.

Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
…ired

Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
@kevinliao852 kevinliao852 marked this pull request as ready for review February 4, 2026 11:54
Copy link
Copy Markdown
Member

@machichima machichima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, just a few comments

Comment thread flyteplugins/go/tasks/plugins/k8s/spark/spark_test.go
}

if sparkJob.GetExecutorPath() != "" {
if sparkJob.GetExecutorPath() != "" && sparkConfig["spark.pyspark.python"] == "" && sparkConfig["spark.pyspark.driver.python"] == "" {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add the changes here to the PR description?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rechecked this logic and realized the current conditional handling could be problematic. I’d prefer to revert to the previous behavior for now. Let me know if you see any concerns.

Comment thread flyteplugins/go.mod
@kevinliao852
Copy link
Copy Markdown
Contributor Author

As discussed with @pingsutw , we should run the tests against the v1 CRD to match the currently compatible operator version.

kevinliao852 and others added 5 commits February 25, 2026 09:29
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
…tor-2.1.1

Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
pingsutw
pingsutw previously approved these changes Apr 13, 2026
@pingsutw pingsutw changed the title [WIP] Upgrade spark-operator to v2.1.1 and Go to v1.24.1 Upgrade spark-operator to v2.1.1 and Go to v1.24.1 Apr 13, 2026
…tor-2.1.1

Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
…tor-2.1.1

Signed-off-by: Kevin Liao <q85292542000@gmail.com>
@kevinliao852 kevinliao852 changed the title Upgrade spark-operator to v2.1.1 and Go to v1.24.1 Upgrade spark-operator to v2.1.1 Apr 18, 2026
pingsutw
pingsutw previously approved these changes Apr 18, 2026
pingsutw
pingsutw previously approved these changes Apr 30, 2026
…tor-2.1.1

Signed-off-by: Kevin Liao <q85292542000@gmail.com>
@pingsutw pingsutw merged commit cd7371f into flyteorg:master Apr 30, 2026
58 checks passed
nuthalapativarun pushed a commit to nuthalapativarun/flyte that referenced this pull request May 1, 2026
Signed-off-by: Kevin Liao <q85292542000@gmail.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Varun Nuthalapati <nuthalapativarun@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants