grpc client addition

[gangavh1008]

Purpose of this PR

This PR adds gRPC client functionality to the Spark Operator, enabling it to submit Spark applications through a gRPC service instead of the default direct submission method(more details in document). This provides a more flexible and extensible architecture for Spark application submission.

Proposed changes:

<Change 1>
Core gRPC Infrastructure: Added a new GRPCSparkSubmitter implementation that implements the SparkApplicationSubmitter interface, allowing the operator to submit Spark applications via gRPC calls to an external service.
<Change 2>
Protocol Buffer Definitions: Introduced comprehensive protobuf definitions (spark_submit.proto) that define the gRPC service contract, including message types for SparkApplication, SparkApplicationSpec, and related Kubernetes objects, along with the SparkSubmitService gRPC service definition.
<Change 3>
Helm Chart Integration: Enhanced the Helm chart with configurable submitter options, including support for gRPC submitter configuration, native submit plugin sidecar container, and comprehensive deployment templates that support both default and gRPC submission modes.
### Change Category
Feature (non-breaking change which adds functionality)

Rationale

This enhancement provides several benefits:
Extensibility: Allows integration with custom Spark submission logic through gRPC services
Separation of Concerns: Decouples the submission logic from the operator, enabling custom implementations
Flexibility: Supports both default submission and gRPC-based submission modes
Native Plugin Support: Includes sidecar container support for native submit plugins
Backward Compatibility: Maintains existing functionality while adding new capabilities

Checklist

[x] I have conducted a self-review of my own code.
[x] I have updated documentation accordingly.
[x] I have added tests that prove my changes are effective or that my feature works.
[x] Existing unit tests pass locally with my changes.

Additional Notes

The PR introduces significant new dependencies including gRPC and protobuf libraries
Includes comprehensive Helm chart configuration for both development and production deployments
Provides example configuration files (values-grpc.yaml) for easy adoption
Maintains backward compatibility by defaulting to the existing submission method
Includes health checks and resource management for the native submit plugin sidecar
The changes represent a substantial enhancement to the Spark Operator architecture, adding approximately 11,000+ lines of new code while maintaining the existing functionality intact.

[ChenYi015]

@gangavh1008 Thanks for the great contribution!
/hold for review
/assign @vara-bonthu @nabuskey @jacobsalway

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jacobsalway. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

I cannot find this image. Could yo provide more details about the implementation?

[nabuskey]

I'd love to have different methods of submitting and having this interface is going to be a very good addition. I would like to hear what others think about the maintenance overhead.

[nabuskey]

Wonder if this bit is going to become a pain when it comes to maintaining the repo.

Here, we do direct conversion of the spark app object to the grpc message type. This means we now have to maintain two types with mostly identical fields and update both types when a new field is introduced. Does it make sense to use plain old REST instead? GRPC does give us better performance but I wonder if it's worth while.

We could convert the spark app object to a json object, then convert that into proto message. But this comes with performance penalty from conversions.