data-processing.md

Ways to use Apache Spark™ clusters in {{ ml-platform-name }}

{{ dataproc-full-name }} allows you to deploy Apache Spark™ clusters. You can use {{ dataproc-name }} clusters to run distributed training.

Cluster deployment options {#types}

To work with {{ dataproc-name }} clusters in {{ ml-platform-name }}, you can use the following:

• Spark connector
• Livy session

If you have no existing {{ dataproc-name }} clusters or you need a cluster for a short time, use temporary {{ dataproc-name }} clusters. You can create them using the following:

• Spark connector (preferred)
• {{ dataproc-name }} template

Regardless of the deployment option, all {{ dataproc-name }} clusters are charged based on the {{ dataproc-name }} pricing policy.

Setting up a {{ ml-platform-name }} project to work with {{ dataproc-name }} clusters {#settings}

{% include preferences %}

See also {#see-also}

• {#T}
• {#T}
• {#T}