Allow the plugin developer to implement a fast merge which will be invoked when:
- Encryption is disabled.
- Compression is disabled or the compression codec supports concatenation of serialized streams.
- The
spark.shuffle.unsafe.fastMergeEnabled option is true.
- The plugin supports fast merge.
Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills.
Allow the plugin developer to implement a fast merge which will be invoked when:
spark.shuffle.unsafe.fastMergeEnabledoption is true.Note the performance of this fast merge function could seriously impact the performance of Spark SQL joins with multiple spills.