[Java]Best practice with Apache/Spark

[jayhan94]

Feature Request

Is there any best practice with apache/spark? Will the community implement such a module?

Is your feature request related to a problem? Please describe

No response

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[chaokunyang]

Hi @jayhan94 , we don't have such documents currently. A better fury integration with spark/flink would need to change the source code of serialization module in spark/flink, which is beyond the scope of this project. Maybe in future we can submit several proposal to spark/flink communities.

Currently, if you want to use fury in spark/flink, you can update your driver program to add several chained(narrow dependency in spark) serialization/deserialization operators.

Here is a simple spark rdd example:

val lines = sc.textFile("data.txt")
val structSet = lines.map(s => Json.parse(s, Struct.class))
kvset = structSet.map(s => (s.key, fury.serialize(s)))
kvset.groupByKey().map(t => (t._1, fury.deserialize(t._2.first))).collect.foreach(println)

Flink program will be similiar:

DataStream<Struct> dataStream = xxxstream.map(s -> Json.parse(s, Struct.class));
DataStream<byte[]> byteStream = dataStream.map(s -> json.serialize(s));
byteStream.rebalance().map(bytes -> (Struct)fury.deserialize(bytes));

[jayhan94]

@chaokunyang Thanks for your reply. I don't learn about the serializer of rdd. I meant to implement spark.serializer based on fury which may be helpful to the shuffle process just like KryoSerializer.