Spline agent require more RAM for Spark application

[dishkakrauch]

Hi everybody!
We faced specific attitute running Spark app with Spline agent - it needs twice more RAM or twice less CPU per executor to build lineage. If we disable spline agent our app runs on 40 executors (5 CPU each) and 64 GB ram each. But if we enable spline agent our app fails with OOM on these settings. Does anybody know how to cure it?
Here is short log of failed stage:

Failure Reason:

Job aborted due to stage failure: Task 55 in stage 20.0 failed 4 times, most recent failure: Lost task 55.3 in stage 20.0 (TID 714) (10.202.72.205 executor 89): ExecutorLostFailure (executor 89 exited caused by one of the running tasks) Reason: 
The executor with id 89 exited with exit code 137(SIGKILL, possible container OOM).

[rdd at <unknown>:0]
+details:

org.apache.spark.sql.Dataset.rdd(Dataset.scala:3849)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.base/java.lang.reflect.Method.invoke(Unknown Source)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror0.jinvokeraw(JavaMirrors.scala:404)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaMethodMirror.jinvoke(JavaMirrors.scala:380)
scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror.apply(JavaMirrors.scala:396)
za.co.absa.spline.commons.reflect.ValueExtractor.$anonfun$scalaReflectClass$1(ValueExtractor.scala:77)
scala.util.Try$.apply(Try.scala:213)
za.co.absa.spline.commons.reflect.ValueExtractor.scalaReflectClass(ValueExtractor.scala:65)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClass(ValueExtractor.scala:58)
za.co.absa.spline.commons.reflect.ValueExtractor.reflectClassHierarchy(ValueExtractor.scala:48)
za.co.absa.spline.commons.reflect.ValueExtractor.extract(ValueExtractor.scala:33)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:114)
za.co.absa.spline.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:120)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$dump$default$2$1(ObjectStructureDumper.scala:37)
za.co.absa.spline.harvester.logging.ObjectStructureDumper$.$anonfun$objectToStringRec$2(ObjectStructureDumper.scala:97)
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)

[wajda]

Hi @dishkakrauch
what Spark version and distribution are you using?
We only had a similar report once from our internal user that was running their jobs on AWS Glue 5. I wonder if the issue is specific for either a particular Spark version, or a particular job pattern.
Can you share more details please?
Ideally if you could provide a sample job that exhibits the behavior, so we could test and profile it to understand what's going on?

[dishkakrauch]

Hi @dishkakrauch what Spark version and distribution are you using? We only had a similar report once from our internal user that was running their jobs on AWS Glue 5. I wonder if the issue is specific for either a particular Spark version, or a particular job pattern. Can you share more details please? Ideally if you could provide a sample job that exhibits the behavior, so we could test and profile it to understand what's going on?

Hi @wajda!
We're using Spark 3.5.5 but it's not related to any cloud provider. It's running inside K8S cluster. Sure, I can share more details - there is no problem. I'll come back with samples or snippets a little bit later.

[wajda]

Hi @dishkakrauch did you have a chance to locate the issue?
I've recently released Spark Agent 2.3.0 that is lighter on start-up and potentially could consume less memory. Can you give it a try?

[sborshikovpa]

Hi @wajda

ods.py

We have prepared a reproducible Spark application that demonstrates a significant performance degradation when Spline is enabled.

The same application behaves as follows:

without Spline: ~20 minutes
with Spline: ~32 minutes

The attached example is based on a synthetic dataset, but it reproduces the processing pattern of our production pipeline (large Delta tables, window function, broadcast joins, and Delta write).

Observation:

When comparing the runs, we also noticed that the application with Spline creates additional Spark jobs/stages.

The number of tasks increases, and in the run with Spline we see heavy stages like rdd at <unknown>. These stages are not present in the run without Spline.

[wajda]

Thank you @dishkakrauch, we'll take a close look at it ASAP.
What you describe sounds impossible as the Spline agent has nothing to do with the Spark job itself. It's merely a listener that is triggered after the job has finished. But I admit there is no magic. I wonder if there's any side effect that might be caused by the mere presence of the agent in the classpath... Anyways, will keep you updated.

[wajda]

What is a test setup? Like, Spark distribution, version, single node or cluster, YARN/no YARN, etc?

[dishkakrauch]

What is a test setup? Like, Spark distribution, version, single node or cluster, YARN/no YARN, etc?

Hi, you're wellcome!
Our setup:
Spark - 3.5.5
Delta Lake - 3.3.2
Spark cluster - Spark K8S Operator with Yunikorn

If you need full env ouput or spark logs we can shere it.
P.S. it seems to me that spline creates additional stages after spark does. Can't understand why.

[wajda]

@dishkakrauch

If you need full env output or spark logs we can shere it.

It would be helpful. Thank you!