`JJAVA_STARTUP_SCRIPT can't start up spark

[clhedrick]

With ijava, the following will set up spark:

"IJAVA_STARTUP_SCRIPT":"import org.apache.spark.; import org.apache.spark.sql.; import org.apache.spark.api.java.; import org.apache.spark.ml.linalg.; import org.apache.spark.sql.types.; import org.apache.spark.mllib.random.; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null; class initspark{ public static void init() { String appName = "app"; String master = "local[4]"; conf = new SparkConf().setAppName(appName).setMaster(master); jsc = new JavaSparkContext(conf); spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate(); sc=jsc.sc(); } }; "

It doesn't work with jjava. I'm not clear why not.

The closest I could come is the following

"IJAVA_STARTUP_SCRIPT":"import org.apache.spark.; import org.apache.spark.sql.; import org.apache.spark.api.java.; import org.apache.spark.ml.linalg.; import org.apache.spark.sql.types.; import org.apache.spark.mllib.random.; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null; class initspark{ public static void init() { String appName = "app"; String master = "local[4]"; conf = new SparkConf().setAppName(appName).setMaster(master); jsc = new JavaSparkContext(conf); spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate(); sc=jsc.sc(); } }; "

the `user than has to type "initspark.init()"

[andrus]

Hi there. Without checking the actual script, one thing that I noticed is the var name. The name of the variable should be JJAVA_STARTUP_SCRIPT per https://dflib.org/jjava/docs/1.x/#environment-vars .

[clhedrick]

That was for ijava. Obviously with jjava I used jjava. On Jun 12, 2025, at 3:28 PM, Andrus Adamchik ***@***.***> wrote:  [https://avatars.githubusercontent.com/u/42668?s=20&v=4]andrus left a comment (dflib/jjava#67)<#67 (comment)> Hi there. Without checking the actual script, one thing that I noticed is the var name. The name of the variable should be JJAVA_STARTUP_SCRIPT per https://dflib.org/jjava/docs/1.x/#environment-vars . — Reply to this email directly, view it on GitHub<#67 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAORUCDZ7OAMFOI4OTXVMQL3DHIGZAVCNFSM6AAAAAB7GHUWGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRXHEZDMMBUGU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[andrus]

Actually I am able to reproduce this myself:

1. Download an unpack Spark spark-3.5.6-bin-hadoop3
2. Set classpath:

export JJAVA_CLASSPATH='/path/to/spark-3.5.6-bin-hadoop3/jars/*.jar'

3. Set the startup script:

export JJAVA_STARTUP_SCRIPT='
import org.apache.spark.api.java.*;
import org.apache.spark.sql.*;
import org.apache.spark.ml.linalg.*;
import org.apache.spark.sql.types.*;
import org.apache.spark.mllib.random.*;

SparkConf conf = null;
JavaSparkContext jsc = null;
SparkSession spark = null;
SparkContext sc = null;
class initspark{
    public static void init() {
        String appName = "app";
        String master = "local[4]";
        conf = new SparkConf().setAppName(appName).setMaster(master);
        jsc = new JavaSparkContext(conf);
        spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate();
        sc=jsc.sc();
    }
}'

Start jupyter and open any Java notebook. The following exception is printed in the logs (failing code details became visible after I implemented #68)

Exception in thread "main" org.dflib.jjava.execution.CompilationException: [snippet: [SparkConf conf = null;], status: REJECTED]
	at org.dflib.jjava.execution.CodeEvaluator.evalSingle(CodeEvaluator.java:150)
	at org.dflib.jjava.execution.CodeEvaluator.eval(CodeEvaluator.java:170)
	at org.dflib.jjava.execution.CodeEvaluator.init(CodeEvaluator.java:84)
	at org.dflib.jjava.execution.CodeEvaluator.eval(CodeEvaluator.java:163)
	at org.dflib.jjava.JavaKernel.evalRaw(JavaKernel.java:337)
	at org.dflib.jjava.JavaKernel.eval(JavaKernel.java:342)
	at org.dflib.jjava.JJavaExtension.install(JJavaExtension.java:28)
	at org.dflib.jjava.JavaKernel.handleExtensionLoading(JavaKernel.java:165)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.dflib.jjava.magics.MavenResolver.initImplicitExtensions(MavenResolver.java:119)
	at org.dflib.jjava.JavaKernel.<init>(JavaKernel.java:157)
	at org.dflib.jjava.JJava.main(JJava.java:99)

When I unset JJAVA_STARTUP_SCRIPT and copy the script into the notebook cell, everything works. We'll investigate.

[clhedrick]

Thanks. But that’s the version that works for me. What fails is if instead of defining an inIt class I actually do the initialization I the script. I’m using spark 4.0.0 and Java 21, if it matters. On Jun 14, 2025, at 10:49 AM, Andrus Adamchik ***@***.***> wrote:  [https://avatars.githubusercontent.com/u/42668?s=20&v=4]andrus left a comment (dflib/jjava#67)<#67 (comment)> Actually I am able to reproduce this myself: 1. Download an unpack Spark spark-3.5.6-bin-hadoop3 2. Set classpath: export JJAVA_CLASSPATH='/path/to/spark-3.5.6-bin-hadoop3/jars/*.jar' 1. Set the startup script: export JJAVA_STARTUP_SCRIPT=' import org.apache.spark.api.java.*; import org.apache.spark.sql.*; import org.apache.spark.ml.linalg.*; import org.apache.spark.sql.types.*; import org.apache.spark.mllib.random.*; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null; class initspark{ public static void init() { String appName = "app"; String master = "local[4]"; conf = new SparkConf().setAppName(appName).setMaster(master); jsc = new JavaSparkContext(conf); spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate(); sc=jsc.sc(); } }' Start jupyter and open any Java notebook. The following exception is printed in the logs: Exception in thread "main" org.dflib.jjava.execution.CompilationException: [snippet: [SparkConf conf = null;], status: REJECTED] at org.dflib.jjava.execution.CodeEvaluator.evalSingle(CodeEvaluator.java:150) at org.dflib.jjava.execution.CodeEvaluator.eval(CodeEvaluator.java:170) at org.dflib.jjava.execution.CodeEvaluator.init(CodeEvaluator.java:84) at org.dflib.jjava.execution.CodeEvaluator.eval(CodeEvaluator.java:163) at org.dflib.jjava.JavaKernel.evalRaw(JavaKernel.java:337) at org.dflib.jjava.JavaKernel.eval(JavaKernel.java:342) at org.dflib.jjava.JJavaExtension.install(JJavaExtension.java:28) at org.dflib.jjava.JavaKernel.handleExtensionLoading(JavaKernel.java:165) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.dflib.jjava.magics.MavenResolver.initImplicitExtensions(MavenResolver.java:119) at org.dflib.jjava.JavaKernel.<init>(JavaKernel.java:157) at org.dflib.jjava.JJava.main(JJava.java:99) When I unset JJAVA_STARTUP_SCRIPT and copy the script into the notebook cell, everything works. We'll investigate. — Reply to this email directly, view it on GitHub<#67 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAORUCGVQNMWEQYSYKBWXJL3DQZAJAVCNFSM6AAAAAB7GHUWGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNZSHAYDQMJRGE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[clhedrick]

The following works for me:

{
    "argv": [
        "/usr/lib/jvm/java-21-openjdk-amd64/bin/java",
        "-jar",
        "/common/system/venv/python3.12-jupyter/share/jupyter/kernels/jjspark/jjava-launcher.jar",
        "/common/system/venv/python3.12-jupyter/share/jupyter/kernels/jjspark/jjava.jar",
        "{connection_file}"
    ],
    "display_name": "Java (jjava), initspark.init() to set up Spark",
    "language": "java",
    "interrupt_mode": "message",
    "env": {
      "JJAVA_JVM_OPTS": "-Dio.netty.tryReflectionSetAccessible=true --add-exports java.base/sun.nio.ch=ALL-UNNAMED",
      "JAVA_HONE": "/usr/lib/jvm/java-21-openjdk-amd64/",
      "JJAVA_CLASSPATH": ".:/common/system/spark-4.0.0-bin-hadoop3/jars/*",
       "SPARK_HOME": "/common/system/spark-4.0.0-bin-hadoop3",
      "JJAVA_STARTUP_SCRIPT":"import org.apache.spark.*; import org.apache.spark.sql.*; import org.apache.spark.api.java.*; import org.apache.spark.ml.linalg.*; import org.apache.spark.sql.types.*; import org.apache.spark.mllib.random.*; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null;  class initspark{ public static void init() {  String appName = \"app\"; String master = \"local[4]\"; conf = new SparkConf().setAppName(appName).setMaster(master);  jsc = new JavaSparkContext(conf);  spark=SparkSession.builder().sparkContext(jsc.sc()).master(master).getOrCreate();  sc=jsc.sc();  }  }; "
    },
    "metadata": {
	"url": "https://github.com/dflib/jjava"
    }
}

However I have to type initspark.init(). (It also works as IJAVA_STARTUP_SCRIPT, which I didn't expect.)

Variations of it that try to run the code during startup fail. Indeed the following fails:

"JJAVA_STARTUP_SCRIPT":"import org.apache.spark.*; import org.apache.spark.sql.*; import org.apache.spark.api.java.*; import org.apache.spark.ml.linalg.*; import org.apache.spark.sql.types.*; import org.apache.spark.mllib.random.*; SparkConf conf = null; JavaSparkContext jsc = null; SparkSession spark = null; SparkContext sc = null; String appName = \"app\"; "

appName ends up null.

[clhedrick]

I don't know whether the JVM opts are still needed. They were in a previous version of Java and Spark.