jscripts/java/PEWO_java/dist/PEWO.jar raises java.io.InvalidClassException

[balabanmetin]

Hi,

The following job fails:

java -cp scripts/java/PEWO_java/dist/PEWO.jar DistanceGenerator_LITE2 /home/balaban/PEWO/examples/1_fast_test_of_accuracy_procedure/run RAPPAS,EPANG,PPLACER &> /home/balaban/PEWO/examples/1_fast_test_of_accuracy_procedure/run/logs/compute_nd.log

with the following exception:

"~/PEWO/examples/1_fast_test_of_accuracy_procedure/run/logs/compute_nd.log" 34L, 2389C 2,1 Top
ARGS: workDir [list_of_tested_software_directories,comma-separated]
example: /path/to/pewo_workdir EPANG,RAPPAS,PPLACER
scripts/java/PEWO_java/dist/PEWO.jar
workDir: /home/balaban/PEWO/examples/1_fast_test_of_accuracy_procedure/run
Loading /home/balaban/PEWO/examples/1_fast_test_of_accuracy_procedure/run/expected_placements.bin
Loading NxIndex
Loading pruningIndex
Loading expected placements
Loading trees
Jan 14, 2021 8:45:06 AM DistanceGenerator_LITE2 main
SEVERE: null
java.io.InvalidClassException: javax.swing.JComponent; local class incompatible: stream classdesc serialVersionUID = 3742318830738515599, local class serialVersionUID = 4588530037560142483
at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1903)
at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1903)
at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1903)
at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1772)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at java.base/java.util.ArrayList.readObject(ArrayList.java:928)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1160)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2216)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2087)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1594)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at DistanceGenerator_LITE2.main(Unknown Source)

I tried reinstalling PEWO and rebuilding the jar file but it didn't resolve the issue.

[blinard-BIOINFO]

Hi,

the following line shows that the java version that you used to compile PEWO and on the machine where you run PEWO are different.
java.io.InvalidClassException: javax.swing.JComponent; local class incompatible: stream classdesc serialVersionUID = 3742318830738515599, local class serialVersionUID = 4588530037560142483

Basically, a serialVersionUID is a unique number assigned to each Class type in Java and it seems the runtime class javax.swing.JComponent has a different UID than the same class at compilation time.
This can happen in many scenario, please double check the following cases and try to figure out if you are not in one of them:

1. check version of java -version and javac -version to confirm that runner and compiler are the exact same version on your machine.
2. if you compiled on one machine and moved the file on another, make sure you have the exact same version on both. But more generally, this is not recommended, you should compile on the machine where you run PEWO.
3. make sure you are not mixing a open-jdk and oracle java installations or libraries. These are the open source and proprietary versions of java.
4. if you use MAC, I noticed this kind of issues is more common as MACOS is packed with its own java built (slightly different from the open-jdk and oracle java).

More generally, I invite you to find threads related to this error to find what is wrong in your java configuration.

https://stackoverflow.com/questions/10378855/java-io-invalidclassexception-local-class-incompatible

One thing that could have happened:
1: you create your serialized data with a given library A (version X)
2: you then try to read this data with the same library A (but version Y)

[balabanmetin]

What had happened was shell commands in snakemake workflows did not inherit PEWO conda environment. So any shell command such as "java" or "raxmlHPC-SSE3" in the workflow failed. The first one that failed was "java" because the it called the java command that is not in PEWO conda environment, which has a different version. I fixed this issue with an ad-hoc solution, I added source activate PEWO to the end of my .bashrc.

Do you know how to make sure shell calls in snakemake runs on PEWO conda environment ?

[blinard-BIOINFO]

Hi,
This is not the correct way to use conda.
Conda is similar to python virtualenv.
You create one or several environments from theirs definitions and load them ion the fly depending on your needs.

Do not set an activate command in your bashrc, this will just load a single conda environement at each bash startup !
In particular because snakemake may rely on more than one conda environment during execution.

Each time you want to use PEWO, you have to call source activate PEWO before using PEWO.
This load the conda environment containing all the software you mentioned in your previous message.
As written in the documentation:

Analysis configuration

1. Activate PEWO environment:
   conda activate PEWO
   By default, the latest version of every phylogenetic placement software is installed in PEWO environment. If you intend to evaluate anterior versions, you need to manually downgrade the corresponding package.

This is the basic idea behind virtual environments. You activate/deactivate them to your needs, BEFORE launching an particular pipeline or analysis that requires a particular set or libraries / commands.

If you are not familiar with python virtual environments in particular, I recommend this 20 minutes lecture.
It describes the most important points :
https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html

[balabanmetin]

I call snakemake after calling source activate PEWO. As I said earlier, shell commands in snakemake fail because for some reason they are not in PEWO environment, despite the fact that I call snakemake command from within a bash session where source activate PEWO is called. Any job from compute_ar_inputs onward fails with errors like "command not found" or "no RAPPAS.jar" etc. When I print $PATH right before I call snakemake, it includes the path /home/balaban/miniconda3/envs/PEWO/bin. When I print path inside one of the workflow files, for example, at line 44 in rules/alignment/hmmer.smk before hmmbuild command it does not include the path /home/balaban/miniconda3/envs/PEWO/bin unless I add source activate PEWO to .bashrc.

[blinard-BIOINFO]

In theory snakemake see the same context as those loaded via conda before its execution.
This is really weird... something must be wrong with your bash or conda installation, because this is absolutely not the normal behavior of conda.

Could you try from a fresh conda install ?
Do you have conflicting conda installation on your machine ?

To verify what goes wrong, you could try the following rapid test.
There we will explicitly force snakemake to load a specific conda environment in its rules.

First build a toy example of conda environment.

#create simple env with python 2.7
$ conda create -n python27 python=2.7
# build environement file
$ conda activate python27
$ conda env export -f python27.yaml
$ conda deactivate

Then build a very simple rule, paste the following in simple.smk :

rule basic:
    input:
        "fake_input"
    output:
        "fake_output"
    conda: 
        "python27.yaml"    #this is the environment definition built with previous commands
    shell:
        """
        echo $PATH
        which python          #should display --> /home/balaban/miniconda3/envs/envs/python27/bin/python
        python --version     #should display 2.7, even is PEWO environement and snakemake are using python 3. 
        touch {output}
        """
rule all:
     input:
        "fake_output"

And execute in your PEWO environment:

$ touch fake_input
$ conda activate PEWO

PEWO environment is using python 3, python --version should return python 3.
I get this:

$ python --version
Python 3.6.7

Now let's see from inside snakemake, targetting specifically the conda 2.7 environment.

$ snakemake --use-conda --snakefile simple.smk

I get this:

Building DAG of jobs...
Creating conda environment python27.yaml...
Downloading and installing remote packages.
Environment for python27.yaml created (location: .snakemake/conda/20a490f1)
Using shell: /bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	basic
	1

[Mon Jan 18 18:26:36 2021]
rule basic:
    input: fake_input
    output: fake_output
    jobid: 0

Activating conda environment: /home/ben/.snakemake/conda/20a490f1
/home/ben/.snakemake/conda/20a490f1/bin:/home/ben/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
/home/ben/.snakemake/conda/20a490f1/bin/python               <--  CORRECT conda environment loaded locally during rule execution
Python 2.7.15                <--  CORRECT conda environment loaded locally during rule execution
[Mon Jan 18 18:26:37 2021]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /home/ben/.snakemake/log/2021-01-18T182624.753873.snakemake.log

So my conda install is correct:

• python 2.7 in my ubuntu
• python 3 after activating PEWO environement
• a different python 2.7 when loading a specific environment definition on a per rule basis.

Do you manage to reproduce this toy example ?
If not, definitely, your conda installation or bash initialization step is broken and you should start with a fresh install.

[balabanmetin]

Thanks for the example I think we are close to get to the bottom of this.

$ conda activate PEWO && snakemake --use-conda --snakefile simple.smk command produces the same output you had:

Building DAG of jobs...
Creating conda environment python27.yaml...
Downloading and installing remote packages.
Environment for python27.yaml created (location: .snakemake/conda/623e35db)
Using shell: /bin/bash
Provided cores: 24
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	basic
	1

[Mon Jan 18 10:31:30 2021]
rule basic:
    input: fake_input
    output: fake_output
    jobid: 0

Activating conda environment: /oasis/scratch/comet/balaban/temp_project/pewotest/.snakemake/conda/623e35db
/home/balaban/bin:/home/balaban//global/src/shell/:/home/balaban//global/src/mirphyl/utils/:/oasis/scratch/comet/balaban/temp_project/pewotest/.snakemake/conda/623e35db/bin:/home/balaban/miniconda3/condabin:/home/balaban/bin:/home/balaban//global/src/shell/:/home/balaban//global/src/mirphyl/utils/:/opt/mvapich2/intel/ib/bin:/opt/intel/2018.1.163/compilers_and_libraries_2018.1.163/linux/bin/intel64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/sdsc/bin:/opt/sdsc/sbin:/opt/ibutils/bin:/opt/rocks/bin:/opt/rocks/sbin
/oasis/scratch/comet/balaban/temp_project/pewotest/.snakemake/conda/623e35db/bin/python
Python 2.7.15

$ conda activate PEWO && snakemake --snakefile simple.smk is outputs python version 3.7.1 :

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 24
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	basic
	1

[Mon Jan 18 10:34:56 2021]
rule basic:
    input: fake_input
    output: fake_output
    jobid: 0

/home/balaban/bin:/home/balaban//global/src/shell/:/home/balaban//global/src/mirphyl/utils/:/home/balaban/miniconda3/bin:/home/balaban/miniconda3/condabin:/home/balaban/bin:/home/balaban//global/src/shell/:/home/balaban//global/src/mirphyl/utils/:/opt/mvapich2/intel/ib/bin:/opt/intel/2018.1.163/compilers_and_libraries_2018.1.163/linux/bin/intel64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/sdsc/bin:/opt/sdsc/sbin:/opt/ibutils/bin:/opt/rocks/bin:/opt/rocks/sbin
/home/balaban/miniconda3/bin/python
Python 3.7.1
[Mon Jan 18 10:34:56 2021]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /oasis/scratch/comet/balaban/temp_project/pewotest/.snakemake/log/2021-01-18T103456.031422.snakemake.log

I am sure in your machine, you would see Python 3.6.7 and that's the normal behavior. I think you are on point that there is something wrong with my bash or conda.