How to run tests?

[shishkin]

I'm interested in contributing to Mill, but I have troubles setting up my environment. When I run tests ./mill -i scalajslib.test from the main branch I get lots of errors like those:

ArraySeq("hello", "world") had length 2 instead of expected length 80 (ArgsParserSpec.scala:15)

or

X mill.scalajslib.NodeJSConfigTests.test.test 2634ms
  java.nio.file.NoSuchFileException: /Users/serega/code/oss/mill/target/workspace/mill/scalajslib/NodeJSConfigTests/helloWorldEvaluator/buildUTest/2.12.6/List()/test/test.log

I use MacOS 12.1, OpenJDK 17.0.1 and Node 16.13.0. What am I doing wrong?

[lefou]

The test error isn't a "test failure of the test suite" but a "test failure in a test suite under test" which itself is successful. I agree the output might be a bit confusing, but clearly we need somehow to test also failure sitations. Mill reports failed tasks at the end and also via a non-zero return code. Both is most probably not the case for you.

[shishkin]

I see. In the end I've got this error and a non-zero exit code:

1 targets failed
scalajslib.test.test 2 tests failed:
  mill.scalajslib.NodeJSConfigTests mill.scalajslib.NodeJSConfigTests.test.test
  mill.scalajslib.NodeJSConfigTests mill.scalajslib.NodeJSConfigTests.test.test2G

[lefou]

Could this be related to your changes? Also, I've seen almost the identical set of test fail when I played with older Scala versions in PR #1713. Namely 2.12.5.

• Less hardcoded Scala versions in tests and some bumps #1713 (comment)

[shishkin]

It is of 13ed055 on clean upstream main.

[shishkin]

openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment Homebrew (build 17.0.1+1)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.1+1, mixed mode, sharing)

[lefou]

This is probably the issue. I never tried JDK 17 yet. We might want to add it to our test suite.

(I think with version 17 the module classloader isn't any longer optional or something like that and special switches need to be turned on to get old behavior.)

[shishkin]

Yes, indeed. Tests pass on JDK11. Although it's surprising for new users that latest LTS version is not supported.

[lefou]

Probably because Scala devs don't get as much from newer JDKs as Java devs do. Also my own background in using OSGi made me stay away from anything Jigsaw related (Java 9 module system) for a long time. I even don't know if that is our problem here, but I just wanted to say it doesn't surprise me. As long as Java 11 gets updates, most will not see an urge to use newer JDKs.

[shishkin]

I understand the reasons. Let me see if I can prepare some documentation PRs to clarify those gotchas.