Failed assertions in Scala 3 produce unhelpful stack traces (500USD Bounty)

[malcolmredheron]

---

From the maintainer Li Haoyi: I'm putting a 500USD bounty on this issue, payable by bank transfer on a merged PR fixing this.

See https://github.com/orgs/com-lihaoyi/discussions/6 for other bounties and the terms and conditions that bounties operate under

---

I'm confused by the stack traces that I get when a utest assertion fails in scala 3. Specifically, the only line in the trace that refers to the test file seems to point to the line with Tests { on it, not the line of the assertion

I forked the utest repo and changed an assertion to fail. Here's how to reproduce it:

• Clone https://github.com/malcolmredheron/utest-confusing-assertion/tree/broken-assertion
• Run ./mill utest.jvm[3.3.3].test.testOnly "*HelloTests*"
• For me, this produces:

[113/113] utest.jvm[3.3.3].test.testOnly
[113] Setting up CustomFramework
[113] -------------------------------- Running Tests --------------------------------
[113] + test.utest.examples.HelloTests.test1 6ms  
[113] + test.utest.examples.HelloTests.test2.inner 0ms  1
[113] + test.utest.examples.HelloTests.test3 0ms  
[113] X test.utest.examples.HelloTests.test4 3ms 
[113]   utest.AssertionError: 1 == 2
[113]     utest.asserts.Asserts$.assertImpl(Asserts.scala:30)
[113]     test.utest.examples.HelloTests$.$init$$$anonfun$1$$anonfun$4(HelloTests.scala:5)
[113] Tearing down CustomFramework
[113] Tests: 4, Passed: 3, Failed: 1
[113/113] =================================================== utest.jvm[3.3.3].test.testOnly *HelloTests* ====================================================== 1s
1 tasks failed
utest.jvm[3.3.3].test.testOnly 1 tests failed: 
  test.utest.examples.HelloTests test.utest.examples.HelloTests.test4

• By contrast, ./mill utest.jvm[2.13.14].test.testOnly "*HelloTests*" mentions line 21 instead of line 5

[strobe]

so far my current assumption that in Scala 3 macro implementation of AssertsCompanionVersionSpecific.interceptProxy some expression transformation happen which leads to throwing exception in incorrect place at runtime. At least with few experiments that I did I've not noticed any issue with code that responsible for getting stack trace.

[mrdziuban]

I don't have a fix yet but what I've found, in case it helps others, is that the logic in Tracer seems to be at least partially responsible for changing the position of an assert.

If I add a println before line 22 to log the position of exprs:

println(s"exprs POS: ${exprs.asTerm.pos.sourceFile}:${exprs.asTerm.pos.startLine}")

Then I see:

exprs POS: /Users/matt/utest/utest/test/src/test/utest/examples/HelloTests.scala:20

Note that lines are zero-based, so line 20 is actually line 21 in @malcolmredheron's code -- this is the correct line where the error should be reported.

If I then add a println after line 24 to log the position of trees:

println(s"trees POS: ${trees.asTerm.pos.sourceFile}:${trees.asTerm.pos.startLine}")

Then I see:

trees POS: /Users/matt/utest/utest/test/src/test/utest/examples/HelloTests.scala:4

Line 4 is the line with Tests { where the error is actually reported.

I've tried changing the owner of trees and a handful of other trees constructed in these macros but haven't had any luck with a fix yet.

[strobe]

it seems there on line 24 line position information in ess are correct but if we try to print it with Expr.ofSeq(ess).asTerm.pos.startLine it will get incorrect lines.
Probably, it means that Expr.ofSeq creating new expression and getting positional information from original expression parent because it can't use original information for newly created expression.

[strobe]

I think I leaped to conclusions prematurely about Expr.ofSeq most likely that issue happen but somewhere deeper in the code, could be at

utest/utest/src-3/utest/asserts/Tracer.scala

Line 101 in 76e7fe5

tracingMap(logger).transformTerm(expr.asTerm)(Symbol.spliceOwner).asExprOf[T]

But overall I don't know yet where exactly.

Challenge seems not only how to locate it but what approach should be used to preserve positional information.