Skip to content

Commit

Permalink
[SPARK-43173][CONNECT][TESTS] Ignore write jdbc when test `ClientE2…
Browse files Browse the repository at this point in the history
…ETestSuite` without `-Phive`

### What changes were proposed in this pull request?
`write jdbc` will test failed when test `ClientE2ETestSuite` using Java 11 & 17 without `-Phive` because the test case need `derby` in the classpath of `SimpleSparkConnectService` and `derby.jar` will only be copied to `assembly/target/scala-xxx/jars/` when build with `-Phive`.

So this pr add check to ingore `write jdbc` when test without `-Phive`.

### Why are the changes needed?
Ingore `write jdbc` when test without `-Phive`.

### Does this PR introduce _any_ user-facing change?
No, just for test.

### How was this patch tested?
- Pass GitHub Actions
- Manual test `ClientE2ETestSuite` using Java 11 & 17 without `-Phive`

```
build/mvn clean install -Dtest=none -DwildcardSuites=org.apache.spark.sql.ClientE2ETestSuite
build/sbt "connect-client-jvm/testOnly *ClientE2ETestSuite"
```

Before

```
- write jdbc *** FAILED ***
  io.grpc.StatusRuntimeException: INTERNAL: No suitable driver
  at io.grpc.Status.asRuntimeException(Status.java:535)
  at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
  at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
  at scala.collection.Iterator.foreach(Iterator.scala:943)
  at scala.collection.Iterator.foreach$(Iterator.scala:943)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  at org.apache.spark.sql.SparkSession.execute(SparkSession.scala:458)
  at org.apache.spark.sql.DataFrameWriter.executeWriteOperation(DataFrameWriter.scala:257)
  at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:221)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:218)

```

After

```
[info] - write jdbc !!! CANCELED !!! (0 milliseconds)
[info]   org.apache.spark.sql.connect.client.util.IntegrationTestUtils.isSparkHiveJarAvailable was false (ClientE2ETestSuite.scala:196)
[info]   org.scalatest.exceptions.TestCanceledException:
[info]   at org.scalatest.Assertions.newTestCanceledException(Assertions.scala:475)
[info]   at org.scalatest.Assertions.newTestCanceledException$(Assertions.scala:474)
[info]   at org.scalatest.Assertions$.newTestCanceledException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssume(Assertions.scala:1310)
[info]   at org.apache.spark.sql.ClientE2ETestSuite.$anonfun$new$16(ClientE2ETestSuite.scala:196)
```

Closes apache#40837 from LuciferYang/SPARK-43173.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
  • Loading branch information
LuciferYang authored and HyukjinKwon committed Apr 19, 2023
1 parent cb5805c commit 1c6202c
Showing 1 changed file with 1 addition and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,7 @@ class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper {
}

test("write jdbc") {
assume(IntegrationTestUtils.isSparkHiveJarAvailable)
if (SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9)) {
val url = "jdbc:derby:memory:1234"
val table = "t1"
Expand Down

0 comments on commit 1c6202c

Please sign in to comment.