Skip to content

[SPARK-23078] add tests for Spark Thrift Server on Kubernetes #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[SPARK-23078] add tests for Spark Thrift Server on Kubernetes #38

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0e7a4ed
add test for spark thrift server on kubernetes
Jan 17, 2018
b43d137
fixed some dependency issues
ozzieba Jan 18, 2018
66b1771
dynamic port
Jan 18, 2018
befc2cc
eventually
Jan 18, 2018
0236305
fix dependencies, again
ozzieba Jan 18, 2018
95b6b55
build spark with hive and thrift
ozzieba Jan 18, 2018
528b064
merge master
ozzieba Jan 19, 2018
72d79e4
removed duplicate dep
ozzieba Jan 19, 2018
0b5f09e
changed remaining PP_LOCATOR_LABEL
ozzieba Jan 19, 2018
4c8fe2f
addressed comments
ozzieba Jan 19, 2018
36dee22
spacing
ozzieba Jan 19, 2018
cef7f5e
resolved conflicts
ozzieba Jan 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dev/dev-run-integration-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ then
fi
cd $SPARK_REPO_LOCAL_DIR
git checkout -B $BRANCH origin/$BRANCH
./dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes -DskipTests;
./dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes -Phive -Phive-thriftserver -DskipTests;
SPARK_TGZ=$(find $SPARK_REPO_LOCAL_DIR -name spark-*.tgz)
echo "Built Spark TGZ at $SPARK_TGZ".
cd -
Expand Down
13 changes: 13 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
<exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
<extraScalaTestArgs></extraScalaTestArgs>
<guava.version>18.0</guava.version>
<hadoop.version>2.6.5</hadoop.version>
<hive.group>org.spark-project.hive</hive.group>
<hive.version>1.2.1.spark2</hive.version>
<jsr305.version>1.3.9</jsr305.version>
<kubernetes-client.version>3.0.0</kubernetes-client.version>
<log4j.version>1.2.17</log4j.version>
Expand Down Expand Up @@ -84,11 +87,21 @@
<artifactId>log4j</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to depend on hadoop-common?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without it I get java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration. In general, I believe Hive JDBC requires some Hadoop dependencies, as discussed in eg https://issues.apache.org/jira/browse/HIVE-15110. Not sure if it's only configuration. Do you have a cleaner solution?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, thanks for the info.

<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${commons-lang3.version}</version>
</dependency>
<dependency>
<groupId>${hive.group}</groupId>
<artifactId>hive-jdbc</artifactId>
<version>${hive.version}</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
Expand Down
46 changes: 46 additions & 0 deletions src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ import java.io.File
import java.nio.file.{Path, Paths}
import java.util.UUID
import java.util.regex.Pattern
import java.sql.DriverManager

import scala.collection.JavaConverters._
import com.google.common.io.PatternFilenameFilter
import io.fabric8.kubernetes.api.model.{Container, Pod}
import org.apache.hive.jdbc.HiveDriver
import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll, FunSuite}
import org.scalatest.concurrent.{Eventually, PatienceConfiguration}
import org.scalatest.time.{Minutes, Seconds, Span}
Expand Down Expand Up @@ -121,6 +123,10 @@ private[spark] class KubernetesSuite extends FunSuite with BeforeAndAfterAll wit
runSparkPiAndVerifyCompletion(appArgs = Array("5"))
}

test("Run Spark Thrift Server") {
runThriftServerAndVerifyQuery()
}

test("Run SparkPi with custom labels, annotations, and environment variables.") {
sparkAppConf
.set("spark.kubernetes.driver.label.label1", "label1-value")
Expand Down Expand Up @@ -239,6 +245,46 @@ private[spark] class KubernetesSuite extends FunSuite with BeforeAndAfterAll wit
appLocator)
}

private def runThriftServerAndVerifyQuery(
driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should be 4 spaces indention.

appArgs: Array[String] = Array.empty[String],
appLocator: String = appLocator): Unit = {
val appArguments = SparkAppArguments(
mainAppResource = "",
mainClass = "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2",
appArgs = appArgs)
SparkAppLauncher.launch(appArguments, sparkAppConf, TIMEOUT.value.toSeconds.toInt, sparkHomeDir)
val driverPod = kubernetesTestComponents.kubernetesClient
.pods
.withLabel("spark-app-locator", appLocator)
.withLabel("spark-role", "driver")
.list()
.getItems
.get(0)
driverPodChecker(driverPod)
val driverPodResource = kubernetesTestComponents.kubernetesClient
.pods
.withName(driverPod.getMetadata.getName)

Eventually.eventually(TIMEOUT, INTERVAL) {
val localPort = driverPodResource.portForward(10000).getLocalPort
val jdbcUri = s"jdbc:hive2://localhost:$localPort/"
val connection = DriverManager.getConnection(jdbcUri, "user", "pass")
val statement = connection.createStatement()
try {
val resultSet = statement.executeQuery("select 42")
resultSet.next()
assert(resultSet.getInt(1) == 42)
} finally {
try {
statement.close()
} finally {
connection.close()
}
}
}
}

private def runSparkApplicationAndVerifyCompletion(
appResource: String,
mainClass: String,
Expand Down