Setting up Scala for Spark Development

Oct 5, 2020 | 2 min | #apache-spark #scala

This post is merely a reference-like article to set up your Scala environment for Apache Spark development.

Simplest Thing

Your build.sbt should looks like:

1name := "My Project"
2
3version := "0.1"
4
5scalaVersion := "2.11.12"
6
7libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.6"

Your Entry.scala:

 1import org.apache.log4j.{Level, LogManager}
 2
 3object Entry {
 4  def main(args: Array[String]) {
 5
 6    val spark = SparkSession
 7      .builder()
 8      .master("local")
 9      .getOrCreate()
10    LogManager.getRootLogger.setLevel(Level.ERROR)
11      
12    // use spark variable here to write your programs
13
14  }
15}

Integrating with Azure Pipelines

Azure Pipelines has built-in support for sbt, therefore you can build and package with the following task (simplest version):

 1- task: CmdLine@2
 2  displayName: "sbt"
 3  inputs:
 4    script: |
 5      sbt clean
 6      
 7      sbt update
 8          
 9      sbt compile
10          
11      sbt package
12    workingDirectory: 'project-dir'

To pass version number, you can use a variable from your pipeline. Say it’s called projectVersion, then pipeline task is:

 1- task: CmdLine@2
 2  displayName: "sbt"
 3  inputs:
 4    script: |
 5      sbt clean
 6      
 7      sbt update
 8          
 9      sbt compile
10          
11      sbt package
12    workingDirectory: 'project-dir'
13  env:
14    v: $(projectVersion)

which merely creates an environment variable called v for the sbt task. To pick it up, just modify version line for build.sbt:

1version := None.orElse(sys.env.get("v")).orElse(Some("0.1")).get

You can create uber JAR, however they are relatively large (70kb grows into over 100Mb) therefore I’d try to avoid it.

Have feedback or questions? Feel free to email me.