The whole project is SBT and Scala based.

Dependencies

The tools are known to work with:

  • Java 8
  • Scala 2.11.12 or 2.12.12
  • Spark 2.4.2 to 3.1.X

Test

Depending on your versions of scala or spark use one of:

sbt ++2.11.12 -DSPARK_VERSION=2.4.7 test sbt ++2.12.12 -DSPARK_VERSION=2.4.7 test sbt ++2.12.12 -DSPARK_VERSION=3.1.1 test

Build

Build an executable JAR

For so called Fat JAR, depending on your versions of scala or spark, use one of:

sbt ++2.11.12 -DSPARK_VERSION=2.4.7 assembly sbt ++2.12.12 -DSPARK_VERSION=2.4.7 assembly sbt ++2.12.12 -DSPARK_VERSION=3.1.1 assembly

Build a library JAR

Depending on your versions of scala or spark use one of:

sbt ++2.11.12 -DSPARK_VERSION=2.4.7 package sbt ++2.12.12 -DSPARK_VERSION=2.4.7 package sbt ++2.12.12 -DSPARK_VERSION=3.1.1 package