Access DBUtils from Scala app running under databricks-connect

Apr 8, 2022 | 1 min | #databricks #apache-spark

After a lot of trial and error this is what worked for me.

SBT manifest:

 1name := "test-dbutils"
 2
 3version := "0.1.0-SNAPSHOT"
 4
 5scalaVersion := "2.12.15"
 6val sparkVersion = "3.1.3"
 7
 8libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion
 9
10// https://mvnrepository.com/artifact/com.databricks/dbutils-api
11libraryDependencies += "com.databricks" %% "dbutils-api" % "0.0.5"

(note the com.databricks... dependency!)

Then in Scala code this works:

1val dbutils = com.databricks.service.DBUtils
2val files = dbutils.fs.ls("/FileStore")
3println(files)

Have feedback or questions? Feel free to email me.