Efficiently ingesting thousands of JSON files into a Delta table

In the past, I've worked with a system that produced hundreds of thousands of compressed JSON files every day, each weighting around 1MiB uncompressed. We wanted to ingest all that data into datalake for debug and analytics, and for that we've used Spark to append to a delta table. Conceptually, the flow is very straight forward: a job runs every day it lists the new files that need to be ingested…