Many Databricks users may have missed this silent update: job startup overhead has been significantly reduced.
A few weeks ago, I had a conversation with Anthony Podgoršak about potential improvements to Databricks File Arrival Triggers. After our discussion, he asked me a thought-provoking question:
“If you had one wish, what would you improve in Databricks?”
My response was clear: “Reduce job startup overhead. Currently, it takes about 10 to 20 seconds before the actual code begins to execute.”*
Anthony’s reply was encouraging: “We’re on it. Expect a reduced overhead of up to 70-90% in the coming weeks.”
And just like that, job startup time for serverless or an already running cluster is now almost instantaneous. I compared several Job Runs from March with the same Job that ran in June and I noticed a significant reduction in Job Startup Overhead. This is really great for small jobs that are running on a live cluster with users waiting for the results of this job.
Great Job, Anthony and Team!
* I have a lot more wishes regarding Databricks, but since we were talking about Jobs, this was top of my mind :-)
