On LinkedIn
Jobs
Lakeflow Jobs: tasks, parameters, triggers, repairs and running jobs in production.
16 postsHow SNOWGLOBE helps with this: Production Databricks pipelines
Run a subset of job tasks with their dependencies
You can run a subset of tasks and their dependencies in Databricks Lakeflow jobs.
Databricks Jobs Disable Task
In Databricks Lakeflow jobs you can disable tasks.
Add existing Job to Bundle
Are you tired of copying and pasting the yml definition of your jobs to Databricks Declarative Automation Bundles?
Notebook View Job Run
Did you know you can quickly check the results of your last successful scheduled job run directly in the Databricks notebook UI?
Databricks Job Run Duration Threshold
Don‘t forget to configure Job Run Duration Thresholds on your Databricks jobs over the holidays.
Table Update Triggers
Table Update Triggers in Databricks are a nice way to orchestrate data loads and modularize your data pipelines.
Databricks job git commit
When running your Databricks Jobs, you may want to know on which git commit the job run happened.
Job Startup Overhead Reduced
Many Databricks users may have missed this silent update: job startup overhead has been significantly reduced.
Get Arguments from Notebook of Job-Run (notebook.exit())
Here is how to retrieve the value set by dbutils.notebook.exit(value) from a Job Run in Databricks: The feature to set a value on the notebook exit is quite well documented (see first comment for the link to the docs).
New Job Run from recent Job Run Parameters
I discovered this quality of life feature in Databricks by accident: If you want to run one job multiple times with the same or similar parameters.
Run subset of tasks from Job
You can now run a subset of tasks from a Databricks Job: * Open the Task overview of your Job * Click the task you want to run * Click the play button * Select "Run 1 Task" Alternatively, click the caret in the upper-right …
Run Jobs as Service Principal
Always run your production Databricks Workflows in the name of a Service Principal.
Databricks Job Parameter Dynamic Value
In Databricks, I don't use
spark.conf.get("spark.databricks.workspaceUrl")anymore ...
Let's talk about your lakehouse.
Thirty minutes, no slides. Bring the question that keeps coming back.