You can run a subset of tasks and their dependencies in Databricks Lakeflow jobs. Here is how:
- in the Databricks UI, go to the job and select Run with different parameters
- click Deselect all
- click on the task you want to run
- select dependent tasks upstream and/or downstream of this task
- click on Run tasks
This is very helpful if you have a large job and want to run only a specific part of the job end-to-end.
This is also possible through the API/Databricks SDK. Comment if I should post about that, too!
