Table Update Triggers in Databricks are a nice way to orchestrate data loads and modularize your data pipelines.
This is how it works:
- When any or all the tables defined in the table update trigger are updated, the job is started.
- Like in file arrival triggers, minimum times between triggers can be defined as well as a time to wait after the last change. This allows to wait for other updates that come into this or other tables.
- The best thing is, that new dynamic job parameters can be used. A list of updated tables, their most recent delta version and the update timestamp can be passed to the job that is triggered. This allows granular control of what happens in the job.
With table update triggers (and file arrival triggers respectively), there are plenty of opportunities to create lightweight orchestration for your data pipelines within Databricks.
Gone are the days when you had to trigger a job from a task via subsequent task/API call or handle parameters like table-lists on your own.
