On LinkedIn
Quick tips from daily Databricks work.
Everything Martin has posted on LinkedIn, in full. Browse by topic or by year.
2026
Prints to Std-Out
Every print statement in your Databricks code goes straight to stdout in the cluster logs.
Migrating Secrets to Unity Catalog? Watch the Names
Migrating secrets to Databricks Unity Catalog?
The Lakehouse Path - Unity Catalog Secret Migration
Databricks Unity Catalog Secrets are GA.
The _object_metadata Column
The
_object_metadatacolumn is in public preview (requires Databricks Runtime 18.2 or above).Hamburg Databricks User Group Meeting 2026/09/17 - recap
Full house at the Databricks User Group Meeting in Hamburg hosted by Accenture.
Disable full refresh for Lakeflow Pipelines tables
I accidentally nuked a Databricks Lakeflow Pipelines table that should never be deleted.
Hamburg Databricks User Group Meetup 2026/09/17
Data Mesh, Genie and Pizza come together at the Databricks Hamburg Users Group on 17th September.
Databricks Vibe Data Modeling
I wish I had this ten years ago while working in grocery retail.
Databricks Lakebase new Azure regions
Lakebase is now available in Azure region Germany West Central and three other regions.
- Lakebase
- News
Databricks serverless compute access control
This is the one feature I've been waiting for ever since interactive serverless was introduced in Databricks.
SQL Dynamic Parameters
In Databricks SQL, you can use dynamic parameters with values derived from a query.
DAB - Retrieve an object's ID value
Are you configuring your Cluster IDs in Databricks Declarative Automation Bundles still per environment?
Declarative Automation Bundles - Deploy from Workspace
Declarative Automation Bundles can be deployed from within the Workspace.
The Lakehouse Path - Unity AI Gateway: Budgets
Every platform team goes through the same sequence with AI endpoints: first the excitement, then the adoption, then the invoice.
Databricks DAB Direct Engine as Default
Databricks Declarative Automation Bundles use Terraform under the hood, but ...
Run a subset of job tasks with their dependencies
You can run a subset of tasks and their dependencies in Databricks Lakeflow jobs.
Lakeflow Designer - Generally Available
Databricks Lakeflow Designer is generally available now.
Databricks DAIS - Unity Catalog Updates
Although the focus of the Databricks Data + AI Summit was largely on Agents and AI, Unity Catalog got some updates along the way as well.
Databricks DAIS 2026 - DABs with Hubert Dudek
Last man standing at the last round of sessions at Databricks Data + AI Summit.
Databricks DAIS 2026 - Reyden engine
Databricks introduced Lakehouse//RT, powered by the new Reyden engine.
Databricks DAIS 2026 - Moscone Center
The Moscone Center in San Francisco is all dressed up for the Databricks Data + AI summit.
Databricks DAIS 2026 - Announcement 2
In less than a week, I will be in San Francisco at Databricks Data+AI summit.
Building a LinkedIn Analytics Pipeline on Databricks - Part 3
The third part of my blog series on how to build a Databricks Pipeline for LinkedIn Analytics is out now.
Generated Columns
Generated Columns in Databricks is a great feature to automatically create surrogate keys and populate columns using generation expressions.
Databricks Jobs Disable Task
In Databricks Lakeflow jobs you can disable tasks.
DAB Target Variables
In Databricks Declarative Automation Bundles, you can define per-target variable overrides.
Building a LinkedIn Analytics Pipeline on Databricks - Part 2
Getting data from the LinkedIn API can be tricky.
Databricks DAIS 2026 - Announcement 1
Databricks Data+AI Summit takes place in San Francisco from June 15–18.
Databricks DAB reference dynamic schema name in Development Mode
After my recent post about Databricks Declarative Automation Bundles' development mode one question kept coming up: how do you actually reference the user-named schema it creates?
Declarative Automation Bundles Development Mode
Deploying Databricks resources to a shared workspace with your team?
The Lakehouse Path: Building a LinkedIn Analytics Pipeline on Databricks - Part 1
I got tired of checking LinkedIn analytics in the UI.
Add existing Job to Bundle
Are you tired of copying and pasting the yml definition of your jobs to Databricks Declarative Automation Bundles?
Databricks Asset Bundles are now Declarative Automation Bundles
Databricks Asset Bundles are now Declarative Automation Bundles.
Databricks Asset Bundle Create in Workspace
Kickstart your journey with Databricks Asset Bundles straight from the Workspace.
The Lakehouse Path: SQL Warehouse Query Caching
Most data engineers know about Databricks query caching, but few understand how it actually works under the hood.
Databricks Serverless Workspace GA
I just tried out Databricks Serverless Workspaces in Azure and got a fully functional environment running in less than 3 minutes.
Databricks SQL Lead Lag Function
The lead and lag window functions in Databricks SQL are strong when analyzing sequences.
Notebook View Job Run
Did you know you can quickly check the results of your last successful scheduled job run directly in the Databricks notebook UI?
Workspace Explorer File Tree
I am a big fan of coding in the Databricks UI, but this one thing is missing.
Create a DataFrame
Creating a DataFrame in Pandas and PySpark may seem almost identical at first glance, but there's a key difference.
2025
Databricks Job Run Duration Threshold
Don‘t forget to configure Job Run Duration Thresholds on your Databricks jobs over the holidays.
Wishlist: Unity Catalog Browse Permission for Schema
So this one goes on my wishlist for Databricks Unity Catalog: I am missing one feature in the recently introduced BROWSE permission in Databricks Unity Catalog.
Table Update Triggers
Table Update Triggers in Databricks are a nice way to orchestrate data loads and modularize your data pipelines.
Explore Table using LLM in UC
Databricks is working hard to make data more accessible to business users.
Databricks job git commit
When running your Databricks Jobs, you may want to know on which git commit the job run happened.
DAIS World Tour Munich Announcement
I’ll be in Munich on November 3-4 for the Databricks Data & AI Summit World Tour!
Databricks spelling rant
A little reminder to everyone out there in the data ecosystem: It is simply Databricks.
Serverless SQL Warehouse Query Cache
I played around a little with the Databricks Serverless SQL Warehouse and this is what I found out about caching: - The same query once submitted as upper case and repeated as lower case comes from cache.
The Lakehouse Path: Databricks Logger Part 2
Last Thursday I published the second part of my three part series on logging in Databricks in my blog The Lakehouse Path.
Spark Session not needed
Did you know that you can easily spot whether someone is new to Databricks or has copied and pasted code from an LLM when they create a Spark Session in a Notebook?
Databricks One
When I logged into Databricks this morning, Databricks One showed up in my Azure based Workspaces in regions Germany West Central and Western Europe.
Data Sculpting by Eevamaija Virtanen
If you are a Data Engineer, you will be able to relate and enjoy what Eevamaija Virtanen calls Data Sculpting.
The Pedantic Layer by Joe Reis
This is a great blog post by Joe Reis about the endless debates over definitions.
Lakebase in Public Preview
Finally, Lakebase showed up in my Azure Databricks Workspace in Azure Western Europe region.
- Lakebase
The Lakehouse Path: Databricks Logger Part 1
Over the past few weeks, I’ve been exploring Substack as a platform for publishing long-form technical content.
Databricks Release Notes
Back from my summer break and found 44 new entries in the Databricks Release notes!
Humanizing Data Strategy
I read "Humanizing Data Strategy" by Tiankai Feng over the weekend and wanted to share a few thoughts.
Databricks Apps available in GWC by 31 Jul
Databricks Apps will be available in Azure Germany West Central on July 31st.
Databricks Runtime 17 Pandas Version
Databricks Runtime 17 is set to make some Pandas users happy.
UC Data Certifications
I discovered this feature when I was playing around with the new Databricks Tag Policies.
Job Startup Overhead Reduced
Many Databricks users may have missed this silent update: job startup overhead has been significantly reduced.
Multi-Cursor
Here is a feature in Databricks Code Editor you would not expect in a Web UI: More often than not, you end up editing something in multiple places in your code.
Databricks Free Edition: First Impressions
I checked out the Databricks Free Edition and here are my first impressions: During Data & AI Summit last week in San Francisco, Databricks announced a Free Edition that has almost all the features enabled as the paid versions.
Get Arguments from Notebook of Job-Run (notebook.exit())
Here is how to retrieve the value set by dbutils.notebook.exit(value) from a Job Run in Databricks: The feature to set a value on the notebook exit is quite well documented (see first comment for the link to the docs).
New Job Run from recent Job Run Parameters
I discovered this quality of life feature in Databricks by accident: If you want to run one job multiple times with the same or similar parameters.
Run subset of tasks from Job
You can now run a subset of tasks from a Databricks Job: * Open the Task overview of your Job * Click the task you want to run * Click the play button * Select "Run 1 Task" Alternatively, click the caret in the upper-right …
Preinstalled Python Libraries
Before you install Python packages on your Databricks Cluster, do this: Did you know that every Databricks Runtime comes with a suite of preinstalled Python libraries?
Run Jobs as Service Principal
Always run your production Databricks Workflows in the name of a Service Principal.
Never Use Pandas in Databricks for Data Transformation
Here’s a bold statement: I never use Pandas for data transformation in Databricks.
Luciano Ramalho: Fluent Python
After years of working with Python, I recently gained a deeper understanding of its object model, thanks to _Fluent Python_ by Luciano Ramalho.
Using IDENTIFIER clause in Databricks SQL
A dynamic way to use catalog, schema, and table names in Databricks SQL scripts.
Databricks Notebook Improvements
It’s easier than ever to interact with notebook outputs in Databricks.
Databricks Delta Live Tables supports multiple schema/catalog targets
One of the biggest obstacles in Databricks Delta Live Tables has now been solved.
Serverless SQL Error querying tables
Encountered an error while trying to read a table from Unity Catalog using Databricks Serverless SQL?
Use Cases are worthless without a high-quality data pipeline that fuels it
Here’s a hard truth: Your data use case is only as good as the data pipeline behind it.
Databricks Job Parameter Dynamic Value
In Databricks, I don't use
spark.conf.get("spark.databricks.workspaceUrl")anymore ...Databricks Serverless Compute Startup
Save 4–5 workdays of waiting in 2025 with Databricks Serverless Compute Here’s how I calculated it: Waiting for a Databricks cluster to start up is frustrating—especially when you just want to test something or run a small piece of code against your data.
Let's talk about your lakehouse.
Thirty minutes, no slides. Bring the question that keeps coming back.