Announcing dbt Temporal worker
Running dbt on my favorite runtime -- as a Proof of Concept -- for now
If you’re not familiar with dbt or Temporal, I guess this post is mostly a curiosity about how powerful Claude can be when asked to integrate two code bases. Because that’s what this is — glueing together analytics’ engineers favorite SQL orchestrator with everyone’s new favorite durable workflow executor.
What does all that even mean?
Well, dbt (yes all lowercase), the data build tool, revolutionized how analysts and data scientists — an then data engineers and analytics engineers — create SQL based data pipelines.
In short it marries Jinja2 templates with SQL files letting you define inter-dependencies between the files and have a framework execute them in the correct order. Funny enough, a pattern I introduced at Spotify already in the Hadoop/Hive-days back in 2014.
Temporal is a frontrunner in durable execution — meaning you can specify any script as a set of steps — activities, and Temporal ensures that when the script runs, it really finishes.
Naïvely, think of it as a simple matter of storing the input/ouptut of the activities in a database. That means Temporal can recover a crashed script, roll back execution to any previous state, spawn a replica of a script execution or keep a script dormant for months on end and resume it from where it was paused.
Where I work, at Gilion, we use Temporal with Pydantic AI to run our agentic platform. It’s insanely powerful for long-running tasks like that, especially as they are a bit flakey. If the agent fails on the last of 50 tool-calls, it can try again from that tool call — even if the whole process panicked and computer crashed. If the worker’s kubernetes pod is re-deployed or flushed mid-workflow, the end users barely notices as the activity seamlessly picks up from another worker.
The Marriage
In https://github.com/thejens/dbt-temporal I decided to hook these two up. Funnily enough, dbt recently open sourced their new blazing fast (lol) Rust-based backend dbt-fusion and I really wanted to leverage Claude Code and tinker with it (it’s not done/stable yet btw)
Whilst I was tinkering, temporal launched the first alpha version of their Rust worker SDK — and a few days later, I officiated the marriage.
For now as a proof of concept, but I will likely try fork-lifting some actual production workloads onto this as it matures. One clear benefit is the hooks, allowing me to run any custom code pre/post the SQL pipeline, including triggering another dbt model afterwards. We also get an always-on worker, ready to start executing dbt jobs, meaning less overhead spinning up worker pods etc.
If you’re interested in running this yourself, you should! But given I had Claude write most of the code — perhaps use the repo more as a blueprint and re-implement it yourself with your favorite coding assistant.
That’s all for now, more info in the repo itself
/Jens