It’s hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them? Most of us are overwhelmed and unsure about the many possibilities, yet we need to keep going and do our work. You might use AI agents all day long, parallelize them with AI Orchestrators, tmux, git worktree, and so on, using AI IDEs, but in the end, you still need to coordinate and understand what the agents produced, potentially test it, which makes it even harder to keep up.

Luckily, Maxime Beauchemin, the creator of Airflow and Superset and the person who defined what “data engineer” meant for a decade (more on him below), joins us to show how he uses agents and what he’s built for working with them. I tried to extract the patterns behind how he actually uses AI in his data work today. This is the fourth interview in ‘How to use AI with DE’.

In this article, we go into four parts: (1) How to balance quality with messy data warehouse work, and how to manage agents with Figma for agents. (2) We elaborate on the future of the context layer and the return to semantics, (3) how Okta for Agents is needed for security, and (4) how the future of agentic workloads can be done in teams, whose yap-to-ship ratio is best, and why Amdahl’s law still counts.

Introducing the Guest: #4 Maxime Beauchemin

Our guest in this interview Max Beauchemin, the creator of Airflow and Superset. He’s known as one of the OGs of defining how data engineering worked back in 2017, and founded Preset, the company behind Superset, and currently serves as its CEO.

He is heavily involved in the AI workflow, which is another reason I wanted to interview him for this series, but he has also been building in the space himself: Agor (Ag: AI agent + Or: orchestration), earlier tooling like claudette-cli¹, and db-agents, an experiment to embed agent context directly inside databases. We’ll get into it all.

Max and I talked about many things, among them how to use AI in data engineering, how security plays a role, how shared, secure, context-rich agent workspaces work within teams, and how he uses AI assistants to run his business and ease his life as a CEO.

Max is a true open-source enthusiast, and he wants open source to win. Everything we discuss here is somewhere on GitHub, which I have happily linked throughout the interview.

Figma for Agents: Visualizing Tasks and Jobs with Agor

Before we start using Figma for Agents, coordinating them on canvas, we need to ask why we need coordination and orchestration in the first place.

Balancing Quality with Quantity: Messy DWHs

That’s where we started, with the challenge of messy data warehouse environments that most people find themselves in. I asked how he balances quality and quantity, aiming for high quality.

Max says that the new models, Opus 4.5 or 4.6², are not making many errors anymore and are very clever when they get the right context as above with all the database schemas of tables and data types, and even querying it with MCP. He says they almost run in self-serve mode, but he still prefers that users know what they are doing and can either read the generated code or verify the generated numbers on a dashboard or chat results.

But the setup is critical. With these three prerequisites, the agents handle almost all queries really well:

1. You need some preparation claude.md/ agents.md
2. Access to SQL (e.g., execute dbt)
3. Access to MCP or CLI for BI tools (e.g., Superset supports sup!, a CLI to interact with Superset)

The only problem, and always has been, is the messy structure and sources that most organizations have, growing from an initial small project into a certain stage. There are always obscure tables or strings, timestamps not aligned, or hidden information that is not encoded in code or written down. Or there’s the hidden knowledge, like that a certain table shouldn’t be used anymore or has bad data, which is known to the people using it but might not be to agents.

The Canvas in Which Your Agents Can Run: Automate Most CEO-stuff

When he recently saw the power of agentic coding, Max went all in and has been building the Figma for agents ever since. Something he can use to collaborate with agents within his company, instead of everyone running the same prompts locally and needing to sync with each other manually. That’s when Agor was born.

Agor stands for Ag: agent and Or: for orchestration. As the creator of Airflow and CEO of a data company, he knows exactly how a tool needs to improve his workflow. He also called it:

The goal is to automate most of the automatable CEO-stuff

Building an Internal Knowledge Base: Shared Canvas

Agor was Max’s answer to “how he uses AI beyond a research tool”, but doing data modeling, writing data pipelines, even legal or HR roles he added later to Agor, so you can give company-wide roles to agents that can be fed with dedicated documents and context, and triggered by any employee internally. In contrast, others see the jobs and avoid asking the same questions, reusing the output for new queries—Andrej Karpathy’s concept of an LLM-maintained shared team wiki—which builds an internal knowledge base.

Initially, when we first chatted, Agor had already changed how he worked as a CEO, but since then, Agor has gone even further. Agor can replace high-level tasks while still being very hands-on by working closely with the code via git worktrees (branch cards in Agor’s UI) and verifying the code in the PRs it produces. Max also added OpenClaw-like features around memory and identity via dedicated Markdown files, such as AGENTS.md, SOUL.md, MEMORY.md, so that Agor’s agents can learn from recent runs and carry a purpose and clear instructions. This led to role-based agents called Assistants (use agor-assistant as a template to build your own).

Inspired by OpenClaw’s agent loop, Assistants became first-class citizens, persistent AI companions with memory, identity, skills, and scheduled tasks integrated into Agor’s canvas, multiplayer workflows, and reachable directly from Slack, for example. Additionally, Agor adds features beyond OpenClaw, such as better multi-user support, RBAC, one-click, full session inspection, and many more.

Asked about the goal of Agor, Max said:

The initial premise was to remove DevOps and set up time for other members of the company. Instead of people needing to connect all the MCPs or CLIs to add API keys, set permissions, or integrate with Slack, the prompt window with the needed context is there and ready to start.

[!note] OpenClaw, what is it? And how does Agor compare?
OpenClaw (formerly ClawdBot) is an open-source agent framework built around a persistent agent loop: a serialized cycle that turns a message into actions, using file-based identity (SOUL.md) and layered memory (MEMORY.md). Agor’s Assistants adopt this pattern and extend it with multiplayer boards, RBAC, and canvas-level orchestration.

Visual and Spatial Memory

When you do a lot of agent work, it’s really hard to keep up with all of it. That’s where Agor’s visual and spatial overview really helps and is unique in its approach.

It brings the local and private session to a server, where everybody can see and work together on the same queries, and use the insights from other results, as dashboards are built for. So instead of keeping output locally, others can source the artifacts generated by agents, stored as Artifacts within Agor, ready to use by anyone, with no integration or deployment needed.

Or how Agor tracks its own spending across sessions:

[!note] Check the full Webinar about the Anatomy of the Internal Data Agent at Preset.

Such as an assistant needing access to all pipelines’ metadata as illustrated data stack.

Or the data needs an analytics agent to support, such as self-serve, the data team, and extras such as memory, skills, documentation, etc:

Context Layer: Back to Semantic Layers?

Max also believes that we are going back to the Semantic layer, or using it for AI as agents benefit from structured information - helping with the data model and SQL part, to make sure it’s correct, especially with the needs of AI agents and the persistent challenge of providing trustworthy self-service analytics.

With the shift of semantics outside of the BI tool, versioned, testable, portable, it’s a chance for better integration between business domain experts and data engineers. His thinking has evolved since he wrote the article, and Max told me:

I see two different semantics: the semantic layer and the YAML. There are the hard constraints — not every area needs that strictness — and then the softer ones with Markdown and Agentic Skills, good for 80-90% but with no guarantees.

AGENTS.md For Databases: Markdown Stored Inside the Database Itself

Based on that idea, Max created an experiment to bring the AGENTS.md convention into the database. DB-AGENTS reserves a dedicated schema and table, _agents._agents, that holds agent-oriented documentation at different scopes (global, domain, schema, table, and even column). You write the docs locally as markdown files with YAML frontmatter, and a small CLI (dba) deterministically syncs them into that table — since databases don’t let you drop files into them, the table becomes the file. Agents then query it at session start the same way they’d read an AGENTS.md, making it a natural companion to INFORMATION_SCHEMA: one holds structure, the other holds meaning. Max calls it a “soft semantic layer”, which maps directly onto the hard-vs-soft split he described above. Check out the repo at db-agents.

With context being key for agents to understand what we humans know, Agor also added a context layer called knowledge. Agor Knowledge acts as a central place where humans and agents can store, organize, connect, and find the context that makes work compound over time, with Slack quickly becoming the main interface to many of the team’s agents.

How Do We Sandbox Agents for Safe Workflows (Okta for Agents)

Another big topic is security when agents have so much access to powerful CLIs, sometimes root access to systems or databases containing private keys, or just downloading random skills from the internet that may contain hidden secret messages.

Max coined the idea of Okta for Agents, which I found super interesting, and something I believe will become ever more important if we want to find a healthy way of working with agents in enterprises or with sensitive data. Okta for Agents means working around identity, scoped delegated permissions, leases, and audit logs.

When asked how he’s managing security, verifying what Agor or the agents are doing, Max responded:

I let the workers run in god mode³, but using dedicated environments/sandboxes, hooked to a dedicated git worktree repo, it can run autonomously and solve problems on an initial prompt, visualized in a shared canvas style.

I asked how he sees Okta for Agents being implemented. We desperately need it, he said, granting agents permissions like impersonation. Delegating the permission is an OAuth. With the roles, we can scope permissions strongly. E.g., the sales agent only has access to sales documents.

When asked at what level to integrate the Okta security layer, Max said it hasn’t been solved yet. Still, he sees it as the same question: whether we have 50 agents or 50 users who use a platform, both need a security layer.

Likewise, Max shared:

I trust agents the same way as I would an employee.

[!note] What does “Okta for Agents” mean in more details?
Okta is the identity layer companies put in front of their tools: it authenticates who you are, decides which systems you can open, and logs what you did. Max’s point is that agents need the same layer. E.g. Clawdbot, when he tried it, was effectively a DIY IAM manager for agents, config hell and all.

The twist is that it adds a whole new dimension to RBAC. It’s not just “the bot gets an email account” — it’s “the bot gets an email account, but can only read mine, and via MCP rather than as a real user.” Not so different from onboarding a human personal assistant, except this assistant can help with nearly everything, so the blast radius is much bigger.

Max’s own example: he saw a 1Password skill and immediately backed off then reconsidered, wondering whether the bot should have its own 1Password account with only safe credentials shared into it. Which is exactly the problem: you can be strict on paper, but the moment the agent has your email, Slack, and calendar, it can leak private things all day. (Full discussion at this post)

[!warning] Security is critical. Here are examples when it’s gone bad
How I Dropped Our Production Database and Now Pay 10% More for AWS, or another one, or when 13-hour AWS outage reportedly caused by Amazon’s own AI tools. Or also just hacks by getting attacked via GitHub PRs or How secret instructions injected into skills.

Declarative and Non-deterministic Outcomes?

Related to security is the deterministic, repeatable behavior of data sets with the same input. Agents are the opposite: probabilistic. I was curious to hear from Max, who initially defined the functional data engineering paradigm for deterministic and idempotent batch data processing, what he thinks about the non-deterministic outcomes of agents, specifically with large language models.

Max said, regarding declarative definitions, that he finds a claude.md is usually sufficient for most tasks that have a git repo, more context, and an issue or PR to work with, given the initial prompts come from users who know what they are doing.

Regarding reliability, Max thinks about using good methodology references. Agents get it and understand it. E.g., data modeling practices such as Kimball are still valid, or the approach shared by him with Entity-Centric Data Modeling (ECM), he says, and when prompted to model in those patterns, agents follow them well (either via research or provided).

The other part is that some non-dangerous work can have vibe data pipelines, and there’s no danger. And there are cognitive-depth tasks, such as a complex Spark cluster, where you can’t just debug quickly with large data sets.

Also, the field varies: not every area is getting agentic-piled as fast. E.g., platform demands go through the roof (see GitHub outages), so we have 10-20x the platform needs, but at the same time, the work is critical to be correct. So it depends.

Future of Agentic Workload, and Canvas Development in Teams

When asked about how Agor has changed how they at Preset develop products (if at all?), or made them more effective, Max said:

There are more agents than humans nowadays. Everyone has a Claude Max plan, and agents handle almost all code writing.

And on a personal level:

I haven’t written a function by hand for a long time, and I might not anymore — except when I feel nostalgic.

He also thinks that the Yap-to-Ship Ratio, a metric that he announced half-jokingly on LinkedIn, describing people’s velocity by just getting stuff done without involving others at every step, will be very low-yap for 10x engineers, as they solve the problem and ship a solution without much back and forth.

They deploy it somewhere for others to use, not only for human consumption, but as a solution or CLI that other agents can use to discover further and solve their problems. A high Yap-to-Ship ratio would mean lots of human interaction, which is the clear new bottleneck.

As human code review gets bottlenecked, I asked how he does the review. He said he uses Codex with sub-agents to review, ensuring everything is DRY (Don’t Repeat Yourself) and that all expected callbacks are made.

You can also ask the operator assistant agents if you are not sure whether an implementation is correct.

Amdahl’s Law: Can’t Go Faster if not End-to-end

One bottleneck is still Amdahl’s Law. We can speed up tooling by using extremely fast agents, but unless the end-to-end workload is sped up, we only increase by a 2-3x factor, not 10 or 100 as any one tool does. This also overlaps with Max’s Yap-to-Ship ratio: if PRs need the human in the loop to review many of them, the overall speed at which we build is not faster.

Another side effect is that it takes a lot of context switching. Max said he has ten active sessions in Agor. He is good at context switching (maybe also learned through recent Agor workflow? 🙂).

Predictions for 2026

It’s hard to predict the future with AI, but Max took a stab and shared his predictions for 2026 and categorized them into wired and tired:

Max’s AI Setup for Data Engineering Work and Managing His Company

We end this interview with Max’s setup for working with agents, since we didn’t have time to go into full details on the call. I’m sharing the one he shared four months ago. I’m sure it changes almost daily. Still, it helps us get a good overview of his software engineering stack for the team at Preset, as well as his personal local computer stack.

For software engineering and data engineering:

• Preset team instance of Agor behind VPN with a dozen boards, boards are mostly repo-oriented. Full Unix impersonation, backed by PostgreSQL.
• doing most of my work on board with the agor-openclaw framework, agent is pushing projects across a kanban-type layout: tons of new automation there. Agent checks on agents, prompts them, moves worktrees to “needs human review” zone if/when needed
• coding workflow is Opus 4.6 as a planner, Sonnet 4.5 / Opus 4.6 for most coding, Codex 5.3 as the reviewer (god it’s so good)
• data engineering stuff: dbt/airflow repo + Superset MCP, superset-sup
• “Command center” is agor-openclaw on Opus 4.6, monitors other agents, intricate HEARTBEAT.md with pseudocode to make a decision for coding project (worktree) on that board

And his new “Personal assistant” local instance of Agor (brand new/sensitive):

• Beefy Mac Studio at home
• connected to “productivity” tools (google-workspace-mcp)
• connected to Slack through a semi-homegrown skill — read-only for now, mostly to summarize activity
• connected to Notion MCP
• connected to “contracts” repo, where I sync with Google Drive for all Preset contracts
• connecting to Hubspot soon
• goal is to automate most of the automatable CEO-stuff as discussed above

Coming up

We’ve learned how to use Figma for agents with Agor and to collaboratively work as a team, using shared prompts and creating artifacts. We’ve seen how Okta for Agents is needed but really hard to implement, and how the future of AI is mostly about context and how to integrate it well. Plus, we learned how Max automates many tasks as a CEO with dedicated AI assistants and still produces low-level code with the same assistants for both his personal and company-wide needs.

I hope you enjoyed this fourth interview with Max. Huge thanks to Max for taking the time to speak with me (twice!) and for sharing his experience with all of us. Follow him on LinkedIn, GitHub, or on Preset Blog, where he shares his distilled thoughts on the ecosystem, and obviously, if you want to know more about Agor, check it out at Agor GitHub repo.

Max shares a lot of his ideas and thoughts online. Here are some further articles and interviews to read/watch:

• AI Enablement Engineer: The Highest-Leverage Role in Tech
• Check the Joe Reis interview where Max shares more on the discussion From Airflow to AI Agents: On Building Agor and Running a Company with AI Agents

More interviews are coming out, so please share feedback, questions you might want to ask, or your experience working with AI in the data space. We’re all in this together, figuring it all out.

Full article published at MotherDuck.com - written as part of my services

These are my recently read books and some comments and notes from when I read them.

As I love books and recommendations by others, I want to share them in a collective format so others can read gems of books, as I think books are still the best way to read new information these days. Even more with the fast pace we are going at, as books are well structured and made for the long term.

Book Recommendations I’ve Read

Over the years, I have made recommendations in Writer’s Room, my newsletter, on my now page or in person. Here I have collected and curated them over the years including some comments. I hope you enjoy.

2026

• Poems & Prayers (Matthew McConaughey): Another one read by Matthew himself. It’s about life and its rhymes. He said life is like rhymes as he likes them, sounds good, and prayer because of the meaning. Together applied to life, it’s like a dace to the day. He also says he has a rhythm in his head when he wakes up, and then he dances to it somehow during the day, and life fits into the song, making his day like a dance. Also, people say he is so laid back, but it’s because he has a plan, he does not rush, he takes time. So that’s on purpose. Something I also actively try to do in my life. Not yet finished.
• Maintenance of Everything (Stewart Brand): This is just so good, it tells the story of how maintenance is everything if you rely on a boat on the sea, for example. Making the product easy to repair so you can fix it even in a rough sea with limited tools. It’s an analogy you can apply to every part of life, and the way the story is told is so great. I couldn’t stop listening to the audiobook. I really recommend it to everyone; I’m sure it will give you lots of insights into your current work.
• Big Time (Laura Vanderkam): Started with the goal of getting more out of my time. So far, I’ve learned that time management is essentially about controlling time, but it’s not the only way. And that “Shifting Left” is not only used in data engineering, Laura describes doing tasks long before they are due as shifting left too. Good reminder of how to use time better, nothing revolutionary, but lots of real-life examples from her and her clients.
• The Writing Life (Annie Dillard): A poetic way of what it means to be a writer. Great storytelling. Learning how to write with a family as part of her stories.

2025

• Novelist as a Vocation (Haruki Murakami): Such an amazing book on writing. It really reminded me of my writing style, as he translates words and tries to write beautifully for the sake of beauty, not to impress the reader. No timeline. It takes as long as it takes until the quality is good. And he likes a good challenge to keep him engaged, e.g., when he goes abroad to start from scratch. His style is so unique and amazing. It’s a joy to listen to.
• Greenlights (Matthew McConaughey): This book was so real. Matthew reads and shares from his 35 years of journaling and life. So inspiring and [[Writing from The heart|straight from the heart]]. It was interesting and motivating to hear from a Hollywood actor and celebrity what truly matters: family and keep livin. Alright, aaaalright, alriiight. → I listened to the audiobook, and his Texan accent was the best. He was rhyming, slowing down, calming down, getting loud, all of it. It was like sitting next to him and listening to your dad or uncle.
• Consider This (Chuck Palahniuk): The author of Fight Club shares how to write if you were his student. A masterclass if you want to enhance your writing.
• You Can Negotiate Anything (Herb Cohen): Interesting perspective that you can negotiate even at places you’d never thought of, even at a shopping mall. Always keep in mind what the other’s objective is. There are always three elements that are part of a negotiation: information, time, and power.
• The Bible: I also started to read the Bible, the most sold book on earth. I wanted to understand what’s in there. Obviously, it’s so massive, and it’s actually not one book, but a collection of many. I get a similar sensation to one of my favorite books The Daily Stoic (366 Meditations). It’s like a daily meditation. While Ryan Holiday’s book helps me feel calmer, the Bible nourishes my love and care for others.
• Stolen Focus: Why You Can’t Pay Attention (Johann Hari): This book inspired me to write an article about flow myself: Finding Flow. Super insightful and probably will make you want to ban social media by the government.
• Useful Not True (Derek Sivers): Derek is the master of stripping away. This book has so much wisdom in short one-page stories from Derek’s life. It’s a gem. Perfect to re-read, over and over. It’s next to my sleeping place to just grab.
• Tao Te Ching (Lao Tzu): Dao and Daoism: Men - earth - universe - dao. If you don’t want to read the bible, this is a none religious way of getting solitude and compassion.

2024

• The Good Work (Paul Millerd): A continuation of the famous Pathless Path, not as good, but OK. Please read it if you haven’t yet.
• The Extended Mind (Annie Murphy Paul): A powerful book. I started reading it again.
• Do The Work (Steven Pressfield): It’s not the best book, maybe also because I’ve heard or read lots of the content already elsewhere, but I appreciated that it was short and that he didn’t try to extend it to 300 pages.

2023

• Slow Productivity (Cal Newport): This is a fantastic book. It showcases why we should quit the race of everyday life and slow down for a compounding effect instead of overnight success. The same is true for money: investing long term instead of gambling in a casino. He describes that instead of pseudo productivity, which is the norm these days, where we try to be as busy as possible to showcase we are doing something, we should try to be as productive as possible.
• Elon Musk (Walter Isaacson): It’s very long, but I loved every page (or word as I listened to it on Audible). It is inspiring. It shows how an exceptional, hard-working Musk is doing everything for humanity in an unhealthy way. He is also, to an extent, sick. He has his dark sides that just come out sometimes, which are also a sign of his mental state that he got from his dad, and also from his hard work and pressure. He is thriving in chaos. Whenever there is a calm time, he will do something new as he can’t stand the status quo.
• How Will You Measure Your Life? (Clayton M. Christensen): A good book if you want to know more than just work and how you’ll measure it. This book was reassuring and strengthened many things I already knew. It reminded me that finding your principles is the key. Follow them and align your life so you have a happy life. It was also helpful for me as a dad and family member to pass on the same principles and values to my kids. Be intentional about your values. It’s hard to find them. They won’t be sent to you. You need to make them. But be aware that it is a process, not an event.
• The Good Enough Job (Simone Stolzoff): A compelling book that challenges our conventional thinking about work. Instead of idolizing our jobs or incessantly chasing a better one, Stolzoff advocates for finding satisfaction in a “good enough” job. His ideas offer a refreshing contrast to the pervasive Instagram-era narrative that equates career success with personal fulfillment. A highly recommended read for anyone feeling pressured by the modern-day cult of work.
• The Daily Dad (Ryan Holiday): This book is a sequel to one of my all-time favorite reads, The Daily Stoic. In the same tradition of offering daily philosophical advice, this book focuses on the challenges and rewards of parenthood. TODO
• The Extended Mind (Annie Murphy Paul): An exploration of the intriguing ways our environment influences our thinking processes. Discover the surprising ways in which experts think beyond their brains, how harder thinking often leads to fewer results and the controversy over brain-training games and smart pills. Explore how we can use tools beyond the brain, such as the Body Scan technique and meditation, to tap into our intuition and sensations, and learn about the significance of the amygdala in our responses to stress.
• All the Wrong Moves (Sasha Chapin): A captivating narrative where the world of chess serves as a backdrop for introspection and self-discovery. A story that shows how the love of chess can fully dominate one’s life, as it’s the most beautiful and worst thing in life at the same time.

2022

• Atomic Habits (James Clear): The main argument: If you want to add a new habit, chain it to an existing one to make it stick.
• Building a Second Brain (Tiago Forte): I knew most of it from his articles, podcasts, etc., but now it’s also available as a book.
• Getting Things Done (David Allen): This book is life-changing if you apply its principles correctly to your life.
• How to Take Smart Notes (Sönke Ahrens): The base for a Second Brain and where Sönke reveals how our brain is wired and how we can implement a note-taking style that supports our brain, mainly with the method called Zettelkasten.
• The Pragmatic Programmer (David Thomas & Andrew Hunt): Although a lot was clear, summarizing it and putting it together as one piece, plus hearing it from two professionals, was very helpful and suitable for applying to my work.
• On Writing (Stephen King): He gives deep insights into the life of a successful writer.

2021

• The 5 Love Languages (Gary Chapman): Really eye opening, as sometimes we want to show love to our spouse, but becausewe give attention, or buy a gift, but if it doesn’t match the “love language” of the reciever, it won’t arrive properly. So knowing each others love language, and communicate in that, is huge for any relationship.

2020

• The Courage to Be Disliked (Ichiro Kishimi, Fumitake Koga): Great book to learn new psychology, especially the Alfred Adler theories of individual psychology and Adlerian therapy. And learn about Interpersonal Relationship. This books makes you stronger.
• Your Music and People (Derek Sivers): Fantastic. It tells the story about how to sell your music, which is an analogy to every business.
• Mindwise: How We Understand What Others Think, Believe, Feel, and Want (Nicholas Epley): Approaching our mind from an angle you probably haven’t thought of. Very insightful.

2019

• Thirst: A Story of Redemption, Compassion, and a Mission to Bring Clean Water to the World (Scott Harrison): This book is truly heart-touching and lets you rethink today’s world and donations. Very well-written book.
• It Doesn’t Have to Be Crazy at Work (Jason Fried & David Heinemeier Hansson): Talks about the crazy work hours usually suggested or heard at places like Google, Amazon, etc. to be able to achieve something. They tell you how you can tackle it otherwise.

2018

• Principles: Life and Work (Ray Dalio): I recommend this to everyone out there. The first chapters where he talks about himself you can skip, but the rest is worth reading or listening to. TODO
• Unshakeable: Your Financial Freedom Playbook (Tony Robbins): If you want to save some money in the long run, please read it. It is outstanding and straight to the point!
• The Tao of Seneca: Practical Letters from a Stoic Master (Seneca): Puts me back to the essentials of life whenever I feel all over the place.
• Tribe of Mentors: Short Life Advice from the Best in the World (Tim Ferriss): Entirely recommendable.

General Recommended Books

These are some of my all-time favorite books that I recommend to everyone.

Top Books

• The Daily Stoic (366 Meditations) (Ryan Holiday): One of my all-time favorite books. Daily meditations that help me feel calmer and more grounded.
• Hell Yeah or No (Derek Sivers): Anything by Derek Sivers is worth reading. His writing is stripped to the essentials and packed with wisdom.
• How to Live (Derek Sivers): The second book by Derek on top list. He writes 27 conflicting, and short chapters on how to live life. He has done so much, that he writes almost each from experience. As it’s short, each chapter can give you a nudge and insight into your own life, and what we should strive, and more importantly, what not. Be aware, every unnecessary word is removed and stripped to the bare essentials. This means you can’t read more than two chapters in one go as it’s that condensed and makes you think a lot.

Business

• It Doesn’t Have to Be Crazy at Work (Jason Fried & David Heinemeier Hansson): Challenges the norm of crazy work hours and shows you can achieve more with less chaos.
• Deep Work (Cal Newport): Essential reading on focused work in a distracted world.

Self-Help & Life Philosophy

• Thinking, Fast and Slow (Daniel Kahneman): A deep dive into how our minds work, exploring System 1 and System 2 thinking.
• The Pathless Path (Paul Millerd): Challenges the “default path” (going to school, work, mary, and have a family) and following your instict. Unconventional way. The new 4 hour work week book. One that I recommend the most lately. I wrote about Finding my Pathless Path.
• Four Thousand Weeks: Time Management for Mortals (Oliver Burkeman): A refreshing take on time management that embraces our mortality rather than fighting it.
• Principles (Ray Dalio): Life and work principles from one of the world’s most successful investors.
• Emotional Intelligence (Daniel Goleman): Understanding and managing emotions for better relationships and decisions.

⭐ 5-Star Picks (GoodReads)

• On Writing Well (William Zinsser): The classic guide to writing nonfiction. Essential for clear writing.
• So Good They Can’t Ignore You (Cal Newport): Why skills trump passion in the quest for work you love.
• Steal Like an Artist, Show Your Work!, & Keep Going (Austin Kleon): Probably the best book on creativity and if you are a creative yourself. It’s really helped me sharing more of the process and how creative works. It’s a trilogy, I listened the audiobook, which you get three for one, fully recommend it.
• Still Writing (Dani Shapiro): Honest reflections on the creative life.
• The Psychology of Money (Morgan Housel): Best book on understanding money and behavior.
• Moonwalking with Einstein (Joshua Foer): Fascinating journey into memory and how it works.
• Open (Andre Agassi): Raw, honest autobiography. Surprisingly insightful about life and finding yourself.
• Anything You Want (Derek Sivers): 40 lessons for entrepreneurs. Short, punchy, full of wisdom.
• The Subtle Art of Not Giving a F*ck (Mark Manson): Counterintuitive approach to living a good life.
• The Alchemist (Paulo Coelho): Beautiful fable about following your dreams.
• Hooked (Nir Eyal): How to build habit-forming products. Essential for product designers.
• The Startup Way (Eric Ries): How modern companies use entrepreneurial management.
• The 4-Hour Work Week (Tim Ferriss): A game changer that questions conventional work structures.

Further Reads

• Reading Books for a Happy Life: My thoughts on why reading books matters.
• Audiobooks: I listen to most of my books as audiobooks. It works better for me and my brain, as I read all day long already. Books read by the authors on Audible are just another level. It feels like listening to them in person. Although a physical book has advantages, audiobooks are just something different. No wonder they are ever-growing.
• Writer’s Room: Where I write and update on latest insight in my design process.

Other Recommendations

• Derek Sivers book notes

Last updated: December 30, 2025

Creation is the ultimate form of pursuing ourselves, giving to the world when shared, and using the power of our subconscious. It gives us joy, and to every artist, it is the ultimate (flow) state of happiness.

The Act of Creation

The act of creation is an outlet. It gives joy to us when we create something out of nothing, we block out anxiety or boredom.

Creating should be done like:

• a kid in mind: effortless, exploring your thoughts, and seeing where it leaves you, as Picasso said: He wouldn’t have bothered to start a painting if he knew the outcome.
• a play¹: flawless and free.
• calm like the water: flowing wherever your mind is going.
• to fulfill yourself: by portraying your never-resting thoughts. Share it with the world. Create with joy and love in mind.
• follow your train of thought: see where it leads you. You start somewhere, on an unfinished emotion, a task, or anything to ponder on.

Creation, and especially writing, can help resolve the unresolved.

Creation to Make a Difference: Joy and Laughter

Creating is also about making a difference. If nothing changes in you, the people, not enough value was added to your creation. Any creation should trigger something. But foremost, the process of creation should change something in you, some release, a good feeling, it must feel right. If it does nothing, an empty feeling, the creation might not be there yet.

Create with love and empathy in mind. Liking someone is one thing, but giving love, caring for someone, and integrating that into your work is the ultimate way to create.

Create joy and laughter. Speak the language of people, make them happy by creating enjoyment.

Following Our Emotions without Boundaries

When we follow our emotions without any boundaries, following our instincts, we get into a state of deep focus and deep concentration where all thoughts can be resolved.

Creation is our inner [[Gut Feeling|Instinct]]. If we follow and nourish it, great things can arise from it. Treat it as an outlet for your body and mind. Go with the flow, follow along. Create for the sake of creation.

Learn for Life, Follow Your Own Uniqueness

The act of creation is following your own life rhythm, for example, the [[Pathless Path]]. It is the ultimate form of connecting with yourself.

All creation is unique, unique specifically to you as the creator, but also (hopefully) to the reader, viewer, or consumer of your work. Your creations live long after your death; others will connect and create new works from them. It’s giving joy after you’re gone.

Learn for Life, as I like to say, has been my earliest mantra when starting any of my creations shared online. The common definition of success is reaching a desired outcome. The Scientist’s definition of success is: if you learn something new, you haven’t failed, as Anne-Laure Le Cunff says on How to Design Tiny Experiments Like a Scientist.

For example, a scientist defines it like this: if you say you want to write more, instead of just writing, you define ‘I will write for at least one month, or at least 10 articles. And then you don’t stop before that. Success is the experience, not the outcome.

The Outcome of Creation

The act of creation is a slow process that needs time and experience. A slow life, an [[The Ordinary (Boring) Life|ordinary, boring life]] even, to focus on the details, using the craftsmanship refined over the years, and retrieving joy from the process of creation.

What matters is the quality and outcome we are proud to release to the world.

Gifts Shared with the World

You create for yourself with no return in mind, you just share it as a gift for the world, for anyone to consume.

Creation thought of as a gift is an easy way for you to create without the burden of pleasing people, as it’s take-it-or-leave-it, like a gift. No strings attached, just making gifts.

A Curious Mind is Mostly a Subconscious Mind

When we create, most of our subconscious is driving the thought. Just let it cruise and see what the outcome is. Use the conscious mind for fixing errors and making sure the sentences make sense later, but don’t start with it.

The Spark of Joy Philosophy

Different flairs for designs to add ease or user delight to your creation. Similar to the “Spark Joy Philosophy”, which is very fitting when creating for the enjoyment of the reader.

To me, creation sparks joy for myself, it’s the outlet for me to release and organize my thoughts. It avoids the [[shallow happiness]] that I get from social media, binge-watching Netflix, or other brainless activities on the phone or TV. Sure, there’s time for that too, to calm down, but if we are not careful, the algorithms will take over and we default to always choosing the easy choice.

The Reward is Long-term

Creating is hard, there’s friction, it does not directly work, but the reward is long-term, with [[deep happiness]] and a deep flow state: “It’s magical that they just tried to be there”, which is the highest form of happiness artists get (see The Artist’s Way) as we discovered in Finding Flow, with escaping digital distractions through deep work and slow living.

I hope you find your outlet for creation, an act you can partake in to develop happiness and joy for yourself, getting into that deep flow and having a slow process where many small gifts of creation can be shared with the world.

This article was created with the inspiration of reading How to Live by Derek Sivers in Cavallino, Italy.

Grammars for languages or any other field are a beautiful thing. They compress complex systems into a language with a couple of rules. For the spoken language example, we know when to capitalize a letter or how to start a sentence. There are clear rules. Grammars also help us remember, as we do not need to recall every little rule, but apply them in a structured way.

For text editing, we have Vim motions that help us navigate a text document with 1000s of shortcuts, but because there is a grammar, we do not need to remember them all, but learn the structure of the grammar and combine them. But what if you work in data? What if we could have the same for data, a grammar for data engineering, or a language that defines it?

Expressing our needs declaratively and decisively? Also, expressing it in a way that leads to reproducible outcomes, or works with multiple parts and execution engines already out there. This is what we will discuss in this article. How existing tooling, such as Ibis, provides some capabilities, and how xorq extends them by adding full lineage and transparency for humans, with included executable memory for useful tabular data, all manifested in a single git repository.

Expressions for Data Engineering Workloads

Having a grammar for data engineering means we can express the workloads in a declarative manner, and then be sure we can deterministically reproduce and apply that exact definition.

It’s similar to the concept of a Declarative Data Stack I introduced a while back, but it gives the stack not only configurations but also a language with in-built manifestation and execution engines.

In the above image, we see:

1. How to express (write) our transformations and business logic. It’s the context of every ML or DE pipeline.
2. We can build the expression into a manifest that has a unique hash, runs input validations, tracks lineage, creates a deterministic cache, and produces a human-readable expr.yaml you can diff and review in a PR.
3. Lastly, we can execute it in any execution engine with the same manifest.

This is hugely powerful and separates the concerns of defining logic, verification in the manifest step, and execution as a composable data stack, as Wes McKinney called it, with multi-compute engine possibilities.

How the DE Language Works: Different Expression Types

Every grammar starts with nouns, and here the noun is the source, a node that holds data but carries no transformation yet. It might be an in-memory table, a registered connection to a warehouse, or just a lazy pointer to a file on disk that hasn’t been read. They’re simply referenced, the way a noun refers to a thing before any verb acts on it.

The verbs in our language are transforms such as filter, select, mutate, aggregate, join, order, limit. Each one takes a source (or another transformed expression) and returns a new, immutable expression. You do not mutate anything before it, only describe what should happen next.

Looking at a definition such as .filter(...).aggregate(...).mutate(...), we can see this as a sentence. The moment a verb is applied, the expression stops being a plain noun and becomes a statement, a description of “data plus what should happen to it.” But the sentence isn’t spoken yet, it stays inert, fully composed but unexecuted, until something finally asks it to run. That’s the deferred part of the grammar: writing the sentence and saying it out loud are two different acts.

There’s a third part of speech worth naming: the template. Instead of writing a sentence about a specific noun, you can write one about a noun’s shape, a schema with no rows behind it. A template says “given something with a column of this type, here is what I’ll do to it,” and only later gets bound to an actual source, at which point the placeholder resolves and it becomes an ordinary statement again.

And we have modifiers that ride alongside a statement without changing what it computes. They’re small tags of metadata that say “this expression also represents a fitted model” or “this is a saved reference to something else.” It’s like a footnote with additional metadata that doesn’t change the surface meaning, but adds context for later use.

This analogy makes the grammar compose the same way regardless of which engine eventually executes it. There are more parts, but with just these four, noun, verb, template, modifier, you can read (and write) arbitrarily complex data pipelines the same way learning a handful of verb-and-object combinations in a text editor lets you compose arbitrarily complex edits.

[!tip] Avoids building “Inner-Platform Effect” with repeated tools
With this grammar, we can avoid repeatedly implementing the same logic we already have, but manifest and express our logic once, and reuse it with different execution engines, exactly what Ibis and xorq allow. Similar to what the inner-platform effect means for software best practices.

Why a Grammar is Really Good for LLMs

Having a grammar is really good for LLMs, too. It helps them first to declare data artifacts and second to execute them reproducibly.

On top, expressions can be LLM-agnostic, and we can interchange the LLMs we use just with an expression. Also, the chart is just an expression, or the data catalog and the metrics.

Model Once, Represent Everywhere: Expressing the Full Data Stack with a Single Expression

Like UDA (Unified Data Architecture) from Netflix, we define our expressions once and represent them everywhere. Netflix built UDA to solve duplicated models, inconsistent terminology, and siloed systems, where the same concept like ‘actor’ or ‘movie’ gets modeled differently across teams, with no shared foundation. Their answer was a full knowledge graph with a metamodel, making the conceptual model part of the actual control plane.

Not everyone needs Netflix-scale tooling, though. For a code-first approach, xorq gives you the same core principle: define once, execute anywhere by writing a declarative Ibis expression, serializing them as content-addressed YAML artifacts, and running against any supported engine, fully reproducible.

The difference worth noting: UDA is a semantic layer defining what data means across systems. Xorq is a computational layer defining what transformations do across engines. Both reject the same anti-pattern of re-implementing the same logic for every system.

Entering Xorq: The Horizontal Data Architecture

Xorq is an executable memory system for tabular data that works horizontally across your data stack, supporting everything from discovery with a catalog to defining transformation logic to modeling.

It has declarative transformation (Pandas style), and you can build ML pipelines and prepare data with its semantics in a single stack that is not vertically integrated, but horizontally integrated, giving your agents a catalog of executable pipelines and turning short-lived agent work such as wrangling scripts, sklearn pipelines, ad-hoc tables into durable, composable, executable artifacts that any future agent or human can discover, reproduce, and reuse.

The horizontal data stack shows what Xorq brings to the table. Xorq’s origins started from a git-native semantic layer, for data analysts out of college, to build semantic models for a living, to make their lives easier.

From point-and-click tools, dragging tables and drawing joins manually, only to add more reporting tools on top to create pixel-perfect reports. Also performance-wise, it didn’t scale, meaning we needed cubes to make it faster, adding another layer of complexity.

And there was no lineage that shows from source to dashboard. The question asked was: “what if we could do this end-to-end data engineering workflow locally?”. This is what the horizontal data stack and xorq are providing.

To add semantic layer capabilities, Julien Hurault and Hussain built the Boring Semantic Layer + the Xorq catalog, providing a semantic model you define in Python, check into git, and query from the CLI.

Compressing Logic into a Single Executable

Compression of a full data stack into a single executable is hard, but xorq tries exactly this with the help of Ibis, git, uv, and DataFusion.

The design choices of xorq showcase even better what it is, and what they enable:

• Ibis as expression layer (v9.5.0+, partial): Declarative dataframe expressions compiled to multiple backends (xorq supports a subset of the Ibis API, not the full surface)
• Git for state and storage: The catalog is a git repo of entries with git-annex support for large files
• uv for reproducible environments: Each entry ships with a wheel and pinned requirements.txt.
• DataFusion for embedded compute: Pipelines execute in-process with SQL and UDFs

Composable Data Engines

Another big advantage of expressions and having a grammar for data engineering is easily switching between backends, with no change to the transformation or business logic. It’s just defining the backend from Apache Arrow Flight to DuckDB or any other engine.

We write the definitions and express our tabular data and computations. The engine, in this case xorq, can build it into a manifest file that is deterministic and hashed.

Xorq uses Ibis as the expression layer for single-backend logic, then builds the cross-engine expression tree into a serialized YAML artifact. When moving data between backends, xorq transfers Apache Arrow RecordBatch streams between them—each backend acts as a RecordBatch transducer. No CSV serialization, no JSON encoding needed. This makes backend switching fast and memory-efficient. Write declarative Ibis expressions that run like a tool—xorq extends Ibis with caching, multi-engine execution, and UDFs.

Here’s an example of using DuckDB and Postgres in conjunction:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import xorq.api as xo

# Connect to engines
pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

# Load data from different sources
batting = pg.table("batting")
awards = xo.examples.awards_players.fetch(backend=db)

# Filter in respective engines
recent = batting.filter(batting.yearID == 2015)
nl_awards = awards.filter(awards.lgID == "NL")

# Move data to postgres for join
result = recent.join(
    nl_awards.into_backend(pg),
    ["playerID"]
)

result.execute()

Move data between different engines within a single expression using into_backend(), here Postgres and DuckDB

You can see how easily you choose your most optimized execution engine, whether in the above example choosing DuckDB for filtering recent batting and using Postgres to filter NL (National League) awards, and joining the two with the Postgres engine.

Engines supported by xorq as of now, with the ability to move data between them, are (check Supported backends for the latest):

• Embedded: DataFusion, DuckDB, SQLite, Pandas
• Warehouses: Snowflake, Databricks, Trino, Postgres
• Lakehouse: PyIceberg
• Arrow Flight: GizmoSQL (DuckDB over Arrow Flight SQL)

Cross-Engine Expression Tree

With different engines supported, we can use the compressed single executable logic across engines. We can build expression graphs before executing them, which works like this, with one expression, many engines:

1
2

expr = penguins.into_backend(xo.sqlite.connect())
expr.ls.backends

The output of building a cross-engine expression is a directory containing your serialized pipeline with a unique hash identifying each build and its artifacts and expressions. When executed, the output is the resulting object or data.

And the expressions are tools, Arrow is the pipe. E.g., a Unix pipe streams text between small programs. Xorq pipes Arrow streams between expressions: unix : programs :: xorq : arrow-transforms

That executes like this:

1
2

In [6]: expr.to_pyarrow_batches()
Out[6]: <pyarrow.lib.RecordBatchReader at 0x15dc3f570>

This is quite short and potentially abstract to understand when never used, but we will go into more examples and details in another article.

A Shared Language for Data

This article introduces a new way of describing data transformations for machine learning or data engineering pipelines in a direct and simple way that works locally with any execution engine, without changing the code itself.

It’s a good place if you need a trusted harness for a data engineering persona. We can define once and use it with the engine that works best for your workload and data engineering environment.

We had a look at how we write -> manifest -> execute with xorq, its advantages, and why you might use it for modeling once and representing everywhere. By adding AI agents to the mix, which help us pull the right lever, instead of bigger, more expensive models or more tokens, we improve accuracy with more semantic understanding, with a grammar the model can learn and apply, even pre-manifest before execution, and run them deterministically every time. This is a huge addition to working just with agentic Skills files that are free-form Markdown and pull data all over, or are not defined precisely enough. It’s all about having high-quality context in the right format, with a clear definition where humans and AI agents can interchange and help each other.

There’s a lot more to come, with showcasing the horizontal data stack and the use cases it supports, how we build expressions versus running computations, and how data catalogs are integrated into the picture, too.

–

Check out xorq code and star it on GitHub, or read more behind the scenes at Xorq documentation.

They also have a macOS desktop app coming up that does it all in one unified app, geared towards non-technical users. Join the waitlist for that.

Full article published at xorq.dev - written as part of my services

With ever-changing models, new and better ones coming out every few months, it’s great if we don’t have to rely on them too heavily. The better your tooling, the less dependent you become on any single model. That’s also why the deterministic harness matters: a correctness layer that lets you reproduce outputs and trace lineage regardless of which model you’re running underneath. This is especially true during maintenance or extending the project, where verification is the real job.

The danger isn’t only a crash or an error message, but a wrong number that didn’t break. It might be a clean query, but it introduces duplicated rows.

In this article, we go through the three levels of AI agents in data engineering, how to structure projects so the AI delivers its best outcomes, and how dedicated agents with a deterministic core help us build higher-quality pipelines — ones we can actually trust. And we look at a practical example of how it works with a blast radius analysis.

The Three Levels of AI Agents in Data Engineering

Why should we use agents for data engineering? And at what levels can agents help us productively? As LLMs will always have some error tolerance, as humans do too, we need a way to be more confident in producing the code.

Chat-phase, Autonomous and Dedicated Tooling

There are different levels of confidence and levels on which the agents can help us.

1. The initial chat-phase: the development where we prompt Claude or ChatGPT. The model tries to understand the context based on what it has access to. It takes a decent amount of tokens, as it needs to scan everything from scratch.
2. The autonomous approach, where Claude Code or Codex also have access to the tools humans have, mostly the CLI on the terminal, making it possible to query Postgres with psql or read from S3 or Parquet with DuckDB to verify queries and data. A much higher quality outcome.
3. Dedicated agents for the task at hand. E.g., for data, the tools know dbt or know how to transpile SQL code deterministically, meaning not from training data only, but with an actual tool that does it much faster and more reliably. Built-in checks and features a “general” agent can’t provide.

Ideally, we’d want to always use the dedicated tools, but there isn’t always one.

Where in the DE Lifecycle Each Level Actually Helps

BI Dashboards vs. Plumbing the Data Pipelines, or Creating Source Ingestions, or Maintaining? For data engineering, the question is not only if there is dedicated agent tooling, but also on what part of the data engineering lifecycle AI agents can help data engineers and analysts the most, and potentially even domain experts?

The lifecycle contains the ingestion part, ETL, or understanding the business in great detail, or is it just to visualize the result? Or should it cover maintenance in case of overnight ETL errors, or the full data lifecycle?

In general, before we go into more details later, agents can help us on the full cycle, but it always depends on who you are and what role you play. Building from scratch with no knowledge or seniority is dangerous. Why? Because they can’t verify if the produced code is correct. Okay for a side project or a proof of concept, but not for actual production.

What’s the Engineering Discipline for Working with AI?

There’s also a part that is less technical, a way of guiding the agents in the right direction. Especially if we want to safely use it in large projects or organizations, we can’t just let it run without guidance.

For that we need:

1. clear project structure in which the agents can flourish. The more is given, the fewer tokens are used for this work, and it will be more aligned across the project. (Another reason a deterministic workflow such as uv init is best, because it will always be the same).
2. build with clear instructions (agentic skills, superpowers, etc.) on how the tools are used (basically providing CLIs and API documentation). This is the bulk of the work anyway. That’s the data architecture, the brainstorming with fellow humans before you build something, instead of missing a key insight in the beginning and then letting the agent run down the wrong path. Also, be realistic: prompting “be correct” or “use state-of-the-art” won’t make it more correct or more state-of-the-art than the model was trained on. So if it’s a rather new architecture, it’s a must that you provide these links and hints.
3. set up the project in a modular fashion, so the agents cannot break the whole project if they make a small change, so you don’t end up in a scenario like dependency hell with everything dependent on each other.
4. use a declarative approach, with descriptive configuration that says the what and not the how, so that you can collaborate on these configs with the agents, version them, and easily revert or change something, as well as decouple the implementation logic from the actual business logic.

With these steps, you can get the best out of the agents of today. I’d say the model matters less, but the structure does, and as Mario says, so does the workflow approach. For example, extensively plan (the process before writing a single line) and correct the model before any implementation that could lead down the wrong path is written.

Also, don’t overthink it. But this is only the workflow and learning the soft skills and discipline of working with agents. How does that look in a real-world project?

[!note] The key is to get use out of AI, not to get more work.
E.g., most developers used to think about the problem. Today, most drown in PRs. When the AI tooling gets better, AI can provide more quality code that is correct, that needs less review or fewer iterations, which means fewer PRs and less work for the developers to go through.

The Correctness Layer for Data Engineers

A key insight is that AI agents should support the “human in the loop” for correctness, or a correctness layer. And rather than making more work to verify more code, we should be confident in the process and know that the code it produces is verified and ultimately correct.

But how do we get more “correct” work and a layer in which we can verify it? The biggest argument is a deterministic-validation architecture in full. E.g., Altimate Code splits the agent into a probabilistic layer on top and a deterministic Rust/TS layer underneath that does the actual SQL ops such as parsing, validating, and equivalence checks, so that the agent itself never has to be trusted on those questions.

Altimate Code, for example, is built on a probabilistic agent, deterministic harness, and deterministic core. The probabilistic agent with the LLM does the creative work of reading intent, picking a strategy, drafting SQL, summarizing results, and recovering when something goes wrong.

Below the boundary sits the deterministic harness, a TypeScript layer that intercepts every tool call: a dispatcher checks hasNativeHandler before the call runs, and routes it either to a native, deterministic handler or back to the model. Those handlers don’t reimplement logic themselves, they call into the deterministic core, a Rust engine (altimate-core) that exposes SQL operations as pure functions over ASTs and schemas, wired in via napi-rs bindings. Parsing, validating, transpiling, checking query equivalence, diffing schemas, extracting column lineage, diffing rows across warehouses — all of it runs sub-millisecond, and all of it returns the same answer on the same input, every time.

Like a compiler, the agent never decides whether two queries are equivalent or a column exists upstream. Instead, it calls a function that proves it against the parsed AST and the schema, the same way a type-checker proves a program compiles rather than guessing.

That’s the distinction that makes the output easier to review, as factual checks have been run and the output is either correct, or there’s a bug that it can fix directly. The rest a human can re-verify. On the dilemma of having stopped to hand-write code and approving it faster than humanly possible to check, you can also read more at You Are the Trust Layer.

[!note] There’s another factor, being wrong
Bare agent use might be cheap, but only until they’re wrong, and then the cost is unbounded.

Improvements for Better Usage of Tokens

Altimate, or data engineering agents that have deterministic functions and integrated understanding of how to work, can help you save tokens and be token lean (the opposite of [[tokenmaxxing]], which is popular on Twitter/X, using as many tokens as possible and having an agent running at all times). Because in large enterprises, token costs are a real budget point.

To slow down the tokens, an easy trick is to instruct the model to use fewer tokens and words itself - caveman is a good example of that, but you can also add a singular prompt to your CLAUDE.md, Codex, or model of choice in combination with Altimate Code.

There’s a second, less obvious cost: the token itself isn’t a stable unit. When Anthropic shipped Opus 4.7, the same prompt that cost X tokens on 4.6 started costing roughly 1.4X (same input, same answer, more tokens, same price per token).

Altimate on The Great Token Heist of ‘26 makes the case that “cost-per-token is the wrong number to optimize”, since the meter itself can move with a vendor’s next model update, and what we should track instead is cost-per-task. I fully agree, and this is where deterministic function calls work around that volatility by not using a model/tokens for every task, making it less expensive.

Typical Use Cases

In this chapter we go through typical AI agent use cases for data engineering.

There are many of them. You can use them to educate yourself or your team, build production data pipelines, build data apps, and visualize your data in new innovative ways (usually HTML web pages with React and other JavaScript frameworks). But in general, the use cases fit into these approaches:

1. Start a new project from scratch example: Building a data landscape with more open source.
2. Extending an existing project or data warehouse: Adding new data pipelines.
3. Maintaining current setup: Update and verify it still works when changes come in.
4. Migration: Migrate from one database or tooling to the next.
5. Finding the Blind Spots: Two similar-sounding IDs might be wrongly used for a join, or missing data in a column that got missed in a nightly load, or anything in between. If agents can do these checks, that would be super beneficial. With more access to CLI, Model Context Layer, and deterministic tooling, these things are truly possible.

Below we go through extending and changing an existing warehouse with a change of column, and using Altimate Code to give us a Blast-radius assessment.

Showcases: Blast-Radius Example

A Blast-radius refers to the potential extent of damage. For example, before you knock down a wall in your house, you want to know if there’s plumbing behind it, electrical wiring within it, or if it’s holding up the floor above.

The same is true for a data warehouse or a data project with lots of ETL. For example, if a data engineer cleans up the table fct_orders by joining orders to order_items and summing order_total. It compiles, the dbt tests pass, nothing errors. But the join changes the grain, so any order with several line items now gets counted once per item, and revenue quietly inflates.

It’s best to know, before you rename a column or add a new join, the downstream (data that comes after the current task) dependencies to the dashboard — that’s what the blast-radius report does.

With Altimate Code we can achieve this. Before any change goes through, it maps out the full impact automatically and produces a detailed blast-radius report with what will break, what’s safe, what needs someone to sign off, and also performs the changes. Here is what this looks like:

Rename and Change Columns and Logic

As an example, in this prepared ecommerce repo with different DWH layers such as staging -> intermediate -> marts, I prompted this request to change unit from cent to dollars:

It recognized the dbt name and invoked dbt-analyze automatically:

It gave me a full Blast-radius report and the impact my changes would have on the project:

Including semantics only, to point out what’s safe and what’s not:

With a fixed order to address breaking changes, semantics and docs, and intentionally untouched:

Notice, I hadn’t said anything about blast analysis or using dbt-analyze — it did it on its own, ran dbt, and analyzed it deterministically.

This shows how Altimate Code looks behind the walls of data engineering, just like blast radius analysis.

If you want to see another example and a full blog post on Blast Radius, check out Blast Radius Analysis Using Altimate Code, and what Altimate Code did as in the video. Or Altimate provides many more examples and Showcase on their website such as Migrate SQL Server to Snowflake with dbt or showing how to resolve An Upstream Schema Changed.

[!example] Connect a model to Altimate
Make sure to connect to a model with /connect and choose an existing subscription with API credits, or any other subscription. I used opencode zen for my example, which includes e.g. Opus 4.8.

Correctness Over Confidence

I hope you got a better understanding of why AI agents can be genuinely useful, especially when provided with the right tools and applied with the right discipline.

You’ve also seen how deterministic tooling, purpose-built for data engineering and analytics problems, gets you both better correctness and better token economics than general-purpose agents alone.

Coming back to where we started: not every task needs a level-three agent. A quick chat-phase agent is fine for exploring a dataset or drafting a query you’ll review yourself. But the moment that output touches production or serious work, a dashboard, a nightly job, a number someone makes a decision on, you want the deterministic core underneath it, not just a model that sounds confident.

That’s the gap Altimate Code is built to close. It runs on deterministic functions purpose-built for DE workloads, it’s open-source via the OpenCode TUI, and for teams wanting more, there’s Altimate Studio — a paid, multi-agent platform with extras like warehouse cost optimization, dbt development acceleration, and migration tooling.

Check out Altimate Code, it’s free and open-source. Give them a star if you like them, and find more information on their docs and new website.

Full article published at Altimate.ai - written as part of my services