[[Writing]] has become my passion over time; I have built a whole company based on technical writing services (even though most say that’s impossible). After doing it for 11 years publicly (much longer privately), I still enjoy it every day. If I could have one dream, I’d probably imagine the book-author lifestyle, where I go to a faraway place or island and just write in my journal, articles from my vast notes system (~3 million words, or 2-3 books.

But also, I wanted to have a medium to express [[My Terminal Workflow with MacOS and Linux|my workflow]], discuss how others write, and learn even more. It’s truly an art form and a craft to write, at least it is to me. It’s not something you just sit down and do for 5 hours (as my upcoming guest liked to say). It’s essentially your life. You observe, you find things to write about, you get angry, you find a cool hack, you create a story. It’s all of it! And you can’t plan it.

Some days you write 2-3 articles/notes in parallel; other weeks you write none at all. And then the imposter syndrome. Everything you write is public. My writing process is usually:

1. Ohh this is so great, this is so interesting, I’m looking forward to sharing
2. Next day: oh what’s that, this is crap, why should I publish this?
3. Then the loss of motivation: the hard part. When you need to make it work, you change, but it’s not right. You change again, you get feedback, you change. And eventually you see the light and can say, OK, yes, this is good!

Writing really is an emotional rollercoaster, and it’s been that way with all the articles I have written. Less so with the notes on my second brain, there I’m less “storytelling”, but more like “curating facts”.

So long story short, yes, this is what my new show is all about: «Show Your Workflow». I hope you like it. Subscribe if you haven’t already to get the next one, if that is something that interests you.

---

Above written is the writer flow at a high level, and how I experience it most of the time. Here I share writing process over time as I have documented that over time. Each part is similar in a way, but also holds some different key pieces and determined how I wrote at that time. These are rather raw [[journaling|journals]], but maybe interesting to see.

My writing process as of 2025-10-23

I start thinking, I write, outline, and, very importantly, I stop. I sleep, I go into [[nature]], I just let it sit.

After coming back, I write some more. Then again, I talk to people and anyone about the topic, and I brainstorm with AI. And I write some more.

All happens in markdown ([[Obsidian]] for me), and I’m constantly changing titles, adding new ones, and reorganizing.

The flow does feel off. I start restructuring again. The key point for me is that when I begin merging related topics, sometimes similar, and putting the essential message further up.

Sometimes I write an intro, add some context, and include some relevant info. I’m adding more insights. And the most important one that I wanted to talk about is very far down.

Now that I’m at the point where the main content will be naturally moved up, I’m deleting or removing content. This is when it will start to feel cohesive. The reading flow starts to make sense. And from there, I just keep putting it together, making the reading flow perfectly.

Each chapter already has tons of notes, links, and insights, so finishing a first draft from here is usually easy and exciting.

Once I have a draft, I fix grammar with Claude Code and get feedback now, requesting very high-level feedback. Before I do another major rework, bring a great first draft. Go over 3-5 more times. I will notice how my changes are getting smaller and smaller until I know deep in my [[Gut Feeling|gut]] it’s ready.

A trick I learned—about deleting writing

It was always hard for me to cut out my hard-earned content. So I discovered a trick.

By just adding a # Take Out chapter at the end of each article, it tricks my brain into thinking: “it’s not deleted”, “I can get it back”, and this way it’s much easier to take out writing than to delete hard-earned hours on a paragraph.

My Writing from as of 2025-09-17

My process of writing is in two phases and the [[My Distraction-Free Typewriter (Micro Journal)|distraction-free typewriter]] helps me with the writing phase a lot in the first writing phase:

1. Outlining

• this is just exploring, listing (!! this is key)
• and write anything that comes to mind, no matter how good, or the other it fits in
• then I have my [[Second Brain]] with ExcaliBrain and Graph view with backlinks that give me interesting insight which notes might link or are related. I like Graph Analysis (Today [[Obsidian Smart Connections]]). This will give me many related ideas that the plugin finds via embedding, locally, without AI. Gives me ideas through the connections that I otherwise wouldn’t have.
• All of this usually happens on my laptop with [[Obsidian]].

2. Writing phase

• Nowadays, as I have the [[My Distraction-Free Typewriter (Micro Journal)]] device, I will commit that state to the private GitHub repo, and pull it on my micro journal. After that, I’m offline, only with these ideas and my backlinks, and I’m trying to figure it out.
  • The Micro Journal has allowed me to think more, take a break, and focus for hours on writing, instead of constant distraction writing at the computer
• A key here is also the [[Markdown]] format: I can simply move stuff around without losing formatting or anything else.
• sometimes I copy existing paragraphs from my [[Second Brain]] that I have written before.

Finalizing

3. After a while, I sit back and review.
4. Write more…
5. Get a mental breakdown as I think all I have written is so bad.
6. Next day, change the flow, use AI to help me with the reading structure, maybe heading names.
7. Finalize, edit, grammar check.
8. Publish.

That’s my workflow for writing an article. There are other [[Type of Notes]] that can be written and other [[Type of Notetakers]]. Which might help to organize your knowledge.

I think it’s important to have a place where you store your knowledge (notes from books, insights you read online, ideas, etc.) that can help you when you write too.

My Writing Process as of 2022-08-19

Write at least two crappy pages a day. The writing is in fact re-writing. A process of reviewing:

• first-round review for yourself, what you like
• the second one do it for your fans
• third for your critics, what they might come up with (bad) and improve

Then questions / proof-reading:

• look for confusing stuff, that should not be in there (if unsure or in doubt: take it out!)
• What are 10% that you would cut if you had to? Or what are 10% that you will keep for sure

And again, the [[Writing is Thinking]]!

I also like to write about things I don’t know, like Morgan Housel discussed in his interview with David Perell. It’s following my curiosity. First, I will learn something ([[Learn for Life]]), I will get [[Clarity]], and it’s interesting. This kind of flow or fun to write, will translate to the reader too.
according to David: laughter is the sound of comprehension. It’s when you get an idea, when it clicks or you make a connection. Happens to me when I find new insight while writing the second brain, I’ll smile.

My writing process initially documented

I always need to remind myself:

[[Writing is hard]]. It’s easy to start, hard to finish. You get stuck, you have a blockage, your creativity is low, you get interrupted, and you lose confidence in your writing. Reminding myself that the first 10 minutes are the hardest and envisioning the end product keeps me motivated.

Writing Phase

This is what brought me to writing. To release my thoughts ([[Brain Dump]]), make my brain accessible for more, and be creative. I like to brainstorm, change, delete, add, research, and do the thinking while writing. When writing on a computer, I can do it almost at the speed of my thinking; I do not lose thoughts as I would when writing on paper ([[Digital vs Paper]]). I can stop at any given time and come back, as all the thinking is written down. As well, I love the editing part. Making it better, finding correlations, and things I wouldn’t have expected before.

But before getting into the editing phase, during the initial phase, when drafting or coming up with the content itself, I try to get into distraction-free [[Deep Work|Flow]]. Long chunks of uninterrupted time, Cal Newport talks about [[Deep Work]] in his book.

Editing Phase

Usually, editing is when I learn the most, but at the same time, it’s also the most challenging step. I will get into a [[Writers Block]] and want to start a new thing instead. But by switching between [[Creativity vs Productivity]] or just going into nature with my thoughts and nothing else, usually one of them helps. Or if not, I leave it for a couple of days, weeks, or months and come back another time. Keeping the [[Ultradian Rhythm]] in mind also helps overcome the initial hardest ten minutes.

When I let the writing stay for a while, that typically leads to better quality because when I come back, I have new insight and might delete or change existing thoughts. The longer the wait, the more insightful that note will get. Therefore, I prefer to keep it slow. In the end, the reader will not mind if you spend one day or three months on an article; he appreciates it if the reader flow is good and he can learn. At least that is my goal as a writer: to give readers some learning.

Writing doesn’t always need to be about an article. You can just put some thought, [[Journaling]], or something you learned or find interesting, into a note. Save it for later. You mostly don’t need the thing you just knew at that moment. Something you find interesting today is doubtful to be of value right now. But maybe, in two years from now.

[!quote] David Perell

Writing is so painful when you write about something you do not care about and so blissful when you write with your [[Principles]] aligned. Writing from a blank page reveals the true things you deeply care about. YouTube with Paul Millerd

This last initial process was is shared as part of On Writing note.

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

💾

It’s hard to keep up with the AI evolution; new AI tools drop every week, but how are experienced practitioners actually using them? Most of us are overwhelmed and unsure about the many possibilities, yet we need to keep going and do our work. You might use AI agents all day long, parallelize them with AI Orchestrators, tmux, git worktree, and so on, using AI IDEs, but in the end, you still need to coordinate and understand what the agents produced, potentially test it, which makes it even harder to keep up.

Luckily, Maxime Beauchemin, the creator of Airflow and Superset and the person who defined what “data engineer” meant for a decade (more on him below), joins us to show how he uses agents and what he’s built for working with them. I tried to extract the patterns behind how he actually uses AI in his data work today. This is the fourth interview in ‘How to use AI with DE’.

In this article, we go into four parts: (1) How to balance quality with messy data warehouse work, and how to manage agents with Figma for agents. (2) We elaborate on the future of the context layer and the return to semantics, (3) how Okta for Agents is needed for security, and (4) how the future of agentic workloads can be done in teams, whose yap-to-ship ratio is best, and why Amdahl’s law still counts.

Introducing the Guest: #4 Maxime Beauchemin

Our guest in this interview Max Beauchemin, the creator of Airflow and Superset. He’s known as one of the OGs of defining how data engineering worked back in 2017, and founded Preset, the company behind Superset, and currently serves as its CEO.

He is heavily involved in the AI workflow, which is another reason I wanted to interview him for this series, but he has also been building in the space himself: Agor (Ag: AI agent + Or: orchestration), earlier tooling like claudette-cli¹, and db-agents, an experiment to embed agent context directly inside databases. We’ll get into it all.

Max and I talked about many things, among them how to use AI in data engineering, how security plays a role, how shared, secure, context-rich agent workspaces work within teams, and how he uses AI assistants to run his business and ease his life as a CEO.

Max is a true open-source enthusiast, and he wants open source to win. Everything we discuss here is somewhere on GitHub, which I have happily linked throughout the interview.

Figma for Agents: Visualizing Tasks and Jobs with Agor

Before we start using Figma for Agents, coordinating them on canvas, we need to ask why we need coordination and orchestration in the first place.

Balancing Quality with Quantity: Messy DWHs

That’s where we started, with the challenge of messy data warehouse environments that most people find themselves in. I asked how he balances quality and quantity, aiming for high quality.

Max says that the new models, Opus 4.5 or 4.6², are not making many errors anymore and are very clever when they get the right context as above with all the database schemas of tables and data types, and even querying it with MCP. He says they almost run in self-serve mode, but he still prefers that users know what they are doing and can either read the generated code or verify the generated numbers on a dashboard or chat results.

But the setup is critical. With these three prerequisites, the agents handle almost all queries really well:

1. You need some preparation claude.md/ agents.md
2. Access to SQL (e.g., execute dbt)
3. Access to MCP or CLI for BI tools (e.g., Superset supports sup!, a CLI to interact with Superset)

The only problem, and always has been, is the messy structure and sources that most organizations have, growing from an initial small project into a certain stage. There are always obscure tables or strings, timestamps not aligned, or hidden information that is not encoded in code or written down. Or there’s the hidden knowledge, like that a certain table shouldn’t be used anymore or has bad data, which is known to the people using it but might not be to agents.

The Canvas in Which Your Agents Can Run: Automate Most CEO-stuff

When he recently saw the power of agentic coding, Max went all in and has been building the Figma for agents ever since. Something he can use to collaborate with agents within his company, instead of everyone running the same prompts locally and needing to sync with each other manually. That’s when Agor was born.

Agor stands for Ag: agent and Or: for orchestration. As the creator of Airflow and CEO of a data company, he knows exactly how a tool needs to improve his workflow. He also called it:

The goal is to automate most of the automatable CEO-stuff

Building an Internal Knowledge Base: Shared Canvas

Agor was Max’s answer to “how he uses AI beyond a research tool”, but doing data modeling, writing data pipelines, even legal or HR roles he added later to Agor, so you can give company-wide roles to agents that can be fed with dedicated documents and context, and triggered by any employee internally. In contrast, others see the jobs and avoid asking the same questions, reusing the output for new queries—Andrej Karpathy’s concept of an LLM-maintained shared team wiki—which builds an internal knowledge base.

Initially, when we first chatted, Agor had already changed how he worked as a CEO, but since then, Agor has gone even further. Agor can replace high-level tasks while still being very hands-on by working closely with the code via git worktrees (branch cards in Agor’s UI) and verifying the code in the PRs it produces. Max also added OpenClaw-like features around memory and identity via dedicated Markdown files, such as AGENTS.md, SOUL.md, MEMORY.md, so that Agor’s agents can learn from recent runs and carry a purpose and clear instructions. This led to role-based agents called Assistants (use agor-assistant as a template to build your own).

Inspired by OpenClaw’s agent loop, Assistants became first-class citizens, persistent AI companions with memory, identity, skills, and scheduled tasks integrated into Agor’s canvas, multiplayer workflows, and reachable directly from Slack, for example. Additionally, Agor adds features beyond OpenClaw, such as better multi-user support, RBAC, one-click, full session inspection, and many more.

Asked about the goal of Agor, Max said:

The initial premise was to remove DevOps and set up time for other members of the company. Instead of people needing to connect all the MCPs or CLIs to add API keys, set permissions, or integrate with Slack, the prompt window with the needed context is there and ready to start.

[!note] OpenClaw, what is it? And how does Agor compare?
OpenClaw (formerly ClawdBot) is an open-source agent framework built around a persistent agent loop: a serialized cycle that turns a message into actions, using file-based identity (SOUL.md) and layered memory (MEMORY.md). Agor’s Assistants adopt this pattern and extend it with multiplayer boards, RBAC, and canvas-level orchestration.

Visual and Spatial Memory

When you do a lot of agent work, it’s really hard to keep up with all of it. That’s where Agor’s visual and spatial overview really helps and is unique in its approach.

It brings the local and private session to a server, where everybody can see and work together on the same queries, and use the insights from other results, as dashboards are built for. So instead of keeping output locally, others can source the artifacts generated by agents, stored as Artifacts within Agor, ready to use by anyone, with no integration or deployment needed.

Or how Agor tracks its own spending across sessions:

[!note] Check the full Webinar about the Anatomy of the Internal Data Agent at Preset.

Such as an assistant needing access to all pipelines’ metadata as illustrated data stack.

Or the data needs an analytics agent to support, such as self-serve, the data team, and extras such as memory, skills, documentation, etc:

Context Layer: Back to Semantic Layers?

Max also believes that we are going back to the Semantic layer, or using it for AI as agents benefit from structured information - helping with the data model and SQL part, to make sure it’s correct, especially with the needs of AI agents and the persistent challenge of providing trustworthy self-service analytics.

With the shift of semantics outside of the BI tool, versioned, testable, portable, it’s a chance for better integration between business domain experts and data engineers. His thinking has evolved since he wrote the article, and Max told me:

I see two different semantics: the semantic layer and the YAML. There are the hard constraints — not every area needs that strictness — and then the softer ones with Markdown and Agentic Skills, good for 80-90% but with no guarantees.

AGENTS.md For Databases: Markdown Stored Inside the Database Itself

Based on that idea, Max created an experiment to bring the AGENTS.md convention into the database. DB-AGENTS reserves a dedicated schema and table, _agents._agents, that holds agent-oriented documentation at different scopes (global, domain, schema, table, and even column). You write the docs locally as markdown files with YAML frontmatter, and a small CLI (dba) deterministically syncs them into that table — since databases don’t let you drop files into them, the table becomes the file. Agents then query it at session start the same way they’d read an AGENTS.md, making it a natural companion to INFORMATION_SCHEMA: one holds structure, the other holds meaning. Max calls it a “soft semantic layer”, which maps directly onto the hard-vs-soft split he described above. Check out the repo at db-agents.

With context being key for agents to understand what we humans know, Agor also added a context layer called knowledge. Agor Knowledge acts as a central place where humans and agents can store, organize, connect, and find the context that makes work compound over time, with Slack quickly becoming the main interface to many of the team’s agents.

How Do We Sandbox Agents for Safe Workflows (Okta for Agents)

Another big topic is security when agents have so much access to powerful CLIs, sometimes root access to systems or databases containing private keys, or just downloading random skills from the internet that may contain hidden secret messages.

Max coined the idea of Okta for Agents, which I found super interesting, and something I believe will become ever more important if we want to find a healthy way of working with agents in enterprises or with sensitive data. Okta for Agents means working around identity, scoped delegated permissions, leases, and audit logs.

When asked how he’s managing security, verifying what Agor or the agents are doing, Max responded:

I let the workers run in god mode³, but using dedicated environments/sandboxes, hooked to a dedicated git worktree repo, it can run autonomously and solve problems on an initial prompt, visualized in a shared canvas style.

I asked how he sees Okta for Agents being implemented. We desperately need it, he said, granting agents permissions like impersonation. Delegating the permission is an OAuth. With the roles, we can scope permissions strongly. E.g., the sales agent only has access to sales documents.

When asked at what level to integrate the Okta security layer, Max said it hasn’t been solved yet. Still, he sees it as the same question: whether we have 50 agents or 50 users who use a platform, both need a security layer.

Likewise, Max shared:

I trust agents the same way as I would an employee.

[!note] What does “Okta for Agents” mean in more details?
Okta is the identity layer companies put in front of their tools: it authenticates who you are, decides which systems you can open, and logs what you did. Max’s point is that agents need the same layer. E.g. Clawdbot, when he tried it, was effectively a DIY IAM manager for agents, config hell and all.

The twist is that it adds a whole new dimension to RBAC. It’s not just “the bot gets an email account” — it’s “the bot gets an email account, but can only read mine, and via MCP rather than as a real user.” Not so different from onboarding a human personal assistant, except this assistant can help with nearly everything, so the blast radius is much bigger.

Max’s own example: he saw a 1Password skill and immediately backed off then reconsidered, wondering whether the bot should have its own 1Password account with only safe credentials shared into it. Which is exactly the problem: you can be strict on paper, but the moment the agent has your email, Slack, and calendar, it can leak private things all day. (Full discussion at this post)

[!warning] Security is critical. Here are examples when it’s gone bad
How I Dropped Our Production Database and Now Pay 10% More for AWS, or another one, or when 13-hour AWS outage reportedly caused by Amazon’s own AI tools. Or also just hacks by getting attacked via GitHub PRs or How secret instructions injected into skills.

Declarative and Non-deterministic Outcomes?

Related to security is the deterministic, repeatable behavior of data sets with the same input. Agents are the opposite: probabilistic. I was curious to hear from Max, who initially defined the functional data engineering paradigm for deterministic and idempotent batch data processing, what he thinks about the non-deterministic outcomes of agents, specifically with large language models.

Max said, regarding declarative definitions, that he finds a claude.md is usually sufficient for most tasks that have a git repo, more context, and an issue or PR to work with, given the initial prompts come from users who know what they are doing.

Regarding reliability, Max thinks about using good methodology references. Agents get it and understand it. E.g., data modeling practices such as Kimball are still valid, or the approach shared by him with Entity-Centric Data Modeling (ECM), he says, and when prompted to model in those patterns, agents follow them well (either via research or provided).

The other part is that some non-dangerous work can have vibe data pipelines, and there’s no danger. And there are cognitive-depth tasks, such as a complex Spark cluster, where you can’t just debug quickly with large data sets.

Also, the field varies: not every area is getting agentic-piled as fast. E.g., platform demands go through the roof (see GitHub outages), so we have 10-20x the platform needs, but at the same time, the work is critical to be correct. So it depends.

Future of Agentic Workload, and Canvas Development in Teams

When asked about how Agor has changed how they at Preset develop products (if at all?), or made them more effective, Max said:

There are more agents than humans nowadays. Everyone has a Claude Max plan, and agents handle almost all code writing.

And on a personal level:

I haven’t written a function by hand for a long time, and I might not anymore — except when I feel nostalgic.

He also thinks that the Yap-to-Ship Ratio, a metric that he announced half-jokingly on LinkedIn, describing people’s velocity by just getting stuff done without involving others at every step, will be very low-yap for 10x engineers, as they solve the problem and ship a solution without much back and forth.

They deploy it somewhere for others to use, not only for human consumption, but as a solution or CLI that other agents can use to discover further and solve their problems. A high Yap-to-Ship ratio would mean lots of human interaction, which is the clear new bottleneck.

As human code review gets bottlenecked, I asked how he does the review. He said he uses Codex with sub-agents to review, ensuring everything is DRY (Don’t Repeat Yourself) and that all expected callbacks are made.

You can also ask the operator assistant agents if you are not sure whether an implementation is correct.

Amdahl’s Law: Can’t Go Faster if not End-to-end

One bottleneck is still Amdahl’s Law. We can speed up tooling by using extremely fast agents, but unless the end-to-end workload is sped up, we only increase by a 2-3x factor, not 10 or 100 as any one tool does. This also overlaps with Max’s Yap-to-Ship ratio: if PRs need the human in the loop to review many of them, the overall speed at which we build is not faster.

Another side effect is that it takes a lot of context switching. Max said he has ten active sessions in Agor. He is good at context switching (maybe also learned through recent Agor workflow? 🙂).

Predictions for 2026

It’s hard to predict the future with AI, but Max took a stab and shared his predictions for 2026 and categorized them into wired and tired:

Max’s AI Setup for Data Engineering Work and Managing His Company

We end this interview with Max’s setup for working with agents, since we didn’t have time to go into full details on the call. I’m sharing the one he shared four months ago. I’m sure it changes almost daily. Still, it helps us get a good overview of his software engineering stack for the team at Preset, as well as his personal local computer stack.

For software engineering and data engineering:

• Preset team instance of Agor behind VPN with a dozen boards, boards are mostly repo-oriented. Full Unix impersonation, backed by PostgreSQL.
• doing most of my work on board with the agor-openclaw framework, agent is pushing projects across a kanban-type layout: tons of new automation there. Agent checks on agents, prompts them, moves worktrees to “needs human review” zone if/when needed
• coding workflow is Opus 4.6 as a planner, Sonnet 4.5 / Opus 4.6 for most coding, Codex 5.3 as the reviewer (god it’s so good)
• data engineering stuff: dbt/airflow repo + Superset MCP, superset-sup
• “Command center” is agor-openclaw on Opus 4.6, monitors other agents, intricate HEARTBEAT.md with pseudocode to make a decision for coding project (worktree) on that board

And his new “Personal assistant” local instance of Agor (brand new/sensitive):

• Beefy Mac Studio at home
• connected to “productivity” tools (google-workspace-mcp)
• connected to Slack through a semi-homegrown skill — read-only for now, mostly to summarize activity
• connected to Notion MCP
• connected to “contracts” repo, where I sync with Google Drive for all Preset contracts
• connecting to Hubspot soon
• goal is to automate most of the automatable CEO-stuff as discussed above

Coming up

We’ve learned how to use Figma for agents with Agor and to collaboratively work as a team, using shared prompts and creating artifacts. We’ve seen how Okta for Agents is needed but really hard to implement, and how the future of AI is mostly about context and how to integrate it well. Plus, we learned how Max automates many tasks as a CEO with dedicated AI assistants and still produces low-level code with the same assistants for both his personal and company-wide needs.

I hope you enjoyed this fourth interview with Max. Huge thanks to Max for taking the time to speak with me (twice!) and for sharing his experience with all of us. Follow him on LinkedIn, GitHub, or on Preset Blog, where he shares his distilled thoughts on the ecosystem, and obviously, if you want to know more about Agor, check it out at Agor GitHub repo.

Max shares a lot of his ideas and thoughts online. Here are some further articles and interviews to read/watch:

• AI Enablement Engineer: The Highest-Leverage Role in Tech
• Check the Joe Reis interview where Max shares more on the discussion From Airflow to AI Agents: On Building Agor and Running a Company with AI Agents

More interviews are coming out, so please share feedback, questions you might want to ask, or your experience working with AI in the data space. We’re all in this together, figuring it all out.

---

Full article published at MotherDuck.com - written as part of my services