Welcome back to the Abstract! Here are the studies this week that lived and died by the spear, drew unequal lots, paged Fox Mulder, and peered at an alien moon.
First, scientists discover a mortal spear wound in a Neanderthal who lived 75,000 years ago, possibly opening a window into the origins of hominid violence. Then: disparities from beyond the grave, an unexplained population of near-Earth objects, and the word from Jupiter’s moon Ganymede.
Some 75,000 years ago, an adult Neanderthal died from a brutal strike to the chest by a sharpened wooden spear, and was subsequently laid to rest by their kin.
Eons later, in 2020, the remains of this individual were unearthed from deep in the floor of Shanidar Cave, an ancient Neanderthal hangout in what is now Iraqi Kurdistan. The bones of the deceased, named Shanidar A, preserve the spear puncture in the shoulder blade, suggesting that they were impaled with the weapon through the heart to their back.
Shanidar A—whose sex is unknown—also sustained a brutal face injury that had partially healed at their time of death, suggesting that this individual had been cared for while they recovered from this non-fatal wound. The body appears to have been carefully interred in a small area of the cave that contained other Neanderthal remains, implying it was an intentional burial ground.
The entrance to Shanidar Cave. Image: Osama Shukir Muhammed Amin FRCP(Glasg)
“Shanidar A’s shoulder injury appears to have been fatal, but the apparent placement of the body in a location that subsequently became part of a tight cluster of Neanderthal individuals, who were likely intentionally deposited within living memory of each other, strongly suggests a level of care during and immediately after death,” said researchers led by Emma Pomeroy of the University of Cambridge.
Taken together, the discovery of Shanidar A adds to the evidence that Neanderthals nurtured each other in life and performed symbolic funerary rituals after death. These bones may also represent one of the earliest direct examples of interpersonal violence in any hominin, the term for humans and their close relatives, assuming that the Shanidar A was intentionally slain.
However, it’s hard to distinguish whether the death blow was an attack from another Neanderthal—or perhaps a member of our own species Homo sapiens—or if it was an accidental injury sustained while hunting.
Shanidar Cave was occupied by Neanderthals for at least 50,000 years leading up to their mysterious extinction. While we will never know the complexity of their stories, our human cousins were clearly capable of compassion, care, and respect for the dead.
For the past several years, researchers led by astronomer Beatriz Villarroel have discovered strange “transients”—unexplained objects or phenomena in space—in historical photographic plates from before the launch of the first spacecraft, Sputnik, in 1957. The nature of these transients has become hotly debated among astronomers, with theories ranging from instrumentation defects to lurking alien spaceships.
In a recent study, the team re-analyzed the transients with machine learning, an effort reinforced previous findings that “transient numbers diminish significantly in Earth’s shadow”—hinting they are real objects that reflect less sunlight in our planet’s shade—and that they are more likely within one day of atmospheric nuclear tests during the mid-20th century.
The left image presents examples of three transients (green circles) identified on the POSS-I red plate taken on August 11, 1953. The image at right is the same region in a POSS-II red image taken on September 3, 1989. The yellow circle indicates a plate defect on the left image. Bruehl, Stephen et al.
“If one accepts that both key findings in this study represent real phenomena…we are left with a very small set of provocative hypotheses that would be considered highly implausible by most,” said researchers led by Stephen Bruehl of Vanderbilt University and co-authored by Villarroel.
“For example, is it possible that unknown to the public there were multiple launches of artificial satellites long before Sputnik with some launches timed to coincide with U.S. nuclear tests? Or rather, do the current findings represent detection of a non-human technosignature? Finally, might our results represent detection of an entirely new yet unknown physical phenomenon?”
“Additional research on this topic is clearly warranted,” the team concluded.
Hard agree. When a mystery’s leading explanations include 1) secret satellites that upend spaceflight history, 2) ALIENS (!!!), and 3) exotic novel phenomena, it’s fair to keep digging for answers. My own preference is that it turns out to be some unholy combination of all three, but for now, we will have to just keep literally watching this space.
Let’s keep playing in haunted graveyards—it's spooky season, after all. This time, we’ll take a long walk through about 2,300 ancient burials in the Central Plain of China. In a new study, scientists probed the emergence of wealth inequalities in these ancient communities by studying this huge sample dating from 9,000 to 3,500 years ago.
The team assessed the relative wealth of the deceased individuals with a number of proxies, including the value of grave goods buried with them, their physical health, and chemical clues about the quality of their diets, which can be preserved in bones.
The results suggest that inequality was present in many of these communities long before the rise of an official state hierarchy, and that “no single pathway led to persistent inequality across early centers of social complexity,” according to the study. However, one clear correlation that emerged in the analysis was the association between violence and inequality, with the former exacerbating the latter.
Geographic setting and archaeological sites in the Central Plain. Archaeological site locations were compiled and mapped by the authors based on published excavation reports. Image: Li, Dongdong et al
“In the Central Plain…the emergence of violence signals a structural transformation” that suggests that “violence contributes to the amplification and structuring of inequality, although reverse or reciprocal dynamics cannot be excluded,” said researchers led by Dongdong Li of the Chinese Academy of Social Sciences.
While the rise of organized states and kingdoms can accelerate wealth stratification, this study proves that humans are capable of pioneering local homegrown inequalities in myriad ways.
We’ll close with an update from Ganymede, the largest moon in the solar system, which orbits Jupiter. As the only moon with its own magnetic field and a candidate in the search for alien life due to its subsurface ocean, Ganymede has long fascinated scientists.
Now, a team has compiled comprehensive observations of Ganymede’s surface terrain using observations from a range of advanced telescopes, including the Very Large Telescope in Chile and the James Webb Space Telescope. The aim of the study, in part, is to prepare for two spacecraft currently en route to Jupiter—the European JUICE mission, and NASA’s Europa Clipper—which will arrive in the early 2030s.
Ganymede being Ganymede. Image: Juno Mission – NASA/JPL-Caltech/SwRI/MSSS/Kevin M. Gill
Ganymede “remains a cornerstone object of planetary science due to its geological complexity, internal magnetic field, and potential astrobiological interest based on the presence of a subsurface conducting ocean,” said researchers led by Clement Royer of Paris-Saclay University. “Its surface morphology…argues for a history of significant endogenic activity, possibly including plate-like tectonics and cryovolcanism.”
Indeed, one of Ganymede’s most interesting surface features is a mysterious “darkening agent" that extends over much of the moon’s equatorial regions. The composition of the darkening agent “remains unresolved,” according to the study, though likely guesses include carbon-rich graphite or iron-rich magnetite.
The team anticipates that the Jupiter-bound spacecraft “will provide an unprecedented view of the surfaces of the Jovian moons” that will reveal many of their secrets. At least that’s one thing we can look forward to from the 2030s.
This is Behind the Blog, where we share our behind-the-scenes thoughts about how a few of our top stories of the week came together. This week, we discuss a new pod setup, video games, and Elon's spending habits.
JASON: Exciting news: I am building a podcast studio / riso printing room / place to film videos in. I believe all of my colleagues and many people in my life have been laughing at me about this, but this also may just be the narrative I’ve created for myself to inspire me to prove the haters wrong. Since the beginning of 404 Media, I’ve worked out of my garage, but feel like I’ve been going a bit crazy sitting mostly in the same room since essentially the start of the pandemic. And so I found not a coworking space but a very affordable windowless box that I shall turn into a content palace.
I’m just going to try it for a few months to see if it makes sense or if I should return to working from the couch, but I’m quite excited about building it out. I only got it a few days ago and so at the moment it’s largely a bunch of boxes and random furniture I no longer had space at my apartment for. It’s got a drop ceiling with fluorescent lighting and landlord special laminate floors, but I also can do whatever I want to it, so I will likely do some sort of design on the walls, throw a rug down, whatever.
We start this week with Joseph’s article about Raven, previously known as Geospy. This is an AI tool that can geolocate photos, and now says it can pinpoint a specific house based just on one photo taken inside. After the break, Jason explains why a judge ruled a particular Flock search as unconstitutional. In the subscribers-only section, we talk about the wild AI video that has derailed a manslaughter case.
Listen to the weekly podcast on Apple Podcasts,Spotify, or YouTube. Become a paid subscriber for access to this episode's bonus content and to power our journalism. If you become a paid subscriber, check your inbox for an email from our podcast host Transistor for a link to the subscribers-only version! You can also add that subscribers feed to your podcast app of choice and never miss an episode that way. The email should also contain the subscribers-only link for the extended video version too. It will also be in the show notes in your podcast player.
Police arrested a Chicago area high school maintenance worker who attempted to meet a child for sex after investigators posed as a 15-year-old boy using AI generated images.
Christopher Thomas Mertens started talking to who he thought was a child through a dating app in September. According to an affidavit written by the department’s FBI Task Force Officer McKailey Scott, the investigation was done by the Livonia Undercover Resistance to Exploitation Unit (LURE), which identifies and prosecutes child sex offenders. Scott set up an account on the queer-focused dating app Taimi with “an AI generated image of a teenage-appearing male with the name ‘Caleb,’ and no age listed on the profile.
Scott wrote that an unknown user messaged “Caleb,” and they moved to Whatsapp. After a few minutes of talking, the man asked Caleb for their age, and when Scott — posing as Caleb — said he was 15, the man said “Im interested” and “Im hard now.”
The investigator used a combination of tools, including checking the man’s phone number using Thomson Reuters CLEAR and running images of his face through Clearview AI to identify him as Mertens. For days, Caleb and Mertens exchanged sexually explicit text messages and Mertens made plans to drive to meet Caleb on his day off, believing Caleb’s (non-existent) parents weren’t home. Cops used Flock to look up and track Mertens’ car and see that he was driving to the meetup spot, but Mertens was also texting Caleb the entire time to let him know he was on the way and when he arrived. When he got out of the car, Livonia police took him into custody.
Two hours after being booked, Mertens told jail attendants that he wanted to speak to Scott. When Scott got to the jail, Mertens admitted everything, including that he works at a high school in Illinois. The complaint doesn’t mention which high school.
Investigators have been generating images of children for years. One of the earliest examples of these is Sweetie, a computer generated child developed by Dutch children’s rights organization Terre des Hommes in 2013 that the organization used to talk to men online. Today, police departments at the U.S. border use AI-generated online personas designed to interact with and monitor protestors and activists, according to documents obtained by 404 Media.
In August, a 24-year-old Connecticut man was arrested after soliciting an undercover police officer who used what was reported as “age regression technology” to appear as a 14 year old in videos. He was charged with enticing a minor and criminal attempt to commit risk of injury to a child.
In 2025, a 45-year-old man was arrested after investigators used AI generated images of children to catch him distributing child sexual abuse material. He was recently sentenced to 12 years in federal prison.
And in 2024, the state of New Mexico sued Snapchat after investigators set up a fake account, including AI generated images of children, that the platform allegedly recommended to “dangerous accounts, including ones named ‘child.rape’ and ‘pedo_lover10,’ in addition to others that are even more explicit,” according to the New Mexico Department of Justice.
AI generated child sexual abuse material is a rapidly growing problem for law enforcement. The National Center for Missing and Exploited Children has reported a massive rise in generative AI child abuse material, with criminals using it to extort and entice children, and children using it to bully and harrass other children.
Livonia police did not respond to a request for comment.
In the most complete analysis yet, Protect Our Care has issued a devastating report card on the failure of the Trump administration to keep its MAHA promises.
For over a year, Donald Trump and his Health Secretary Robert F. Kennedy Jr. promised the Make America Healthy Again movement that this administration would finally take on Big Food, Big Ag, and Big Chemical. Instead, they have delivered a steady diet of broken promises, corporate quid pro quos, and foodborne illnesses….But a year into his [RFK Jr’s] tenure, the receipts tell a different story: cancer-linked chemicals back on the market, food safety agencies gutted, families paying more for groceries, and even the movement’s founders wondering whether they’ve been put out to political pasture.
The report details and documents broken promises about chemicals, pesticides, and corporate capture, especially to women.
Here is one item as an example:
Kennedy Stays Silent While EPA Betrays MAHA on Toxic Chemicals. While MAHA leaders have spoken out about air and water pollutants, Trump’s EPA has now shut down plans to release a MAHA agenda and the NIH has stopped accepting submissions for its flagship journal on environmental health and yanked research grants.. EPA scientists were pressured to alter safety reviews of chemicals in household cleaners and cosmetics to make risks appear less serious than the data shows — whitewashing health risks of the products MAHA’s core constituency was promised would be regulated. At a chemicals summit, Kennedy praised an Oregon State researcher’s microplastics study from the podium — unaware, or unbothered, that an hour earlier she’d learned EPA had just terminated the grant funding that research.
Kennedy has stayed similarly silent as EPA rolls back the very mercury rules he spent decades fighting for. As an environmental attorney, he sued over mercury pollution and personally developed “word retrieval and memory problems” from mercury-tainted fish. EPA announced its rollback of mercury emissions rules the day after the MAHA Commission, which Kennedy and Zeldin both sit on, held its first meeting. A recent report from Climate Power and Protect Our Care details how Trump has eliminated 31 pollution protections that, together, were set to deliver $275 billion in benefits per year and prevent 30,000 deaths annually.
Take a look at the report. The list is long, alas.
Scientists have made a breakthrough toward the mind-bending goal of creating computers made of brains by demonstrating that human neural tissue can classify letters of the Braille alphabet, according to a study published on Friday in Frontiers in Artificial Intelligence.
A team transmitted electrical signals corresponding to Braille letters to human brain organoids, which are living neural tissues derived from stem cells. The results revealed that a single organoid, consisting of about 10,000 neurons, could classify the letters with about 61 percent accuracy, while teams of three organoids reached 83 percent accuracy.
“The human brain can perform incredibly complex information processing while consuming relatively little energy, and artificial neural networks were originally inspired by biological nervous systems,” said Tianyi Liu, a PhD candidate at Bristol University who led the new study, in an email to 404 Media.
“If we've been trying to imitate biological neural networks all along, why not try using real biological neural networks directly?” she continued. “What kind of information processing capabilities might these tiny neural organoids have? Could they show some form of learning or adaptation? These are all questions we're very interested in.”
The new study falls under the emerging field of organoid intelligence, which aims to use organoids as biological “wetware” for computing applications. Previous studies have established that organoids are capable of recognizing speech Japanese vowels or even playing the classic game Pong.
Since Liu and her team specialize in how robots sense and respond to touch, they homed in on Braille as a way to convert tactile information into electrical stimuli that an organoid could receive, process, and classify.
In their experimental setup, the researchers used a tactile sensor to scan data from the 26 letters of the Braille alphabet, which is built from arrangements of raised dots. This data was converted into electrical stimulation patterns that were fed to the organoids, which are smaller than a millimeter in scale and positioned on arrays that keep them alive.
“Our original goal wasn't to prove that organoids could read Braille like humans, or that they were already ‘intelligent’ enough to understand Braille,” she added. “Rather, we wanted to use this task to explore whether real biological neural networks could participate in processing information from robotic tactile sensors.”
The tactile sensor used to scan Braille letters. Image: Dr Benjamin Ward-Cherrier.
In other words, the researchers aimed to discover if these stimulated inputs could be decoded by the organoids and returned back in discernable responses corresponding to the Braille letters. The results revealed that a single organoid could accurately classify the letters nearly two thirds of the time, while an "ensemble" of three organoids performed even better, with an accuracy of 83 percent.
“We still don't know the exact mechanism behind this improvement,” Liu said. “Our hypothesis is that different organoids respond to the same stimulation in similar, but not identical, ways.”
She added that “you can think of it as three people looking at the same object. Although they're all looking at the same thing, each person might notice different details. Some features might not be obvious to one person but could be quite easy for another to spot. By combining their observations, you might be able to make a more accurate and reliable judgement.”
🌘
Subscribe to 404 Media to get The Abstract, our newsletter about the most exciting and mind-boggling science news and studies of the week.
The results are undeniably trippy, conjuring up the classic scifi trope of a hyperintelligent brain-in-a-vat. But Liu emphasized that this was an open-loop experiment, meaning that the organoids only received initial inputs without getting new feedback about their performance on the tasks. For this reason, the study does not prove that the organoids actually learned Braille, or were altered in any permanent way, just that they could classify Braille letters after receiving the electrical inputs.
“In the future, if we can build closed-loop systems where neural activity from the organoids actually participates in a robot's perception, decision making, and control, while the results of the robot's interactions with its environment are fed back to the organoids, we could begin to investigate whether these biological neural networks can adapt to new tasks through ongoing interaction,” she said.
Liu also noted that while there are many potential advantages to organoid intelligence, there are plenty of limitations to biological computing systems. For example, organoids cannot be standardized, leading to natural variations that can complicate results, plus they have to be kept alive in cultures that are more difficult to maintain than electronic devices.
“Compared with established computing chips, organoid systems still have a long way to go in terms of reliability, reproducibility, and scalability,” Liu said.
Even so, organoid intelligence “gives us the opportunity to explore a way of computing that's very different from traditional computers,” she concluded.
Last week, Secretary of War Pete Hegseth announced the Pentagon would livestream the execution of convicted killer Nidal Hasan. Today, Trump said the execution would move forward but that it wouldn’t be livestreamed. But the propaganda damage is already done. Experts who watch terrorist spaces online told 404 Media that Hasan is already turning up in memes, videos, and recruitment material for groups like Al Qaeda.
Hasan—a former US Army Major who had once attempted to contact Al Qaeda—killed 13 people and injured 32 others at Fort Hood in 2009. He was shot and captured at the scene and later sentenced to death. Hegseth’s plan to livestream Hasan’s execution and the furor around it brought Hasan and his crimes back into the spotlight 17 years later. Hasan has mostly been a forgotten figure in Islamist and extremist circles, but is now once again a fixture of their propaganda.
“It's already started,” Moustafa Ayad, research chair in global Islamism and counter-terrorism at the Institute for Strategic Dialogue, told 404 Media. “We're already seeing edited videos in support of Nidal Hasan, and they're specifically based around how he's a martyr and that his legacy lives on.”
In one image Ayad shared with 404 Media, Al-Qaeda in the Arabian Peninsula (AQAP) referred to Hasan as a “heroic mujahid brother” then quoted Osama Bin Laden. “The day America makes this decision, it will have made a decision to execute whoever among you falls captive.”
According to Ayad, an Al Qaeda affiliated propaganda group made Hasan the feature of its newsletter last Friday. “So it's being viewed in that sense already in these ecosystems as he is a hero, and it's going to be used for recruitment and potentially galvanizing inspiration,” Ayad said. “We're talking about a guy who hasn't really been referenced since 2017 really or that much at all. So it has reignited conversations around his attack, how quote unquote glorious it was, and is being used as a means to to recruit people in some shape or form, and will continue to do so.
Ayad also described "rampage" edits of Hasan that are floating around extremists circles online that show footage of Hasan dancing over a number that represents how many people he killed. “You have fed the wolves,” he said. “They now have a thing they can latch on to.” In one TikTok edit viewed by 404 Media, a picture of Hasan with laser eyes throbbed to the beat of electronic music.
He added that he thinks the memes and videos are being made by younger people. “I don't think an established guy who remembers the Nadal Hassan attack is going to then go: ‘You know what makes sense? Putting out a dancing Nidal Hasan over his kill count.’ I don't think some 35-year-old dude is doing that,” he said. “You have a younger generation that is now attuned to what he did.”
Livestreaming the public execution would have given propagandists more to work with. Online nihilists, gore groups, and Islamists often use footage of death and violence to lure in young people. “Dead ideologues are continually being revered, lionized in ecosystems in support of groups like Al Qaeda, the Islamic State, or any terrorist group,” Ayad said. “If you look at the saint ecosystems around accelerationism, it's a similar thing in which these individuals are the vanguard of jihad.”
Mass shooters and ideological killers that receive widespread media attention become objects of fascination online. They enter a cannon of sainted killers, totems that inspire others to commit the same acts. Some researchers have suggested news agencies should report a shooter’s crimes but refrain from naming and showing them, thus depriving them of notoriety and stemming the social contagion.
Hasan killed people almost twenty years ago and he was destined to fall out of memory until Hegseth attempted to make a spectacle of his death. “We’re already seeing it being incorporated into talking points, we’re past that,” Ayad said.
“He is still going to be labeled a martyr," Ayad added. “Will it be a public spectacle? Not anymore [...] but terror groups and their supporters took advantage of the news to highlight just how much of a contribution Nidal Hasan made to their respective movements.”
I was interested to see this chart from the USDA: Chart of Note: U.S. food system required greater share of national water use compared to employment or energy use:Water, labor, and energy are resources relied upon throughout the U.S. food system. The food system comprehensively required more than a quarter of total U.S. freshwater use each year from 2010 to 2023….In 2023, the U.S. food system relied on 27.7 trillion gallons of freshwater, or 29.3 percent of total freshwater withdrawals.
There's a new digital pet in town - a cute little marshmallow of a robot that can change its cartoony expressions to match your mood. But if you or your kids talk to "Eiliko" over Wi-Fi, it'll phone home to China. Its AI companion can store literally everything you tell it on Chinese servers, where hackers can easily gain access or even hijack its personality. And the company that made it, Energize Lab, doesn't seem to care.
I bought an Eiliko from Amazon on August 29th. During a 10-minute conversation on August 30th, I asked the little bot: "Do you ever tell anybody else what I'm saying?"
Big Walk's co-op multiplayer walking adventure has proved popular since launching in August, but Xbox players have been excluded from the fun - and now the game's creators have explained why. Developer House House said in a blog post that it decided not to release Big Walk on Xbox consoles in accordance with the Palestinian-led BDS (Boycott, Divestment, Sanctions) movement, which has called to boycott Microsoft, including products under the company's gaming division.
In this episode, we dive into how much models actually matter: more like a preference, like whether you use an iPhone or Android, than their capabilities. We talk about bare-metal disks when running Kubernetes, then move to agents and how data engineering becomes like being an emergency pilot in a plane, and how we automate away the boring work with no more database migrations or unit tests.
We go into spiritual territory, where he feels in his gut whether a pipeline will work. Then we finish with where every data engineer or software engineer eventually wants to end up, as the field is evolving so fast: on a farm with beautiful flowers and no screen. This is interview #5. I hope you enjoy it.
This article is structured into four parts: (1) how Kubernetes and disk abstractions are key for his work, (2) how he works with the models and agents, and how much the model actually matters, (3) what tooling he uses and how agents ease his life at work, and (4) whether data engineering is automated away with agents.
Introducing the Guest: #5 Josh Wills
Josh Wills is a software and data engineer with 25 years of experience. At Google, he worked on the ad auction system and led the analytics infrastructure behind Google+. As head of data science at Cloudera, he founded the Apache Crunch project and co-authored O’Reilly’s Advanced Analytics with Spark, and he went on to build and lead Slack’s data engineering team. He’s also the original creator of the dbt-duckdb adapter.
Since 2024, Josh has been a Member of Technical Staff at DatologyAI, which helps teams train better models, faster and smaller. There he builds tools that automatically select the best data for training deep learning models, along with the distributed pipelines that process it at trillion-token pretraining scale. We go into more detail on what it takes to work with datasets that gigantic.
[!tip] Fun Fact
He basically retired in 2022, and “the AI stuff got him back” 😉
DuckDB Yes, but also Kubernetes Disk Abstractions
As many of you might know, Josh was the initial creator of dbt-duckdb, so I asked him if he still uses dbt and, more importantly, how much DuckDB.
His answer was that everyone uses it: “Everybody uses DuckDB in some capacity. I feel like anyone who knows about DuckDB uses DuckDB for something. Everyone has their own specific use case for it”. He is using it less for his specific pipelines at work, as they have huge datasets, text data, not classical datasets: e.g., 30 TB is tiny. He uses Spark or Ray. He does much of the processing on text and multimodal data (time series, images, etc.), not much classical relational data in his day-to-day.
The use case for DuckDB he uses a lot is reading Parquet metadata that sits on S3, as the above data pipelines “throw off enormous amounts of Parquet metadata, which is still substantial but much smaller, on the order of 100 GBs”. Information about runs, when a dataset was last used to train a model, how much he pays to store it in S3 for indefinite periods. He usually asks his agents to read that on his dev box on EC2, which he runs these commands on, to cut down on transfer times, just for speed.
[!note] DuckDB got integrated into dbt v2
Interestingly, dbt-duckdb just got an update. The dbt adapter for DuckDB received its first pull request on August 27, 2021, and in the meantime has 1.4k stars on GitHub. Since then, dbt users have been able to install one Python package (dbt-duckdb, via pip), point it at a file (a local DuckDB database), and have a working project (models building into tables and views), without having to sign up for (and pay for) servers or warehouses. With the latest release, DuckDB now runs inside dbt with the dbt Fusion engine. See DuckDB Now Ships inside dbt v2.
“We Lost a Whole Generation” of Data Infra People, Retired 2022, AI Pulled Him Back
When talking about what has changed since he worked at Google as a staff engineer (2007–2011) and led data engineering at Slack (2015–2019), and how data engineering workloads differ today with AI, he said Snowflake is great. Still, we do not have many people anymore who know how the data is laid out on disk.
But not for his size of workload. Josh works with such big data that we normies can barely fathom, and that’s why he uses lots of Kubernetes jobs and workloads. He likes Apache Celeborn (Pronunciation: spelled Celeborn, pronounced “Keleborn”), which abstracts the disks (not needing to know where they physically come from), because in Kubernetes you still do that manually, as Kubernetes is not really cloud-abstracted, he says:
Kubernetes is fantastic in that it’s in theory abstracted away from an individual cloud, but in practice it’s not, because where does your disk come from? If you’re on AWS or GCP you can use an EBS-like service, but if you’re running bare metal, that doesn’t exist the same way.
Celeborn has something called push-based shuffle write, where mapper nodes skip local disk storage, ideal for cloud-native architectures, which lets you separate storage and compute, so
The Spark jobs don’t need to worry about where their disk comes from anymore, from a shuffle perspective. There’s the shuffle service, no matter how much data you’re going to shuffle. You have your local disk, and then you have your shuffle service for all your other disk needs.
[!note] Nobody Remembers How to Run a Spark Cluster
Josh says the hardest thing is:
“That there are relatively few people in places that need to think about this stuff anymore, at all. The whole data community during the late teens kind of went Snowflake happy. And I used Snowflake at my last company, it’s a fantastic product. But we have sort of lost a whole generation. There’s not many people left anymore who still remember how to run a large, effectively open source, Spark cluster themselves.”
How Much Does the Model Matter to You?
When asked about the models and how much they matter, Josh said for him it’s like the iMessage blue bubbles vs. green bubbles. Android phones are fantastic, but he wants the blue bubbles. And Josh’s prediction is that all the models will become competent, so the real question becomes:
Above, Josh explains how he disliked the Opus 5 model, and continues to do so. In this interview, he added that it’s also the way it communicates. An interesting insight, he said, as someone who works on building models all day:
the fact that Opus 5 is so bad, it deeply indicates to me that we literally have no idea what we’re doing at all when it comes to training these models.
The good thing: changing might be easier than ever1, and he disliked the previous 5.0 so much that he changed his workflow and switched to OpenAI. He commented that:
with no drop in productivity, it’s kind of insane.
[!tip]
One trick I learned: use the caveman skill, and load it at the beginning. I use only the one MD file of it, and then every model sounds the same, and even uses fewer tokens and is less verbose. It also needs less brainpower to read through.
How to Code with Agents These Days
Next, we talked about working with agents. He jokingly came back to a Tweet where he wants to start a reinforcement learning (RL) environments company, where he does the teaching:
Josh says:
[..] a joke that I want to start my own RL environments company where it’s just me teaching the models how not to write absolutely garbage data pipelines. Because I use agents to write all of my code, with very, very few exceptions. And yet I still find myself deeply unhappy with the data pipelines it writes.
[!tip] His favorite utility is still htop
He said back in 2022 that he loves htop. Asked about his favorite tools, he says that htop is fantastic, and he still uses htop for a lot of stuff he does. In case you don’t know, you see all processes and CPUs of your machine at a glance. See htop - an interactive process viewer.
How to Get Fast Feedback Loops with Agents?
Josh says AI wins where there’s a large training model and fast responses, because if it’s not in the training data, it’s slow and it needs to do a lot of stuff. He says: “Any sort of environment in which the models have highly reliable training data with fast feedback mechanisms, they’re just going to dominate. Absolutely going to dominate. You see this with math, you see this with front-end engineering, any RL environment where there is a fast feedback mechanism.”
And that’s exactly what’s missing in large-scale data pipelines today:
What’s tricky about data pipelines, at least at a large scale, is that you don’t actually have that fast feedback mechanism available to you, because running a giant pipeline can take a long time and a lot of computers.
Fast feedback loops are also what Josh was already referring to almost four years ago, before any of the AI hype. He liked fast feedback loops, and quoted Erik Bernhardsson:
[!note]
Interestingly, today Jev came out to a lot of praise online, and it’s based on Reinforcement Learning For Calibrated Decisions (RLCD), which is basically what Josh wanted.
Feeling the Tuples Flowing through the Pipelines, a Gut Feeling
Josh, with his years of experience, has built a gut feeling:
Through years and years of suffering and writing gigantic pipelines, I can look at a pipeline and it kind of feels like something to me. I can feel the tuples flowing through it, because I have a mental model of how this is going to go.
He explains how SQL databases have a great fast feedback mechanism with their query explain plans. They tell you if it’s a good or a bad query. But for a Spark job, it’s not yet exactly that.2
Josh describes an aesthetic aspect of data pipelines:
I have, weirdly, a kind of aesthetic sensibility when it comes to data pipelines. A data pipeline can be beautiful to me. It can be elegant, well designed, thoughtful. That makes me want to look, in the way other people would visit an art museum, or read math proofs because they find them beautiful.
That’s where he still sees a lot of “ugliness” in what the agents write. They were not yet beautiful when we talked a couple of weeks back. Josh said, “Some of the best programmers I’ve worked with were musicians by training”. And he continues:
One distinguished engineer at Google, his master’s degree is in the violin, and he was the best coder I think I’d ever seen, because his code was just like music. It was beautiful.
I have a similar affinity for writing text, and good writing has a musical quality to it that certain people can hear. You develop an ear for good writing after a while. It sounds like something in your head, like poetry or prose. Josh says that “This is something we have not figured out how to RL in any kind of way”.
The Cost Aspect, Literally Lighting Money on Fire
When talking about cost, Josh says that:
“the models might waste time on the most pedantic, stupid checks for errors that are simply not possible given my knowledge of the structure of the upstream data sources.” “Literally lighting money on fire”
He continues that the pipelines are huge and compute costs a lot. And models might write a pipeline for average-sized data, but not have the knowledge or the big training data on “gigantic pipelines”: “It will work, it’s correct. It’s just stupidly cost inefficient”.
He has seen this problem at companies getting rid of their data teams, as LLMs could vibe code a job in Databricks. He said it like this: “You can easily vibe code a job in Databricks that costs like two hundred thousand dollars a day to run, that a competent data engineer, or at least one who supervised it, would never have written.”
And it’s not the model’s fault, but a function of the training data that has not investigated or run a lot of “gigantic data pipelines for multiple hours across hundreds of machines”.
That’s just not the way the reinforcement learning environments are structured right now:
It’s this fast feedback loop: tiny data, is it correct? Tiny front end, does it render the CSS correctly?
Josh makes a great science analogy:
People talk a lot about how science has the same problem. Labs, biology, physics, these are all real things where you literally have to wait for the culture to grow. Data engineering has a lightweight version of this. We are not as immediately amenable to this kind of feedback style right now.
He found the models to be fantastic at SQL back when we held the interview. “They’re really, really good,” he said. But for his kind of work (the raw Ray pipeline, the Spark pipeline), there is no explain plan. You’re still building the thing from scratch, as he said.
The Job of ‘Emergency Exception Handler’
He mentions that in his job, because the models haven’t learned everything yet, he is basically like a pilot who usually flies on autopilot, but in case of emergency, he is there to fix it or take over.
What Is the Tooling He Uses & How Agents Ease His Life at Work
As initially pointed out, he disliked Opus and switched to Codex, so now he is using Codex all the time, but he used Claude for a long time.
When asked how he works or reviews code, he said: mostly giving feedback, like a staff engineer does. His life as a senior technical IC has been trending towards that for a very long time, he says. He primarily gives feedback. That’s his primary job. Interviewing people, reviewing design documents, and less hands-on, with very few exceptions.
He likes to work on high-demand, high-urgency work that forces him to create. This is back to the ’emergency exception handler':
If there’s a broken-ass data pipeline and it has to run right now, because a customer has a training job coming up and the data has to be ready for it. That’s the primary value I provide.
What Josh Likes about Agents: No Feelings and the Caffeine Dial
Josh also revealed that he likes working with agents because agents have no feelings. He can give agents feedback on demand, force things to be created, he says:
I don’t have to be very nice to the agents.
He can just say things:
I don’t have to care about their feelings or their career development, because they’re going to cease to exist when I’m done with this coding session. I can be like: this is bad and you should feel bad. Ctrl-C, you’re dead, I’m going to start over with someone else.
As someone who calls himself “a fairly mediocre manager, because I don’t generally like caring about these things”, this works well for him 🙂3.
The other interesting thing he says:
I can dial up and down the work on demand as I see fit. If I’m in a moment where I need five agents going, I can do it. If it gets to be too much, I dial it back. In the same way that I try to maintain a certain level of caffeine at any given moment of the day, I can maintain a certain pace to match my energy, to match what I’m up to right now. Am I super excited, am I kind of bored? That’s a virtue of this new world that was not true before.
It’s like the caffeine dose.
No More Unit Tests!
He also likes to automate most of the boring work. Who wants to do one more database migration? Or fix “flaky unit tests”? He likes to do the higher-level strategic stuff, which is how people end up becoming investors or product managers, he says.
Even more so:
I have an agent whose whole job is just to watch for flaky tests and fix them. That’s it, that’s all it does. Database migrations? Absolutely not. I have an agent that does database migrations. I have automated away the shittiest, lowest-value, most unpleasant parts of my life, and I am here for it.
For him, that’s a massive quality increase. And he can focus very hard on things he likes, and where the leverage is high, to make hard decisions.
Decisions Made Are What Is Left, and Communicating Why Is Key
In regard to code review, one thing he is trying to be militant about in the world where agents do everything is:
For every project, I’m focusing really hard on the decisions I am making.
He mentions a concrete example from Datology where he’s building an index over datasets to figure out which parts of a dataset are reusable, wholesale or partially, someplace else:
Option A: query the metadata catalog at read time and figure out the useful assets.
Option B: write an explicit index up front: “this asset exists in this index and it is usable for this purpose”
He says:
Either choice is valid, there are pros and cons. I’m making an explicit trade-off here, and those are the decisions I need to communicate. The details of how the index works are much less important.
He makes a great point that:
It’s really the decisions we need to communicate. The decisions build the shared understanding, the choices we make, the trade-offs. That’s still the meat of engineering work.
I also think this very thing is the most important, and it can turn into a bad spiral if you are not careful, as someone vented in this tweet. Plus, in my opinion, agents force you to focus on architecture, and that’s why they’re handier for seniors.
Is Data Engineering Automated Away with Agents?
Josh says clearly that data engineering is not going away. For the same reason it hasn’t gone away for so long: the fundamental problems won’t go away. We’re still solving the problem of materializing views and caching!
I don’t see data engineering going away in any way, shape or form, even in this world. Aside from the fact that the RL environment problem makes it unusually difficult for the agents to be good at running these very long pipelines, the fundamental problem of data engineering is the same problem it’s been my whole career, and 25 years before I started working. It’s the materialized view problem.
The questions are: “What data should I pre-process and create now to make queries faster later on, and what stuff is not worth doing, what should I just do in the moment?” These are the fundamental problems.
He says it will be a “problem for the superintelligence”, because:
It’s not a fixed thing, it’s going to continuously move, and the right trade-off is going to change all the time.
Josh says, aside from reinforcement learning:
It’s the two hard problems in computer science, right? Cache invalidation and naming things. And the agents still suck at them because they’re legit hard.
“Data engineering is fundamentally the cache invalidation problem”, he continues, “just done in a batch-processing context”.
[!note] Even Hannes Started to Use Claude Code
Hannes Mühleisen said in The Past, Present, and Future of DuckDB, DuckLabs that he now uses Claude Code for DuckDB, since the Quack Protocol launch in May 2026 or so. So he hasn’t written a lot of code since then, but he still looks at things.
Humility and Burnout
Lastly, we talked about humility and how not to get burned out.
It helps to stay humble, as he says he is very good at it, telling his family:
I am simultaneously an idiot and also among the best in the world at what I do.
I say that we all probably feel some kind of ambivalence, as the AI can do part of our job and we feel unneeded. That’s where humility helps us. Josh also adds that he is kind of grateful that he can still be at the keyboard and program. It’s “tremendous fun” for him, and he loves to be here for this moment in time.
I asked if he does not get burned out when agents do not generate code the way he does it. Or does he split into “code I don’t mind” vs “code I want perfect”? He says that his advantage is having done it for a “really, really long time”:
Old me tolerates things that younger me would never, ever, ever tolerate at all. I am used to cycles of feedback with somewhat “stubborn” people.
He said that once, the agent couldn’t find a problem he was telling it about: “And I could not, for the life of me, get the agent to write the pipeline in the way it would do this”. He was telling the agent “you cannot use caching this way”, and it couldn’t help itself. He had to take over.
This again is the perfect example of the exception handling situation. It’s “kind of my job these days,” Josh says, to “identify these situations where, for whatever reason, the model literally isn’t capable of doing what I am asking”. But he says that the models are getting better every day:
The reality is, I don’t encounter that many of these problems anymore.
Conclusion
One main takeaway from this conversation is that data engineering is the cache invalidation problem, just done in a batch-processing context. That’s important: when done right, agents dominate because of a fast feedback loop, like SQL with its explain plans or math. Gigantic Spark and Ray pipelines that run for hours across hundreds of machines don’t have that cycle yet. That’s why Josh still ends up as the emergency exception handler, and why others can vibe code a Databricks job that costs $200k a day.
What’s interesting is how little the model itself matters to him. It’s more about design and feel, e.g., blue bubbles vs. green bubbles: pick the one you enjoy hanging out with, switch when you don’t like it anymore, or hand off the flaky tests and database migrations to an agent. What’s left is what was always the hard part of data engineering: making the decisions and communicating the trade-offs.
We got a bit spiritual, with Josh feeling the tuples flow through a pipeline, and ended where many of us do as the field moves this fast: dreaming of a farm with beautiful flowers and no screen 🙂. Josh admits he periodically flirts with it too: “Maybe I’m just going to retire again and just go play chess. I’m not made of stone”.
He also has no shame about hopping back when the model changes. With the latest model, he went back to Opus 5.5. ↩︎
The same impression someone had on Reddit: “agents are absolutely dogshit AI at optimizing Spark jobs (or any sort of high complexity pipeline). They are laughably bad, in my experience.” ↩︎
I haven’t worked under him, but I’m sure that he is a great manager nonetheless! ↩︎
Login
