As with my last post on AI, let me open with a statement:
Just because most people hold AI a certain way, doesn’t mean that we always must, either now or in the future.
As I noted in that piece, and my shades of gray post, I am well aware — and agree with — the notion that a lot of frontier AI models have disturbing ethical concerns that we shouldn’t dismiss, including concentration of power and wealth, environmental concerns, unethical scraping and training, the financial bubble they seem to be creating, and more. The most poisonous of AI cheerleaders are accelerating enshittification while cutting quality all over the place. AI slop is also a real problem, which I’ll discuss in a bit.
Before I dive in, I need to mention that no AI assistance was used for writing this post, and I have been using em dashes — correctly — for years before LLMs did. Secondly, while popular discourse seems to use “AI” and “LLM” interchangeably, the AI field is much more broad and much more old than LLMs. I will, nonetheless, adopt the common “AI” jargon here, but be more specific when necessary to differentiate.
In The Reverse Centaur’s Guide to Life After AI by Cory Doctorow, he presented two ideas I particularly want to focus on here:
- The present and future of LLMs outside of the cloud-hosted frontier models
- What differentiates productive use of AI from slop
The two are related, in fact. Doctorow discussed how he never uses AI to write anything, but he sometimes uses it as an assistant to help him write. Is that contradictory? Not really. He gave this example: he remembered a quote from a podcast, which he wanted to cite and attribute correctly. But he couldn’t remember which podcast or who said it. So he fired up Whisper, an open-weight model you can download and run locally, even on meagre hardware. Whisper is a speech-to-text model. Doctorow ran likely podcast episodes (from the target range and set of podcasts he listens to) through the model, and at that point it was a regular search. He found the quote, then went back and listened to the episode at that time spot to verify its accuracy.
It would have taken so much time to do this without Whisper that he simply would not have done it otherwise. His writing was better because of AI. Not because AI wrote any words for him, but because it helped him find the right ideas and attributions. He didn’t even have to trust its transcription; it pointed him at the right place, and then he could transcribe with precision.
I too have used Whisper in much the same way. A little while back, I digitized many hours of old family audio recordings going back to the 1970s. Thanks to Whisper, which I ran locally primarily during hours of being fully solar-powered, I have a searchable index to these gems. A person I know recently asked me if I knew any way to transcribe some old family recordings he had. I told him I could run it through Whisper. It wouldn’t have great punctuation, but most of the words would be there. This meant a lot to him and his family; what I did in 2 minutes would have taken him many days (I presume he is not a fast typist as well.)
So we can start by naming: AI isn’t just chatbots, and AI can help even if it isn’t writing anything.
Doctorow goes on to ask the question: what happens after the AI bubble bursts? I do believe it will, as Ed Zitron has carefully documented for us for a long time now. The environmental damage done by frontier training and inference in the last few years will, of course, be done. But as he points out, we’ll be left with a plethora of surprisingly-capable open weight models, a community that has made significant strides running them on low-end devices (even phones), and a massive buildup of clean energy that hopefully will be finally able to offset some polluting energy once these heavy users drop off the grid.
So then what? Will the world just throw away the likes of Whisper and Gemma? Obviously not. Doctorow imagines, credibly, a situation in which individuals and universities tinker with locally-run models, iterating on them and improving them with the more traditional scale of resources available to individuals and universities. In other words, this future sounds a lot like the earlier days of Open Source.
As Ed Zitron has pointed out, the real costs of frontier AI have been hidden as these companies rack up fantastical losses. When the bubble bursts, the real costs of frontier AI are likely to be high enough that CEO’s won’t be able to ignore the poor ROI. But we’ll be left with these artifacts, these open-weight starting points, that people can use.
That brings me to my central thesis: a lot of anti-AI sentiment is focused on the real downsides of the hyped uses of AI, and fails to account for the more ethical advances being made now, and likely to come in the future.
Varying quality of AI use
We are all familiar with AI slop. I’ve seen it, and have an increasingly skeptical eye when I see “new” projects announced in even places as niche as a BBS community (where I see things like new terminal emulators announced, written in shockingly-poor nodejs, and such). I’ve seen documents that take many pages to say what amounts to nothing intelligble to a human, with perfect grammar. I’ve seen pull requests that have a description that amounts to several pages of slop. I’ve personally seen an AI agent do things like correctly detect passing an untrusted argument to a shell, then propose a broken quoting algorithm to fix it (rather than proper sanitization and exec() or, better yet, avoiding the call altogether.)
I’ve also seen AI tools very quickly and accurately identify bugs that had passed human review, even professional security reviews, for years. I’ve seen them bring more joy to some people’s jobs by handling the more drudgery parts of the jobs. One of my favorite podcasts, Oxide and Friends, had an episode called Engineering Rigor in the LLM age and credibly described ways in which careful use of LLMs can increase quality and safety. Yes, I said increase. (Oxide and Friends are not hype cheerleaders by any means; these are very serious people with solid technical chops, and I know some of them.)
What accounts for this disconnect?
We can point to many factors, but Doctorow asks a fundamental question: who is in charge? Is it the person or the computer? Does the AI service people, or do people service the AI?
In some of the most ridiculously AI-first companies, it is very clearly the latter. Mandates from CEOs to increase AI usage lead to perverse outcomes that sacrifice quality. The AI is in charge. I’ve never seen an AI be any good at design questions, and it’s no wonder than all these vibe-coded apps have the same generic feeling of crappiness.
But when the AI is used pragmatically as a junior research assistant, that’s a different approach. Its output is greeted skeptically, its actions distrusted, and it’s never asked to design something. These are the cases where I’ve seen it make a meaningful difference.
It’s not black-and-white
All this adds up to why I was opposed to the Debian proposals that would have banned LLM use. Those votes are hard to change. They would have left Debian in a vulnerable position of being unable to react to accurate security findings if they originated with an LLM, or left Debian unable to adopt ethical use of LLMs (such as the solar-powered local Whisper I described above).
It would be akin to saying “you should never buy a PC because Windows 11 is terrible.” I agree Windows 11 is terrible, but saying something like that is so dismissive of all those that have invested decades of their time in building something better.
So, I find AI hype to be uncredible and dangerous. I also find knee-jerk reactions to AI to be the same. They are actually a form of capitulation: they let the AI megacorps dictate the terms of the conversation to revolve around them.
Case study: Debian’s DFSG scanner
For a long time, my two biggest annoyances with Debian have been the difficulty of packaging Rust and Go packages with their long dependency trees, and how long it takes a new package to get reviewed. About a year ago, the latter process was set on a path to reform, and it has gotten dramatically better. It also cuts off a potential connection with someone that hasn’t yet heard of Linux. Some people may not jump from Windows 11 (or the ChatGPTs of the world) immediately, but pretending that it’s ChatGPT or nothing does a disservice to all those that are trying to do research etchically.
We should state at the outset that Debian takes compliance with the Debian Free Software Guidelines (DFSG) extremely seriously. Every new package gets a review, and these reviews often catch things that upstream repos and other distros don’t. One common example would be a package that has a stated copyright and license, but then includes files that bear a different copyright and license.
You can imagine reviewing packages at this level of detail isn’t a fun task for people, and it’s hard to find volunteers to do it. Perhaps the answer to this should be “Debian should stop caring so much more than everyone else”, and perhaps you’re right (and perhaps you aren’t).
In any case, the DFSG team that reviews this added what amounts to an LLM-backed license linter. Its job is to produce an initial assessment that the license statements documented in debian/copyright are correct, thorough, and in compliance with DFSG. This report was not taken to be authoritative, but was designed to help with some initial assessments.
There were some predictable reactions. People were angry that the DFSG team had added this tool without any communication about it (justifiably so, I’d say). Many were angry just because of LLM. People made assumptions about what kind of model it used, what kind of quality it would produce, how definitive it would be in the project.
It is worth noting here that Debian is uniquely decentralized. There is a central policy, but maintainers are free to maintain their packages how they like. They don’t have to use a standard location for a Git repo, or even use Git.
One person said essentially, “it’s fine for others to use whatever tool they want, but I shouldn’t have to be exposed to LLM slop.” The other perspective boiled down to, “why should they keep the reports secret? If you want to proactively fix things, or flag false positives, why not?”
As Andreas Tille wrote, “I have read the LLM reviews carefully, and in many cases their findings inspired additional commits to improve my packaging. In some cases, they even helped me catch mistakes that led me to reject an upload myself before human inspection.”
Joey Hess raises an interesting risk: that LLMs will remove the incentive for Debian to actually fix its cumbersome processes, by letting an LLM paper them over. I have seen this tendency in other contexts, both at work and at home, as well. It’s a real risk, especially if Debian doesn’t tackle the packaging problem for Rust and Go.
The Mastodon post that prompted me to write this began a chain of events that started with “Uploaders will be forced to address LLMs issues without human inspection.” I don’t believe that will be the case; it goes too far against the Debian ethos.
I encourage everyone to take a deep breath. We are all getting used to a shifting landscape. We will all make missteps, myself included. The issues are complicated, the picture isn’t black and white, and there is incredible hype around the ecosystem that can make finding nuance difficult.
Let’s be kind to each other, fellow humans.
But, I do not care if you are kind to Claude.
Likes