Tag Archives: llm

Must we always use LLMs this way?

Programming, ,

As with my last post on AI, let me open with a statement:

Just because most people hold AI a certain way, doesn’t mean that we always must, either now or in the future.

As I noted in that piece, and my shades of gray post, I am well aware — and agree with — the notion that a lot of frontier AI models have disturbing ethical concerns that we shouldn’t dismiss, including concentration of power and wealth, environmental concerns, unethical scraping and training, the financial bubble they seem to be creating, and more. The most poisonous of AI cheerleaders are accelerating enshittification while cutting quality all over the place. AI slop is also a real problem, which I’ll discuss in a bit.

Before I dive in, I need to mention that no AI assistance was used for writing this post, and I have been using em dashes — correctly — for years before LLMs did. Secondly, while popular discourse seems to use “AI” and “LLM” interchangeably, the AI field is much more broad and much more old than LLMs. I will, nonetheless, adopt the common “AI” jargon here, but be more specific when necessary to differentiate.

In The Reverse Centaur’s Guide to Life After AI by Cory Doctorow, he presented two ideas I particularly want to focus on here:

  1. The present and future of LLMs outside of the cloud-hosted frontier models
  2. What differentiates productive use of AI from slop

The two are related, in fact. Doctorow discussed how he never uses AI to write anything, but he sometimes uses it as an assistant to help him write. Is that contradictory? Not really. He gave this example: he remembered a quote from a podcast, which he wanted to cite and attribute correctly. But he couldn’t remember which podcast or who said it. So he fired up Whisper, an open-weight model you can download and run locally, even on meagre hardware. Whisper is a speech-to-text model. Doctorow ran likely podcast episodes (from the target range and set of podcasts he listens to) through the model, and at that point it was a regular search. He found the quote, then went back and listened to the episode at that time spot to verify its accuracy.

It would have taken so much time to do this without Whisper that he simply would not have done it otherwise. His writing was better because of AI. Not because AI wrote any words for him, but because it helped him find the right ideas and attributions. He didn’t even have to trust its transcription; it pointed him at the right place, and then he could transcribe with precision.

I too have used Whisper in much the same way. A little while back, I digitized many hours of old family audio recordings going back to the 1970s. Thanks to Whisper, which I ran locally primarily during hours of being fully solar-powered, I have a searchable index to these gems. A person I know recently asked me if I knew any way to transcribe some old family recordings he had. I told him I could run it through Whisper. It wouldn’t have great punctuation, but most of the words would be there. This meant a lot to him and his family; what I did in 2 minutes would have taken him many days (I presume he is not a fast typist as well.)

So we can start by naming: AI isn’t just chatbots, and AI can help even if it isn’t writing anything.

Doctorow goes on to ask the question: what happens after the AI bubble bursts? I do believe it will, as Ed Zitron has carefully documented for us for a long time now. The environmental damage done by frontier training and inference in the last few years will, of course, be done. But as he points out, we’ll be left with a plethora of surprisingly-capable open weight models, a community that has made significant strides running them on low-end devices (even phones), and a massive buildup of clean energy that hopefully will be finally able to offset some polluting energy once these heavy users drop off the grid.

So then what? Will the world just throw away the likes of Whisper and Gemma? Obviously not. Doctorow imagines, credibly, a situation in which individuals and universities tinker with locally-run models, iterating on them and improving them with the more traditional scale of resources available to individuals and universities. In other words, this future sounds a lot like the earlier days of Open Source.

As Ed Zitron has pointed out, the real costs of frontier AI have been hidden as these companies rack up fantastical losses. When the bubble bursts, the real costs of frontier AI are likely to be high enough that CEO’s won’t be able to ignore the poor ROI. But we’ll be left with these artifacts, these open-weight starting points, that people can use.

That brings me to my central thesis: a lot of anti-AI sentiment is focused on the real downsides of the hyped uses of AI, and fails to account for the more ethical advances being made now, and likely to come in the future.

Varying quality of AI use

We are all familiar with AI slop. I’ve seen it, and have an increasingly skeptical eye when I see “new” projects announced in even places as niche as a BBS community (where I see things like new terminal emulators announced, written in shockingly-poor nodejs, and such). I’ve seen documents that take many pages to say what amounts to nothing intelligble to a human, with perfect grammar. I’ve seen pull requests that have a description that amounts to several pages of slop. I’ve personally seen an AI agent do things like correctly detect passing an untrusted argument to a shell, then propose a broken quoting algorithm to fix it (rather than proper sanitization and exec() or, better yet, avoiding the call altogether.)

I’ve also seen AI tools very quickly and accurately identify bugs that had passed human review, even professional security reviews, for years. I’ve seen them bring more joy to some people’s jobs by handling the more drudgery parts of the jobs. One of my favorite podcasts, Oxide and Friends, had an episode called Engineering Rigor in the LLM age and credibly described ways in which careful use of LLMs can increase quality and safety. Yes, I said increase. (Oxide and Friends are not hype cheerleaders by any means; these are very serious people with solid technical chops, and I know some of them.)

What accounts for this disconnect?

We can point to many factors, but Doctorow asks a fundamental question: who is in charge? Is it the person or the computer? Does the AI service people, or do people service the AI?

In some of the most ridiculously AI-first companies, it is very clearly the latter. Mandates from CEOs to increase AI usage lead to perverse outcomes that sacrifice quality. The AI is in charge. I’ve never seen an AI be any good at design questions, and it’s no wonder than all these vibe-coded apps have the same generic feeling of crappiness.

But when the AI is used pragmatically as a junior research assistant, that’s a different approach. Its output is greeted skeptically, its actions distrusted, and it’s never asked to design something. These are the cases where I’ve seen it make a meaningful difference.

It’s not black-and-white

All this adds up to why I was opposed to the Debian proposals that would have banned LLM use. Those votes are hard to change. They would have left Debian in a vulnerable position of being unable to react to accurate security findings if they originated with an LLM, or left Debian unable to adopt ethical use of models.

It would be akin to saying “you should never buy a PC because Windows 11 is terrible.” I agree Windows 11 is terrible, but saying something like that is so dismissive of all those that have invested decades of their time in building something better.

So, I find AI hype to be uncredible and dangerous. I also find knee-jerk reactions to AI to be the same. They are actually a form of capitulation: they let the AI megacorps dictate the terms of the conversation to revolve around them.

Case study: Debian’s DFSG scanner

For a long time, my two biggest annoyances with Debian have been the difficulty of packaging Rust and Go packages with their long dependency trees, and how long it takes a new package to get reviewed. About a year ago, the latter process was set on a path to reform, and it has gotten dramatically better. It also cuts off a potential connection with someone that hasn’t yet heard of Linux. Some people may not jump from Windows 11 (or the ChatGPTs of the world) immediately, but pretending that it’s ChatGPT or nothing does a disservice to all those that are trying to do research etchically.

We should state at the outset that Debian takes compliance with the Debian Free Software Guidelines (DFSG) extremely seriously. Every new package gets a review, and these reviews often catch things that upstream repos and other distros don’t. One common example would be a package that has a stated copyright and license, but then includes files that bear a different copyright and license.

You can imagine reviewing packages at this level of detail isn’t a fun task for people, and it’s hard to find volunteers to do it. Perhaps the answer to this should be “Debian should stop caring so much more than everyone else”, and perhaps you’re right (and perhaps you aren’t).

In any case, the DFSG team that reviews this added what amounts to an LLM-backed license linter. Its job is to produce an initial assessment that the license statements documented in debian/copyright are correct, thorough, and in compliance with DFSG. This report was not taken to be authoritative, but was designed to help with some initial assessments.

There were some predictable reactions. People were angry that the DFSG team had added this tool without any communication about it (justifiably so, I’d say). Many were angry just because of LLM. People made assumptions about what kind of model it used, what kind of quality it would produce, how definitive it would be in the project.

It is worth noting here that Debian is uniquely decentralized. There is a central policy, but maintainers are free to maintain their packages how they like. They don’t have to use a standard location for a Git repo, or even use Git.

One person said essentially, “it’s fine for others to use whatever tool they want, but I shouldn’t have to be exposed to LLM slop.” The other perspective boiled down to, “why should they keep the reports secret? If you want to proactively fix things, or flag false positives, why not?”

As Andreas Tille wrote, “I have read the LLM reviews carefully, and in many cases their findings inspired additional commits to improve my packaging. In some cases, they even helped me catch mistakes that led me to reject an upload myself before human inspection.”

Joey Hess raises an interesting risk: that LLMs will remove the incentive for Debian to actually fix its cumbersome processes, by letting an LLM paper them over. I have seen this tendency in other contexts, both at work and at home, as well. It’s a real risk, especially if Debian doesn’t tackle the packaging problem for Rust and Go.

The Mastodon post that prompted me to write this began a chain of events that started with “Uploaders will be forced to address LLMs issues without human inspection.” I don’t believe that will be the case; it goes too far against the Debian ethos.

I encourage everyone to take a deep breath. We are all getting used to a shifting landscape. We will all make missteps, myself included. The issues are complicated, the picture isn’t black and white, and there is incredible hype around the ecosystem that can make finding nuance difficult.

Let’s be kind to each other, fellow humans.

But, I do not care if you are kind to Claude.

AI in Debian: The Vote, Proposals, and Nuance

Debian, ,

Let me start with a hypothesis:

For human developers, using coding LLMs magnifies their difference in skill levels.

I am one that rarely thinks things are always black and white. Back in March, I wrote Artifial Intelligence: Shades of Gray. Since then, I’ve had more of a chance to experiment with LLMs myself. I also happen to work for an employer that is taking a very pragmatic approach to LLMs: teams and individuals use it as they see fit, but if they are causing considerable expense, they have to justify it.

In various settings, I have seen the egregious examples of AI slop we all know about. As I wrote in March, “I have seen it both waste more time than it saves, and save a ton of time.”

I have come to see that, as a tool, it is most valuable when it is running under the supervision of an experienced engineer. It is at its worst when it has no such supervision; the “vibe coding” and other low-quality slop we see.

A coding agent is like a junior developer or research assistant. When properly supervised, they help projects move along more quickly by letting a senior developer focus on the more difficult, less mundane aspects of the project. But one couldn’t expect a junior developer to consistently deliver high-quality code and architecture on their own.

Let’s put a pin in this idea and look at the story in Debian.

LLM use in Debian

There is a vote happening in Debian around the use of LLMs. In typical Debian fashion, there are 8 options to choose from, many of them similar. Most of these proposals acknowledge there are different types of tasks done in Debian, but the proposals don’t differentiate between them well. Let me do so here. These are some of the LLM-relevant tasks people in Debian perform:

  • Packaging upstream software for Debian (by far the largest task)
  • Writing Debian-specific code (eg, apt or the Debian installer)
  • Maintaining Debian infrastructure (build systems, for instance)
  • Writing documentation and translations

I’m going to focus my remarks here on packaging upstream software for Debian, since this is by far the most time-consuming developer task project-wide.

It matters to our users that we get this right, and packaging quality is one of the things that sets Debian apart from other distros. Packaging things for Debian requires knowledge of some specific tools, such as debhelper, that aren’t widely used anywhere else. In most cases, it is fairly rote time-consuming work. In other words, by its design, it requires people with senior-level skills to do grunt work.

I can’t overstate how massive a burden this grunt work is. I maintain some packages for Go and Rust. By Debian policy, all of those packages’ dependencies must also exist as Debian packages, and be used to build against. When upstream adopts a newer version of some library, it can unleash cascading dependencies that can take hours to sort out. Worse, the Rust team and the Go team use entirely different ways of managing packages (Go uses one Git repo per package, while Rust has a monorepo with specialized scripts to import Cargo packages and generate Debian ones). On top of that, we can’t just modify things like usual; we have to use quilt. And on top of that, I’m also a backports maintainer, so all the work (and usually even more) has to be done there also.

Now let’s pull on that pin from the earlier conversation. This is exactly the kind of scenario that a well-supervised coding LLM is most effective in. I could see a seasoned developer saving hours, maybe even days, by turning over the mundane tasks of managing trees of cascading dependencies over to a coding tool — and verifying and directing the process. (Yes, I have been using em-dashes for years; LLMs have copied people like me, not the other way around! This post was not written with any AI assistance.)

Actually, this is almost a dream scenario for a coding assistant. The result is time-consuming to formulate but easy to review, which is the opposite of the way these things often go.

I can assure you with 100% certainty that humans aren’t adding a lot of value in this process. It would be wrong to believe that a human is carefully reading every line of code in dozens of updated or new library packages. The problem set is too big, the time too short, and the code too varied and complex.

Coding agents seem to be most effective when there are strong test suites that they can test changes against. Debian builds, especially of modern packages, tend to have this property. Many packages have test suites that are run during build. And, if the package builds in an isolated environment (and especially if its downstream dependencies do also), then there is a decent chance that it’s fairly correct. Maybe needing some manual tweaking here and there, but generally a successful build is a reasonable indicator.

You can argue that it would make more sense for Debian to just include dependencies in source packages, along with some version information to support security rebuilds, and I’d tend to agree with you. But we are where we are. This would be one of the more significant leaps forward in developer productivity, but it complicates things like copyright reviews.

Where are LLMs run? What is the environmental impact?

Most of the proposals seem to make the assumption that LLMs must always run in some large, hosted datacenter. As I noted in my March article, I have had credible results on even an older GPU running on solar power.

That said, it is undeniable that LLMs are fueling a datacenter boom, and this in turn is producing a significant new demand for resources. Most notably for the global scale: electricity, which is sometimes generated using carbon-emitting technologies.

Bill McKibben, who has been a leading voice in the fight against climate change since the 1980s, has made some interesting points recently: he’s noted that solar power is the fastest kind of generation we can build, and a number of large AI companies are investing heavily in solar, even to the point of fully offsetting new datacenter’s needs. On the other hand, he’s also noted that some companies are buying inefficient and dirty gas turbines. It is decidedly a mixed bag. The heavy investment in solar can have knock-on positive effects for infrastructure. Obviously, not every picture here is rosy. This analysis doesn’t touch on the real land and water use situation, either.

On the other hand, if an LLM allows me to do in an hour what I would have done in a day, that’s a day of not heating or cooling the work area — generally not sustaining a human for the purpose of writing code for Debian. HVAC energy consumption dwarfs my GPU, and I’d imagine probably also the slice of LLM energy used.

Holistically, I would have to conclude the picture is mixed. It is possible to use LLMs in a pretty green way, and also in a pretty dirty way.

Assuming Conditions Never Change

A flaw in most of these proposals is they assume that the conditions at this present moment will always hold. In fact, that the conditions at the present moment will not continue is something both AI cheerleaders and AI skeptics agree on.

For instance:

Ed Zitron has done a ton of research into the financing side of AI, and has concluded that the current model is unsustainable and headed for a significant bubble burst. I’m not positioned to personally evaluate those claims, but if that happens, what is the result? Perhaps it is a steeply increasing cost of inference for the frontier models, slower pace of training/evolution for them, etc.

In a recent episode of Oxide and Friends, Simon Willison discussed the open weight models that are now available. They have been making remarkable strides in efficiency and capabilities, to the point where $50,000 of hardware can now run high-end open weight models with capabilities that are at least in the same ballpark as the American frontier models. This puts running high-end models locally squarely within reach of universities and small- to medium-sized businesses, with power requirements that can be met with standard commercial solar and wind installations.

The lack of nuance in the more restrictive proposals is particularly concerning. Proposal A doesn’t allow “the use or assitance of… LLMs”. So it bans my solar-powered GPU. It bans using LLMs to find security issues. It bans all sorts of things that don’t seem to be ban-worthy, alongside the things that do. And it codifies it in the very hard-to-change social contract.

That proposal, and some like it, seem to imply that all LLM output is bad. I grant you that AI slop is a real and legitimate concern, and many Open Source projects have to deal with it. On the other hand, we have all seen first-hand how the security of the Linux kernel has benefited dramatically from AI analysis. It is certain that black hats are using these tools. If we refuse to use modern security tools, our security will be compromised (and what is the environmental and social impact of THAT?)

I find the statement “Generative AI is characterized by producing output of a nature that would ordinarily be produced and consumed by humans” to be particularly interesting. The same was once said of compilers.

The Real Concerns

You might think from reading this that I am some AI cheerleader. I’m not. I share the ethics of the FLOSS movement, and have for decades. I abhor the power and lack of ethics that many big names in the field are running with at the moment. I’ve had to put up Anubis on this blog, for instance.

I have personally experienced the effects of AI slop, especially at review time. This is a real problem, though I don’t think the more draconian policies are likely to help (the looser “you must disclose” stand a fighting chance, but I’m not sure they would help, either.) Done poorly, AI threatens developer burnout by overwhelming them with poor code and verbose but useless explanations. Done well, AI can help prevent developer burnout by automating tedious and low-value tasks.

Shouldn’t our goal be that humans submit work to Debian, using tools they prefer, and take responsibility for it? Does it matter if someone uses ed, vim, emacs, or vscode? If they use LSP or just run gcc manually? I’d say we benefit from the diversity. Wouldn’t we be better off to benefit from the diversity here, and judge work as we always have: on its merits, not what tools were used to create it?

Fundamentally, a GR is a long and arduous process. It’s not easy to reverse later. Amending the Social Contract is even longer and more arduous (I should know; I may have been the first one to try). The LLM landscape is fast-moving. None of us can really predict where it will be in a year. Will the current market leading companies even still exist? Will it be at all credible to refuse to use AI-assisted security tools? What is the most effective way to deal with AI slop? What level of utility will we be able to achieve with models run locally?

Some of these proposals would make sense if drafted in some way short of a GR, which would allow more maneuverability as the landscape changes.

Brief analysis of the options

Considering the proposals:

  • Proposal A: seeks to amend the social contract, which I am opposed to for reasons already laid out above. It names some real concerns about AI that I agree with, but implies that all LLM uses and models are guilty of the problems, which is not the case with all of the claims. It also sets us behind the curve on security and stability by forbidding the use or assistance of those tools, even if run by others. It requires us to ignore reports of actual security bugs, or correct fixes, if those reports were generated with the assistance of an LLM, which I find to be absolutely untenable.
  • Proposal B: This is the “AI with accountability” approach. It notes the real concerns with LLMs without painting with an overbroad brush. It strikes me as level-headed and sensible.
  • Proposal C: It paints with an over-broad brush and makes some non-binding requests. Then it winds up largely like proposal B, though while it is worded more strongly, has fewer binding requirements (for instance, it lacks proposal B’s prohibition on transmitting sensitive information to untrusted providers)
  • Proposal D: Seems broadly similar to proposal B, an “AI with accountability” approach. I’m not really clear why we need both.
  • Proposal E: Largely the status quo. It is like proposals B and D in that it says humans are accountable for their contributions. It encourages disclosure of LLM use, but does not mandate it. Like proposal B, it prohibits disclosing sensitive information to third-party AI services. Note that both proposals B and D have an appropriate nuance: a local model is fine, a third-party one is not.
  • Proposal F: This seems really similar to proposal E. I’m not sure why we have these two.
  • Proposal G: Disallows “the output of generative AI as direct contributions to Debian.” This is something of a weakened proposal A; it doesn’t seek to amend the social contract, nor does it ban all use; it simply bans the use as a direct contribution.
  • Proposal H: Ban due to climate impacts. “How is this even an argument” is disrespectful to reasoned conversation. I have already noted that LLMs can be and are used in ways that are not climate-harming. It explicitly contains no binding requirements at all, and is effectively a rant. While I agree with the sentiment that climate change is an urgent problem, and that some LLMs are exacerbating it, I disagree with that all LLM usage does so and therefore disagree with the conclusion.

In favor of nuance

I find that black-and-white thinking is almost always something to be avoided. I see it too often. I see it in politics, I see it in our software, I see it in discussions around AI. Are there deeply unethical things happening in AI? Absolutely. Are they doing some impressive things? Also yes.

We have accepted this nuance in other areas. For instance, almost all the hardware Debian runs on has closed-source hardware, and has components manufactured or assembled in countries with some of the worst human rights records on the planet. I’m not saying this is a great state of affairs. It is something we should speak up about and act upon. But the worse state of affairs would be “no Debian because the hardware is impure”.