I built Picodevil using an LLM

So this is a post about how I used an LLM to make my latest project, and how I feel about that.

To say upfront: I expect a bunch of people to be mad at me for this. Sorry! You're allowed to be mad - for the record, I do think LLMs are an unethical technology. I am not going to try to convince you that they are good. I am using them anyway despite thinking this. And I am planning on continuing to use them.

Let's get into it. Why am I doing this?

Being a creative technologist

One of the many things I call myself when I am working is a "creative technologist". To me, this means looking at new technology and going "what are interesting things you could do with this?". And — I honestly have a pretty ambivalent feeling about the job! I don't like how it involves having your agenda set by large companies with something to promote - whether you are making something that makes the technology seem cool or making something that makes the technology seem ominous, either way they are the ones setting the agenda. I don't like that feeling of rushing to claim a territory. I don't like that it fundamentally centers the technology rather than the experience or the people.

Nevertheless, it's a fun job, and I'm good at it, and people pay me money for it. I think it's normal to have reservations about your chosen field of work. I certainly have reservations about the other fields I also work in! It can be a little bit evil, but, like, really not that evil as evil jobs go.

Anyway, it turns out that the exciting new technology these days is "LLMs". And this has been actually quite convenient for me, because at university I studied... well, not large language models, they hadn't been invented back then, but certainly neural networks and autonomous robotics, and a lot of this kind of squishy non-deterministic non-symbolic AI approach. And then as an artist I have stayed connected to people working in this space - like, I followed Allison Parrish's work closely, and I played with char-rnn when that was new, and I understand what it means when you take the "vector-space embedding" of a phrase. Heck, even before university, in 2004 or so - I had an admissions interview where I got a cushy offer ("it's going to be low, but take that as a sign that we want you rather than that we're crap") partly on the basis of having spent a bunch of time messing around with chatbots.

I'm not saying all this to justify myself, really. It's more to explain where I'm coming from. It's a weird feeling. It's like I was into this cool underground band and I saw them a bunch when they were playing basement shows and then a decade later they're really commercial and are doing stadium tours. Except in this case "basement shows" is a metaphor for "well funded university research labs" and "doing stadium tours" is a metaphor for, among other things, significantly thwarting attempts to resolve the climate crisis before it makes the planet much less habitable by the human race.

Anyway, I found myself working for a startup recently, doing game design consulting for them. Usually this involves getting my hands dirty building prototypes, but in this case the founder I was working for was actually writing most of the code. Vibecoding it, actually. And that made me feel like I was fucking up at my job of "knowing the design implications of new technology". How can I confidently give advice on a piece of technology that I haven't used to someone who has? I hadn't written any major software using an LLM. And it seems extremely likely that any future startup I would work with would put me in the same position.

So, I decided to get set up, and try using an LLM to write some software. And there was this idea I'd had, that I had been kicking around for a few years, but always seemed kind of daunting. So I decided — let's try this, what have I got to lose?

The project is called Picodevil

Picodevil is a livecoding tool for making visuals from videoclips. It runs in the browser. You write... well, technically it's Javascript, but mostly you're just chaining methods together to configure how the videoclips should play. Often you're setting these parameters using a pattern language that is taken wholesale from Strudel. The whole thing is really an extension and adaptation of Strudel from the realm of audio to video.

Let's look at a little example:

$: s("ducks, canalboat").index().rows(3).cols(3).gridMod()

produces this, on a loop:

Anyway, I could go on, but this post is about using AI, not about Picodevil itself. But if you do want more detail on how it works, this readme is the best place to look. And you can also just... go off and try it out

In fact let's link that so there's a card you can click on:

picodevil

picodevil is here to help you livecode visuals! let’s chop up some videos with patterns!

Go try it out! I think it's more interesting than this post about how it was made.

There's a big long list of things I want to add to it or change or fix up, but... I'm proud of it. I think it's a good tool, I feel hopeful that other people will want to use it, it does the thing I set out for it to do. I'm just releasing it now, so who knows what it's life will be beyond this point but — I'm glad I made it.

And yes, as I said, I wrote it using an LLM. I wrote very little of the code myself - mostly a few CSS bits. In fact, of the text in the repository, the largest part that I have typed directly is probably the README file.

The project is licensed under the AGPL license. This means that the source code is shared, and has to be linked from the project itself. And it means that anything which builds upon it also has to be shared in the same way. I don't generally license things this way — but in this case, I am building directly on Strudel, and Strudel has the same license, and so I have to also have this license. I don't object to the license, though — this is the kind of tool which benefits from being open source, and it's nice that someone can't package it up as a paid, closed source thing.

Of course, the thing about having to publish your source code is that people might look at it. And when they do, I think it is quite evident that it is a LLM-generated project^[1]. And when I thought about that, I thought that there would be a high chance that people would yell at me online. So as far as I see it I had three options here:

• don't release the project (but I think it's good and I think people will enjoy using it)
• hope people won't notice I used an LLM (but I think that's unlikely, and then I'll still end up getting yelled at)
• write a big blog post about it, and probably get yelled at when I publish the blog post. But at least I'll get yelled at on my own terms (this is the one I chose)

There's a funny thing about existing across cultures. I feel like the different communities I'm part of have widely divergent takes on LLMs. The kinds of people who do startups either genuinely think it'll change the world or they are pretending they think that in order to get jobs or money. The indie dev people are... well, the public sentiment is virulently anti-AI, but behind the scenes a lot of people (especially coders) are cautiously trying it out. The livecoding people are tired and pissed off, because people keep seeing the "code" in "livecoding" and going "I can generate that". And that's not the point?? Obviously???

Anyway, the point being is that for a lot of the people in a lot of the cultures I'm in, using LLMs is a bad thing and can get you yelled at online. I don't even really object to this? It's shame working in a useful way, really. Applying friction against doing something that is, again, bad for the world. But I have got myself into a situation where to stay a member of a different community (which is a major source of my income) I really do need to engage. And rather than make something I can then keep quiet about (and so maintain a public image congruent with multiple communities), instead I have made something that I want to share, and where revealing the LLM part is necessary in order to share it.

But — Picodevil is not a commercial project. I mean, I'll hopefully make some money from it, from doing workshops with it and performing gigs with it, and so on. I want people to use it, sure, but I don't need to sell it. This isn't a commercial videogame I need to maximise the market for. This tool is built for the sake of joy. To let people mess with something that they couldn't mess with otherwise. It's not something that is obligatory for anyone to use. So, if you don't want to use something that has been built with an LLM... that's fine, don't use it. I'd rather you knew ahead of time and could make the choice. I respect it!

What was it like to build it?

Well, getting started was good. Or, I mean, it happened. This is a project I had been thinking about for years, but it seemed like such a large thing to make, with so many stages that felt like they'd be kind of bullshit. Just wrestling with a build system and getting the core bit of Strudel working with a code editor plugin and wired up to make videos happen... it all seemed like a lot of work, and the kind of work that feels like work. But then I decided to use Claude to get started, and... that stuff happened so quickly. And yes, what got set up wasn't the right thing, quite, and I still had to learn a bunch of details I didn't otherwise care too much about. But I could do that from the position of something is working, not from the position of fuck this, this is gonna take forever.

And then, I carried on building it. And the process felt consistently faster than I would have managed otherwise. This is positive! This is the main reason I would use it again in the future! That said, there are some caveats.

Like: I built a lot of stuff I wouldn't have built if it wasn't easy. Did I need to add a complicated font rendering system so I could smoothly animate the properties of variable fonts? Absolutely not! Did I build it anyway? Yeah! And now I gotta maintain it. And there's some much more basic things which haven't been added. It's easy to get sucked down rabbitholes and focus on the wrong things. I mean, that's true with or without using an LLM, but if you don't you're more likely to go "nah, fuck it, not worth it".

And: you can't trust the LLM to build things well. I mean, "well" is a complicated thing to judge, right? So you gotta explicitly think - hey, how should this be structured? Can we combine these things? How is this coupled, where is this going to grow, is there an underlying pattern to this code we ought to follow? Some of this you can deal with by just asking it to review the code it's made, but lots of it comes down to having taste and applying that taste repeatedly.

And that's letting aside that often the first shot at a problem just doesn't work. That's not unique to LLMs, my first attempt at running things I've written doesn't usually work^[2]. Often at the start of this project I'd ask it to make something, it would, I'd try it and it wouldn't work. So I paste in the error message, it would whirr away... new error message. And repeat. And then I got smarter about this, and started asking it to make this cool software thing I've heard of called "tests". This project is way more heavily tested than anything else I've built! Now, mostly this is because it's easier to add tests because I don't have to write them. But also because it means that the whole make a change/find an error/fix the error loop can now be done before I have to get involved.

Well, mostly. The thing with making this kind of thing is that it's fundamentally a tool for human creativity. It doesn't have a prespecified correct behaviour. It's something that's intended to be operated by a human. Which means that anywhere that there's a creative choice or an aesthetic one, it's back to being awful again. I think it says something that a lot of the times I have written code myself, it's CSS code. Because I care about how it looks, and also I can see how it looks and interact with it myself. That's not something you can delegate to automated testing.

And so - you do have to still think about what you're making. Even how it's made. And I do worry — I have that taste & those thoughts, because I have been making stuff like this for a long time. Would I be able to pick that up if I had been working through an LLM? I'm not sure, and for a different (again self-directed) project I've been trying to keep to writing code myself, just because it's using tech I haven't worked with before and I want to learn it more deeply. But on the other hand — maybe you learn these things more quickly, because you can build faster and so get to try more things, see how different architectures work out, different design patterns, etc? It could go either way. Or both ways, or some kind of complicated way.

So, on this project: I feel like I have learned a lot about the internals of Strudel. I can explain how random seeding works, the difference between events "coming from the left" and the "set.mix approach"^[3]. I've looked at a lot of the Strudel code! But does this count if I would still stumble to reimplement things, if I don't have that practice actually making changes? There's some bits that I think are cool enough I'd think about contributing them back to the Strudel project — but they don't accept LLM-assisted contributions^[4], so I'd have to reimplement them myself.

But on the other hand: what if the real thing I am unlearning is the habit of perseverance. Of sticking with something even when it's annoying? I think this is a weakness of mine — I'm bad at doing things I don't enjoy. Mostly I have solved this by arranging my life such that I can avoid doing things I dislike wherever possible... but it's probably not ideal for me to get even worse at this.

LLMs smooth out a lot of discomfort. Both in doing tasks that are difficult or boring, and also in providing affirmations which remove a lot of the psychological difficulty of making. Overall, the thing I'm most reminded of is being sucked into the kind of hyperfocus I remember from playing, like, Zachtronics games. Where there are interesting problems to solve, but also the UX and the game design has been iterated on so that you're never stuck, never left uncertain, never left without the tools to accomplish the task set for you. It feels great, but those are games that are clearly unhealthy for me. I remember one Christmas break, emerging after 48 hours of pretty much doing nothing but playing Shenzen IO in bed, not sleeping enough, not eating enough, not talking to anyone. And then I told myself - don't play any of these games ever again, they're not good for you. This is probably not the healthiest comparison to be making.

In conclusion

But! Despite that feeling, I still plan to use LLMs to code in the future. Again, they're useful. I've been using them on other projects just this week^[5].

I feel proud of this project. I think it's a cool thing, and I have put my name on it and I want people to use it. I can't wait to see what people make with it — I think they will make cooler stuff than I could.

I have no plans to hook up an LLM to generate patterns within it. I just don't think that an LLM could make anything interesting. I mean, I have no doubt that an LLM can generate valid code for it, I have seen this happen. And maybe when there are enough patterns out there, then an LLM could ingest them and mash stuff up, and accidentally spit out something. But fundamentally it's about taste, and it's about a viewpoint, and it's about someone expressing themselves.

So there's a bit of a contradiction here. I've used an LLM to write the code of a tool which you control with code. But I don't think the LLM can write the code that the tool runs.

I guess there's something here about — I consider myself an artist, and one of the things I care deeply about is making creative tools. But I don't consider myself to be doing especially creative work when I am wrestling with the CodeMirror plugin system in order to change how it applies syntax highlighting. I'm, like, just bodging stuff together til it works well enough for the actual creative bit^[6].

And yet! And yet, I can't deny that there are not signs, when you use Picodevil in the state that I am launching it, that it was built with an LLM. I defend LLMs for writing code, but I hate reading their prose, and yet it launches with documentation that was generated (as well as some I have written myself). There are janky corners, where something got added by an LLM and has not yet gotten iterated on til it actually works well^[7]. And, of course, it's open source, and if this thing is widely used, I hope people will take the code and do weird and wonderful things with it. And when they do, they will have to interact with code that an LLM has written.

And while there are plenty of parts to this project that feel human-made, that are human made, that have that direct connection between me and you^[8], there are parts of it that have that inhuman greasy feeling of slop. And I regret that. But at the same time, I think I would not have made this, or would have made this in a much smaller, much less capable form, if I had not used an LLM. And I wanted to make it. So I did.