Not all AI authored code can be considered slop, which is why a blanket ban on AI would be counterproductive. My definition of “slop” is work that is AI generated, with very little involvement by the human operator. It may seem like a good deal if somebody is spending their tokens to help your project, but without a passing understanding of the project or issue in question, the author can’t always prompt their way to a good solution.

As far as I can tell, most AI slop PRs are generated by a relatively small number of individuals. They tend to arrive in batches, and I can see the author has submitted dozens or even 100s of PRs to other projects. The work is typically a poor solution, not required, or simply broken. And the author will never follow up on comments.

It is in effect a DDOS for FOSS maintainers; it takes much longer to review the PR than it did to create it.

Nonetheless, I still feel bad about closing a PR without comment. But at the same time resentful at having to spend time formulating a response, that will likely be ignored.

So I plan to include a text file in my project, to clarify my stance on AI PRs. I’m calling this AI_POLICY.md. I did consider adding an AI policy to CONTRIBUTING.md, but that file tends to be used to inform how to contribute, which seems a different purpose entirely.

I’m hoping this could become a standard file, and AI agents would refer to this when generating a PR. Until then, I can link to it when I close slop PRs.

If something like this exists already, or there is a more agent-friendly way of doing this, then let me know.

Here’s the text I’m going with. I don’t think this is particularly challenging to meet, and only a slightly higher bar than what I’d expect from a mammalian brain.

AI_POLICY.md

This project accepts AI generated Pull Requests, as long as the following guidelines are met.

• The Pull Request must fill in the repository’s pull request template.
• The Pull Request must identify itself as AI generated, including the name of the agent used.
• The Pull Request must link to a issue or discussion where a solution has been approved by a maintainer (@willmcgugan).

The maintainer reserves the right to close PRs without comment if the above are not met.

Update

Turns out AI_POLICY.md is being used in a bunch of projects, for this purpose.

Oddly, Claude told me there was no standard for this. I suspect this is an organically emerging “standard”, and I hope it is adopted by more projects!

Good AI

AI for writing code is a skill multiplier.

We haven’t reached the point where a normie can say “Photoshop, but easier to use”. Will we ever? But for now it seems those who are already skilled in what they are asking the AI to do, are getting the best results.

I’ve seen accomplished developers on X using AI to realize their projects in a fraction of the time. These are developers who absolutely could write every line that the LLM produces. They choose not to, because time is their most precious commodity.

Why is this good AI? It means that skills acquired in the age before AI ¹ are still valuable. We have a little time before mindless automatons force senior developers into new careers as museum exhibits, tapping on their mechanical keyboards in front of gawping school kids, next to the other fossils,

Bad AI

The skill multiplier effect may not be enough to boost inexperienced (or mediocre) developers to a level they would like. But AI use does seem to apply a greater boost to the Dunning-Kruger effect.

If you maintain an Open Source project you may be familiar with AI generated Pull Requests. Easily identifiable by long bullet lists in the description, these PRs are often from developers who copied an issue from a project into their prompt, prefixed with the words “please fix”.

These drive-by AI PRs generate work for the FOSS developer. They can look superficially correct, but it takes time to figure out if the changes really do satisfy the requirements. The maintainer can’t use the usual signals to cut through the noise when reviewing AI generated PRs. Copious amounts of (passing) tests and thorough documentation are no longer a signal that the PR won’t miss the point, either subtly or spectacularly.

This is bad AI (more accurately a bad outcome), because it typically takes more time for the maintainer to review such PRs than the creator took to type in the prompt. And those that contribute such PRs rarely respond to requests for changes.

In the past you could get around this with a blanket ban on AI generated code. Now, I think developers would be foolish to do that. Good code is good code, whether authored by a fleshy mammalian brain or a mechanical process. And it is undeniable that AI code can be good code.

The Experiment

This makes me wonder if the job of maintainer could be replaced with AI.

I want to propose an experiment…

Let’s create a repository with some initial AI generated code: “Photoshop, but easier to use” is as a starting point as good as any. An AI agent will review issues, respond via comments, and may tag the issue with “todo” or close it if it doesn’t reach a bar for relevance and quality.

PRs are accepted for “todo” issues and will be reviewed, discussed, and ultimately merged or closed by the AI. These PRs may be human or AI generated—the AI doesn’t care (as if it could).

Note that PRs could modify any of the prompts used by the AI, and those edits will be reviewed by the AI in the same way as any other file.

Would the end result be quality software or a heinous abomination, succeeding only in creating a honeypot for prompt-injection attacks?

I have no intention of making this happen. But if somebody does, tell me how it goes.

Skills which I have used to create a terminal app that offers a more pleasant experience for agentic coding. Toad (a play on Textual Code) is a front-end for AI tools such as OpenHands, Claude Code, Gemini CLI, and many more. All of which run seamlessly under a single terminal UI, thanks to the ACP protocol.

At the time of writing, Toad supports 12 agent CLIs, and I expect many more to come online soon.

Here’s a screenshot:

So what does Toad offer over the CLI apps from big tech?

It has most of the UI interactions users have come to expect from agentic coding, but hopefully more refined. For instance the “@” character to bring in files into the context. Here’s Toad’s implementation:

A snappy fuzzy search which filters patterns from the project’s .gitignore (if there is one).

The prompt editor offers an experience which you might be surprised to find in a terminal. You can navigate and select with the keyboard and mouse, select, cut, copy, paste, etc. The prompt will highlight Markdown as you type (even syntax highlighting code fences before you close them).

Toad has really nice Markdown streaming, based on the techniques I described here. It remains fast with large documents, and renders everything from tables to syntax highlighted code fences.

Many other tools either don’t bother to render the Markdown, or they do a fairly half-hearted job.

Another goal I had for Toad was to integrate a shell. I wanted the conversation with AI to feel like a natural extension of a traditional terminal based workflow.

Most tools stop at displaying monochrome output from commands. Some will break if you run something interactive, like a TUI. Toad doesn’t have this limitation, and will let you run all your CLI apps with full color, interactivity, and mouse support.

At the time of writing the only terminal based agentic coding tool I know of that runs dynamic commands inline is Gemini.

Toad adopts the convention of using a ! character to introduce a shell command. There is also a list of commands in settings which will automatically trigger shell mode. In practice, this means that you rarely need to explicitly introduce shell commands—just type what’s on your mind.

Toad borrows tab completion from the shell. You’ll appreciate this if you have worked in the terminal long enough to commit this interaction to muscle memory. Hit tab to complete the command or path. If there is more than one possibility you can hit tab again to cycle through them, and enter to accept.

In addition to the shell, Toad implements a few concepts from Jupyter notebooks. You can cursor through previous conversation, moving a logical block at a time, and interact with it again. At the moment that feature is used as a convenience to copy content to the clipboard or prompt, and a few other niceties like exporting a SVG.

Toad will lean more heavily in to this kind of interaction in the future.

Friends of Toad

I was very fortunate to collaborate with OpenHands, who are doing some amazing work in this space. Check out their blog post on Toad!

I also collaborated with Hugging Face on this release. Check out their blog post on their inference explorers!

Try Toad

When this post is live you will be able to install Toad yourself.

The work is ongoing: a few missing features and interface improvements to be done, but Toad is solid enough to use as your daily driver for AI. I used it to create batrachian.ai, where you will find install instructions.

For more details, see the Toad repository.

I need a break (sabbaticals are tiring), but I’ll be picking things up in 2026. I’m hoping that by the time my year off ends, Toad could become my full-time gig. If you want to help make that happen, consider sponsoring my work.

To date, most of the my work on Toad has been focussed on building the “conversation” view where the user can converse and interact with their chosen AI agent. This has come together rather well, with a polished enough user experience for Hugging Face to start building integrations with their technology! And all before Toad’s official release.

The conversation view can be considered the body of Toad. But a body needs a head. In the case of Toad, it needs an interface where the user can find, install, and launch agents. Which is what Ive been working on recently.

The following will be the first view the user will see if they launch toad with no arguments:

From here the user can navigate the (growing) list of compatible agents. When they select one of interest, Toad will pop up a dialog with information (in beautiful markdown) regarding the agent.

In this dialog is a pull down list of possible actions associated with the agent—where there will typically be at least an “install” action.

Clicking the “Go” button downloads and installs the agent without ever leaving the app:

When an agent is installed, it is placed in the “quick launch” area. Each entry in the quick launch has a digit associated, so they can jump straight to that agent.

When an agent is selected (quick launch or other), the user can press space to launch the conversation view and begin their agentic coding session.

I wanted the experience to be as friction free as possible. There are many coding agents out there, but not all of them have the same visibility as those offered by big tech. I’m hoping this will promote the less well funded agents.

Here’s a video a recorded of the installation process. The interface is a little older than the screenshots, but you should get the idea…

Found this interesting?

Follow me on the socials where I will be posting regular updates. You can also join the Textual Discord Server if you want to chat with me or the Textual community.

Join the #toad channel if you would like an invite to the Toad repository. I’ll be sending more out in a week or two.

Thanks for reading!

I have no doubt that you are smart enough not to take medical advice from some random nerd on the internet, but let me say it anyway: none of the following is medical advice. I want to share my experiences and perspectives in addition to a vibe-coded tool, but I’m not trying to convince you of anything. Please don’t take my word for anything health related—talk to a medical professional first!

Got it? Good.

One of my main goals from my year off was to lose weight, which had been creeping upwards over the years. When I started my sabbatical in June I had reached 100kg which pushed me just over the threshold from overweight to obese.

Of course this shocked me. Doubly so, because I didn’t feel that big. If I were to walk out of my house on any given day I would pass plenty of other folk with much larger waistlines than mine. Living in a country 🏴󠁧󠁢󠁳󠁣󠁴󠁿 were deep frying is a religion and “juice” doesn’t contain fruit, clearly gave me a false perspective.

Scottish food

Although I didn’t feel so large, I was very much aware of a number of weight related health issues dragging my quality of life downwards. For years I had managed these issue, but I could see they weren’t going to improve without losing weight.

There was a time (~10 years ago) when I was in good shape and my weight was stable. I attributed this to running. A 5km three times a week and the occasional 10km plus some resistance training meant I could eat more-or-less what I wanted. The turning point was when I developed a herniated disc. An hour’s run would be followed by 48 hours of constant back pain.

Incidentally, a herniated disc must be the dumbest flaw in the human body. The human spine is a miraculous thing, but the “discs” (essentially washers that separate vertebrae) are prone to (literal) bursting. Around 50% of the population have a herniated disc by the age of 50. Most cause no symptoms, but in some people (like myself) the bulging disc presses on a nerve, causing pain. If there were a git repository for the human body, I would surely file a passive-aggressive issue. I might remark on that issue how it was clearly a poor choice to install a component with 50 year life span into something expected to last at least 80 years.

Exercises recommended by a physio didn’t help (they actually made it worse). Heat packs, cold packs, ibuprofen, and massage offered only short term relief.

When I stopped running for a few days the pain became an occasional sharp piercing pain rather than a constant ache. Not great, but bearable.

And so ended my running career.

I replaced the running with nothing at all. Other than walking (more as my default method of transport) I didn’t really exercise. I can’t attribute this entirely to the back pain. There are several forms of exercise that don’t inflame the back pain, such as swimming. I just didn’t find anything that I enjoyed doing regularly in the same way as running.

My diet is fairly healthy—for a Scot at least. I suspect I am in the top 0.1% percentile in the country for eating vegetables (and I’m not counting potato). My downfall is more sweet foods, which I typically indulge in my morning pastry and coffee. The rest of the day I try to eat healthy. I watch portion size and routinely deny myself things I would enjoy eating (trust me: if I ate all the sweet things I wanted to, my diet would be 90% cinnamon buns).

Alas, this wasn’t enough to keep the weight down. The scales don’t lie, and they were telling me I was getting heaver over time, which meant I was consuming more calories than I was expending. But how many more calories was I eating?

Turns out that is easy to calculate. There are approximately 7,700 calories in 1kg of human fat (eww—try not to imagine 1kg of fat). I had gained 8.5kg in 1400 days, which works out as a ~47 calorie excess per day. Around half an apple’s worth.

Half an apple? Two or three additional mouthfuls a day is all it takes to put you on a trajectory to being overweight / obese and all the health issues that entails. Yeah, well that was an eye opener.

If you are gaining or losing weight yourself, you might want to try this tool I vide-coded with Toad and Claude:

So the weight gain is no mystery—I was simply eating to excess. The solution was also no mystery. I would have have to stop consuming that additional 47 calories a day to stabilize my weight. And then reduce it further to reverse the trend.

If I were to subtract another 47 calories in addition to the 47 excess calories, then I would lose weight at the same rate I had previously been gaining. In other words, eating 94 calories less a day I would return to my healthy weight (~80kg) in about 9 years.

Screw that.

I have a year off to focus on my health, not 9 years. Losing 20kg in a year would require a 422kcal deficit, on top of the 47 excess I had been eating.

That’s quite doable with healthier food choices and exercise. I could give up my morning pastry and take up swimming. But its hard to avoid hearing about weight loss medications, and after looking into it I was tempted. Ultimately I settled on Mounjaro as it is reported to lead to the greatest weight loss.

The medication comes in a format described as “pen”. In reality of course it is a syringe, albeit a cleverly designed one designed to deliver 4 measured doses.

I don’t have a problem with needles but I was a little apprehensive about injecting myself. I needn’t have been. The needle is tiny (less than 1cm and almost as thin as a human hair), and only needs to go just under the skin to a fat layer. You bring it down in a stabbing motion and press a button at the top of the “pen” until it delivers the dose.

In the first week it was immediately apparent that the medication was doing something. It did suppress my appetite, but in those first weeks I consciously didn’t eat much at all as any meal resulted in gastric symptoms. I’ll spare you the details, but that first month the weight loss was dramatic. The second month was a little better and I could really appreciate the reduction in my appetite.

My wife and I have a running joke. When I’m hungry I tell her “if I don’t eat in the next 15 minutes I am going to die”. She replies “give it a try”. Well I never did give it a try until recently and it turns out she was right. The sensation of hunger was far less urgent, and I could put it off for way more than 15 minutes. And when I did eat, I felt like I was done after a very small portion.

Paradoxically I also found that my energy levels were positively brimming during the first two months. From morning until evening I had loads of energy and of course I wasn’t suffering from a post-lunch slump or evening sluggishness. I was still suffering from stomach issues, but the increase in energy and clear weight-loss felt like fair compensation.

The stomach issues improved somewhat but were relatively constant throughout the whole experience. Alas the high energy levels disappeared after two months or so. The weight loss was rapid, averaging 1kg a week, which is borderline concerning. I always ate to my appetite and intentionally chose nutritious high-energy foods, hoping to slow down the weight loss just a bit. But I just couldn’t face eating after a point.

At about 3 months or so after starting Mounjaro I started experiencing another symptom: frequent head-rushes. I would stand up from a sitting or laying position and instantly feel woozy. Often leaning against the wall in case I blacked out (I never did). I would be back to normal after 30 seconds, but this occurred often enough to be a worry.

This turned out to be only indirectly related to the Mounjaro. I had been on blood pressure medication for a couple of years, and with my blood pressure coming down naturally from the weight loss the medication was too effective. My GP reduced the medication by half and monitored my blood pressure for another month. The head-rushes abated and I eventually come off the blood pressure medication altogether. What a result!

My back pain has also improved. It hasn’t gone away, but the frequency and intensity of bad-back days has reduced remarkably. If that wasn’t enough, I haven’t had a single bout of GORD since losing the weight.

It’s now 5 months on since I started Mounjaro, and I have taken my last dose. I lost just over 20kg which puts me in my ideal weight bracket. It has been no cake-walk, but I consider the months-long stomach ache to be worth it for the health benefits it has delivered. I’m looking forward to enjoying food again, but I’m highly motivated not to eat the 47 calories a day that made me obese.

Reflecting on the wider impacts of Mounjaro and similar drugs. These must surely be the largest change to public health since anti-biotics. Many of the conditions that kill people in their 50s onwards are made worse if not outright caused by being overweight or obese. If everyone can get their weight under control it will dramatically reduce rates of diabetes, heart disease, stroke, and cancer. We will all be living longer more productive lives as a result.

Let me finish by re-iterating that I’m not giving health advice. Talk to a doctor if you are considering taking any medication, not a Python developer.

Things are moving (hopping) fast in the Toad world. I recently implemented the Agent Client Protocol in Toad, which means that Toad can now act as a front-end for Gemini, Claude, Codex, and a variety of other agentic coding solutions.

This is very exciting, as you can now use all these services from a single terminal interface. And the UX is already leaps and hops ahead of big tech’s CLIs.

Here is a video I recorded of Toad running Claude and Gemini head-to-head.

Welcome OpenHands

I’m very happy to welcome OpenHands as my first corporate sponsor! These guys are pioneers in agentic coding, and I look forward to getting OpenHands running on Toad.

Toad’s shell

Claude tells me that there is no species of toad that has a shell. But Toad with a capital T does.

Having an integrated shell within agentic coding tools just makes sense. But if Toad’s shell can’t match the basic features developers have developed muscle memory for, then it may as well not have a shell at all. So in the last week I have added a number of traditional shell features.

The first is history: The ability to navigate through your recent shell commands and potentially re-use one. Pressing up or down in Toad’s prompt will move through the history. Here it is in action:

Toad will offer to auto-complete commands from the history (in addition to some pre-configured favorites). Hitting return will accept the suggestion, while escape will dismiss it.

Here is Toad auto-completing shell commands:

Finally, still in the realm of auto-completion, Toad will auto-complete filenames and paths. If you hit tab, then Toad will look for any matching files or directories. If it finds a single file, it will insert that filename. If it finds more than one, it will insert any common characters and suggest the remaining characters in the filename. You can hit tab again to cycle through those suggestions, and return or escape to accept or cancel.

Here is path auto-completion in action:

Found this interesting?

Follow me on the socials where I will be posting regular updates. You can also join the Textual Discord Server if you want to chat with me or the Textual community.

Join the #toad channel if you would like an invite to the Toad repository. I’ll be sending more out in a week or two.

Thanks for reading!

Toads are a common name given to certain species of frog, which are characterized by dry leathery skin, short legs, and a bumps on their back. But “toad” is not a biological classification; two toads of different species may not be more closely related that a frog and a toad. A possible exception would be the Bufonidae family in which all members are known as toads (although some may also be called frogs).

Toads are long lived

Toads can be surprisingly long lived, reaching 10-12 years in the wild. While others have been recorded living 30 years in captivity.

Toad has Agent Client Protocol support

The universal agentic coding application currently known as Toad has added support for Agent Client Protocol, developed by Zed industries.

Toad now provides an alternative terminal-based front-end to Gemini CLI, and Claude CLI, with no flicker and a greatly enhanced User Interface.

For further updates and chat about amphibians, join the #toad channel on the Textualize Discord Server.

To quickly recap: Toad is intended to be a universal front-end to AI services, which is entirely agnostic to the models and providers, while providing an altogether more humane user-experience.

Recently I’ve been working on the prompt input, which will be the focus of this post.

The Prompt Input

Like all good things in computing, Toad presents you with a flashing cursor and space to write something.

Toad’s text input follows a number of conventions established by agentic coding tools. There is plenty of room for innovation here, but if it ain’t broke don’t fix it — so the interactions will feel quite familiar to most devs (although I do intend to address a number of usability and quality of life issues left on the table).

The input is a competent text area with much of what you might expect from a browser or desktop app. You can select text with both the mouse and the cursor, cut, copy, paste, undo and redo.

The input highlights Markdown quite well. You can see this occurring in the previous animation, but it really comes in to its own when writing code fences where it can highlight a variety of different languages. In the future I can see code fences grow a bunch of editing features like smart indentation and LSP support.

The default mode for the input is for editing the “prompt” for the large language model. If you want to enter a shell command you can type a “!” or “$” to enter shell mode. The prompt symbol and the highlight color changes, so you know can be sure which mode you are in.

Additionally, there are a number of blessed commands (which will ultimately be editable in the config) that trigger shell mode. For the most part Toad will know wether you want to run a command or send a prompt to the AI. And if you ever do want to send something which looks like a shell command to the LLM, you can hit escape to cancel shell mode.

Typing a forward slash brings up the slash command auto-complete. You can navigate the available slash commands by moving the cursor through the list, or continue typing and have it narrow down the search, then right to insert the suggested command.

I haven’t implemented any slash commands as yet, but I expect this list to grow. I also plan on implementing the custom command system supported by other agentic coding tools.

Hitting the @ key brings up a list of all the files in the project. These can be refined by entering a fuzzy search, which matches paths with the letters you enter, but not necessarily contiguously. This is way more intuitive that in sounds: you can start typing the name of the file and it will typically put it at the top of the list after a few keystrokes, or a enter a few characters from elsewhere in the path to match a subdirectory.

There is also a tree view of the project docked to the left side of the screen. While not primarily designed to add paths to the prompt, it would be a nice feature to select a file in the tree and have it populate the prompt.

Found this interesting?

Follow me on the socials where I will be posting regular updates. You can also join the Textual Discord Server if you want to chat with me or the Textual community.

A few haven’t been so pleasant, and I have banned a subset of the users involved. If we exclude spam, I think the number of users I have banned over the years may still be a single figure.

There are three broad categories of banned users, listed below.

Spammers

I’m sure there is a place for links to sites that sell male enhancement pills, but my repositories are not it. Spam earns you an instant ban. This doesn’t happen all that often as GitHub is quite proactive about dealing with spam. Often I’ll get a notification, but by the time I open it the spammer’s account will have been deleted.

As well as the usual spam nonsense, I’ve had some other weird stuff posted on my repositories. I recall one user that posted lengthy conspiracy theories, with something about terminals intertwined. I’m pretty sure this user wasn’t a spammer in the traditional sense, and was suffering from psychological issues. I had no choice but to ban them, but I genuinely hope they found treatment for their condition.

Venting

A venting post almost always starts with “I spent X hours / days on this”. Such users want me to know how much I have inconvenienced them and they are rarely genuine in asking for help.

I try to be generous, but that phrase tends to put me on the defensive. I’m not going to treat the issue as a priority, and when I do respond it will be a tad more snarky that usual.

Venting posts are almost always a skill issue in behalf of the user. The user has either misread the docs or not read them at all, and they have consequently made some invalid assumptions about how the API should work. They couldn’t get their code to work, because it was never intended to work in they way they were using it. Rather than stepping back to reconsider their approach, or read the docs again, they want to shift the blame to me.

Venting is not an instant ban, because I know how frustrating programming can be. But virtually 100% of these issues are resolved with a link to the docs, and none of reply with a “my bad” even if I suppress my snark.

So not an instant ban, but if venting crosses the line to personal abuse, then I’m going to be reaching for that ban button.

Almost everyone recognizes that an posting an issue is essentially asking a favor. If you want somebody to help you move house, you wouldn’t start by criticizing their driving. Same deal with issues.

Time wasting

This last category is trickier to define, as its not a single transgression like the others. It’s more of a pattern of behavior that sucks time that could be better spent helping other users or writing code.

So what constitutes time wasting?

It will often start with well meaning issues that are overly long. Pages and pages of text that don’t clearly describe the problem the user is tackling. In the end, the issue typically boils down to a perfectly legitimate “it crashes when I do this”. But it can take a lot of fruitless back and forth to get there.

One time I can overlook. But if it keeps happening, I can feel like I am essentially working for this user at the expense of other users.

Related, is the user who doesn’t listen or choses not to respond to my requests. Simple things like asking for the version of their OS or software they are using. Details that I need to properly investigate their issue. Sometimes it takes the form of ignoring a recommendation. I’ll let them know there is a canonical solution to that issue, and give them a short code snippet that resolves the issue with less work, but they won’t use it or even acknowledge it.

However the most common “not listening” issue is when I ask the user why they are attempting the thing they need help with. This can be vital in avoiding the XY Problem. If there is a better way of solving their problem then I can point them in the right direction. But only if they tell me. A few users have just refused to and repeatedly assert the odd thing they are trying to do is the only acceptable fix.

Other examples of time wasting include asking for LLM hallucinated code to work, posting lists of questions that can be answered with a skim over the docs, and posting the same questions in multiple support locations even after they have been answered (as though they will keep posting until they get a response they like)?

This is a tricky category, because the user can be well meaning. So its a very high threshold to be banned for this. I only consider it if the user is a clear net negative for the project.

Unbanning

I so rarely ban users, and when I do it’s for the good of the project. But I don’t hold grudges. Other than spammers, if anyone I have banned wants to contribute I am happy to un-ban if they reach out. No apology required, but I will require they avoid the ban categories above.

Open Source is awesome

I hope this post doesn’t sound too much like whinging.

I’d like to end by stressing that the community is what makes Open Source awesome. Everyone benefits by contributing in their own way.

When talking to an LLM via an API, the Markdown doesn’t arrive all at once. Rather you get fragments of markdown (known as tokens) which should be appended to an existing document. Until recently the only way to render this in Textual was to remove the Markdown widget and add it again with the updated markdown. This worked, but it would get slower to append content as the document grew. It wasn’t a scalable solution.

Fortunately there are a number of optimizations which made markdown streaming scalable to massive documents. I would expect these tricks to be equally applicable in the browser.

Here’s Textual’s new Markdown streaming in action:

In a Textual Markdown widget, every part of the output it also a widget. In other words, every paragraph, code fence, and table is a independent widget in its own right, with its own event loop. Since the bottleneck was adding and removing these widgets, any solution would have to avoid or dramatically reduce the number of times that needed to occur.

Optimization 1.

The Python library I use for Markdown parsing, markdown-it-py, doesn’t support any kind of streaming. This turned out to be a non-issue as it is possible to build streaming on top of it (and probably any Markdown library). Markdown documents can be neatly divided in to top-level blocks, like a header, paragraph, code fence, table etc. When you add to the document, only the very last block can change. You can consider the blocks prior to the last to be finalized.

This observation lead me to working on an optimization to avoid removing and re-creating these finalized blocks. But there was a sticking point: the last block can change its type when you add new content. Consider a table where the first tokens add the headers to the table. The parser considers that text to be a simple paragraph block up until the entire row has arrived, and then all-of-a-sudden the paragraph becomes a table. Once I took that into account, it worked. It was a massive win and streaming became more practical.

Optimization 2.

The next step was to avoid replacing even the last widget on new content. This is unavoidable if the last block changes type, but if it didn’t, I could add new content without replacing the widget (a far simpler operation in Textual). The paragraph block, for instance, was trivial to update. As was the code fence. The table was more complicated, but still doable. Replacing even a single widget per token could be expensive when new tokens arrive 100 times a second or higher. So this update was a decent win.

Optimization 3.

I could possibly have left it there, but there was one more outstanding optimization. Markdown-it-py is an excellent library and really quite fast, but a very large document could take a few milliseconds to parse. Multiply that by 100 tokens a second and it becomes significant.

I could reduce that parsing cost dramatically by only considering the last block in the document. All it took was to store the line number where that last block began, and feed the parser the data from there to the end of the document. This update meant that no matter how large the document is, parsing was always sub 1ms.

Optimization 4.

This final optimization occurs not within the Markdown widget itself, but at a level above.

No matter how optimized appending to the markdown widget is, tokens could arrive faster than they can be displayed. A naive solution would queue up these updates, so you see all the intermediate steps. The problem with that is that your UI can be dutifully scrolling through content for many seconds after it has arrived. I say if you have the output, let the user see it without delay. They probably paid for the tokens, after all.

I fixed this with a buffer between the producer (the LLM) and the consumer (the Markdown widget). When new tokens arrive before the previous update has finished, they are concatenated and stored until the widget is ready for them. The end result is that the display is only ever a few milliseconds behind the data itself.

Get the code

This work will shortly land in the Textual repository. I would expect this to be a common occurrence working on Toad. If it belongs in the core library it goes in the core library, so expect some Toad features being available in Textual prior to the release of Toad itself.