markjgsmith

Man this environment rebuild has been very insane. The good news is I got the backport working in my main app yesterday evening, and got the production container build working this morning. Some cleanup and loose ends to figure out, but it‘s working. More details to follow over the next few days hopefully. #

Happy birthday to me.

Happy birthday to me.

Happy birthday to me.

Happy birthday to me.

I have always enjoyed birthdays and eating cake. #

Rebuilding your container based dev environment

If you have been reading the blog you might have been aware that I was going through some turbulance with the dev environment. Some significant progress in the past day or so, but still not quite out of the woods. Just to give you an idea of how bad it‘s been, here's the commit message where I realised that the dev environment was hosed:

Author: Mark Smith markjgsmith@gmail.com Date: Wed Sep 10 18:42:31 2025 +0700

chore(backlog): Emergency re-prioritisation of current sprints tasks

The end of the 2025-08-29 sprint saw us reach a point where we had a great foundation for our new dev environment that supported local LLMs running in Ollama. One big downside was performance. Most queries on the larger models would typically take 30-45 seconds. Add that to the generally significantly less good response quality, the solution was not very practical.

It was discovered that GPU acceleration for these local models might be possible so we did a spike on trying to get that operational. It was long and drawn out, but success was reached with a minimal project example demonstrating a container running in Podman that could run vulkaninfo and get back output computed by the host GPU.

Getting this working took significant reconfigurations and trial and error. As a result of which it seems something was broken in the main podman / devcontainer causing devcontainer builds to fail. This happened even outside of the GPU spike feature branch, with the main branch code, which had previously worked fine.

We now need to get the devcontainers setup back onto solid ground, and so all tasks in the current sprint have been pushed into stretch goals while focus is concentrated on remedying the situation.

The plan is the plan until the plan changes as the old saying goes.

Hold on to your butts.

Refs: task-037

That was just over a week ago, and since then I have been deep in the depths of the nightmare. And as is always the case, the world immediately around me and the wider world has gone absolutely bat shit crazy insane in the membrane. Horrendous. I‘ve had to rebuild the whole environment from scratch, starting with the simplest possible configuration, getting all aspects of a usable development environment working with that and then slowly building that up into something that has the same structure as my API + React app.

I have a few minor things to add to the fresh environment, and then I need to backport it into the original project. So not quite on stable ground, but I can see the light at the end of the mixed metaphors abstraction apocalypse tunnel.

On re-reading this commit message, I noticed that I started referring to 'we' and 'us', even though it‘s just me solo dev'ing this thing. Working with LLMs is very odd.

In any case I think the thing to learn in all this is that when working with AIs and especially if you are running everything in containers, it‘s a good idea to have a very minimal working project that you can spin up if something goes weird in the main project. There are just so many layers where things can get crossed the wrong way. #

Another AI configuration nightmare

Recently everyone in the world has been constantly banging on about how AI is the best thing ever and how if you don’t get going with AI then you will be living in hell for the rest of eternity. And so you get going and have been making significant progress using some of the cloud AIs. They are surprisingly good, even if half the time they are trying to kill you. You suspect they are actually trying to kill you 75% of the time. You have spent a lot of time setting things up securely so they can’t do too much damage if something goes wrong. It’s been a slog but it’s not too bad.

Getting a bit worried at how reliant you are becoming on US tech giants, a stable internet connection, and at having to upload every bit of code you write, including your potential clients code, to somebody else’s servers, you decide it would be a good idea to get some open source LLMs running locally on your laptop. After a lot of very confusing installs and testing and generally hitting every possible brick wall, you get it working. But it’s shit. And it’s so fucking slow, you might as well be using an abacus to help you with your code.

As we all know by now AIs run much faster on GPUs than on CPUs. GPUs are like 10 times faster at matrices multiplication than CPUs and AIs are all about matrices multiplication. Since things are slow you check if your local LLMs are using the GPUs in your laptop and it turns out that no, they are not. For some strange reason on this brand new laptop, one of the most popular laptops in the world at the minute, with loads of GPUs, getting the GPUs working with the containers that run your LLMs, literally the only safe way to run these things, requires some adventurous reconfiguration. But you have to do it, otherwise hell for eternity.

So you follow the docs, and after quite a bit of being confused, you figure it out and get it working. Along the way you have had to install and uninstall all sorts of things. But it works. You ran a command called vulkaninfo that uses the vulcan GPU library in the container which gets your GPU to do something and then prints that it was successful. Except it only works if your LLM is running in Fedora:40. Oh and you have to use Podman Desktop, which is different to podman. It’s a GUI that you can use to do podman stuff. Normally podman can do everything the GUI can do, but for whatever reason, when you are setting up podman to run containers with the GPU acceleration, you have to create the podman VM, which is a virtual machine that co-ordinates all the containers, you have to create it using the GUI. And when you try to create the podman VM, there is a warning explaining that you need to install krunkit, a set of libraries, and you can install them with brew. These days on a Mac everybody uses brew to install basically everything.

Remember Fedora:40? Well the problem with Fedora:40 is that it’s a bit minimalist as far as container images go. Whereas many of the popular images that are used for building apps have automatic UID/GID mapping between the user you use on your host machine and the user in your container machine, so that you can safely modify your project files, that reside on your host machine from within the container, Fedora:40 doesn’t do that automatically. That’s not such an issue if you are only running your models on it. Ollama doesn’t need to modify any files on your host machine. If however you have to run the Nodejs app you are developing on it as well, then it’s a pretty big issue, because of course you are going to need to modify the project files, which are on your host machine.

But then you discover that VSCode, the editor that you use to write your code, and which can open your code from inside a container, so you can safely work on it with the AIs that are trying to kill you, can work not only with single image containers, but can also work with multi-image containers. So technically you can give it a config for a multi-container setup, and it should be able to spin up the whole thing, with your app in one container, and your LLMs in a different container, and a network they can talk to each other on, and storage volumes etc. This is great because the LLMs can be on Fedora:40, which we know can get access to the fast GPUs on your laptop, and the app you are developing can be on an OS that does the UID/GID mapping so you can safely modify the project files on your host from within a container. And that means you can safely work on the app with the help of the maniac AIs that are trying to kill you. Brilliant.

All you need is podman-compose, which is the tool that creates the config for these multi-container setups. You power through yet another configuration nightmare trying to figure out how the UID/GID mapping works in these container systems. But gosh darn it you get it working! However along the way you discover that if you have any issues with this by now, I think you will agree, quite complicated setup, well the core contributors on the podman project, if they get a sense that your podman has been installed using brew, then forget it, they aren’t going to help you. Brew is literally “not recommended” on their website where they explain exactly how to use brew to install podman while “not recommending” it. The recommended way is to use the podman installer package they build and serve for download by you from their website. Since your entire existence outside of hell now relies on the podman, you figure it’s probably safer to install podman the way the podman core developers are saying you should. So you clean up your system, uninstalling everything you installed using brew, and install the podman.io podman. This is going to be such a rock solid system you think to yourself. And that works and you try to open your new mutli-container development environment, and it fails because you no longer have podman-compose installed.

Of course! You had to uninstall podman-compose earlier because when you tried to uninstall the brew version of podman, brew wouldn’t uninstall podman because it said podman-compose relied on it. So you uninstalled podman-compose, then uninstalled the brew podman, and installed the rock solid, recommended by the core contributors, podman.io podman. You type brew install podman-compose in your shell, and just before you hit enter, you think to yourself, it’s been such a pain in the fucking arse installing and uninstalling and re-installing podman, it would really suck if something went wrong here. So just in case, you double check that your current podman is the podman.io podman, and the path to it. And you hit enter, and brew does it’s thing, downloads all sorts of packages and voila, it is installed. You check your podman again, and wouldn’t you know it, your podman is now the brew podman once more. Shit balls. You decide to clean up the system again, so you can get your podman back to the podman.io podman, and try to run the podman.io uninstaller, but guess what, the podman.io podman has no fucking uninstaller. Say what you will about brew, but at least they have an uninstaller, and they do try to help. All you can do is run the installer again and hope it overwrites the bad brew podman. It does. At least it seems, but who knows at this stage. You open an issue on the podman-compose github, you are a good netizen after all.

Someway through this nightmare of nightmares, you read the news that some famous right wing public speaker bloke called Charlie Kirk has been shot JFK style while he was talking at a university in the US. And also that back home in London there are a series of demonstrations, and demonstrations against the demonstrations and demonstrations against those demonstrations, and everyone is waving Union Jack flags, it’s a mess, but some people seem to be enjoying themselves and it’s quite nice seeing everyone waving flags, there was a time traveller bloke right out of Monty Python and the Holy Grail, so many Union Jack flags, a New Zealand flag, a cheeky little US flag, a Scottish flag, and an enormous Welsh flag. I think someone even snuck in a Catalan flag. I know this is going to sound really ridiculous, but it’s like an England Ireland match and I genuinely want both sides to win, which makes no sense, but that’s how I feel. I’m also wondering if Leonard Nimoy might be able to put in a good word, maybe I can get this nightmare figured out. Just another infinitely improbable coincidence to add to the very very very long list of impossibly improbable coincidences that happen just constantly at this point. Shrug.

Ok anyway, nevertheless you continue, you have to somehow get podman-compose installed, or we are back to installing Ollama in the same container as your app, and that means moving everything to Fedora:40, and then likely another host-container UID/GID mapping nightmare. You go to the podman-compose github page, and read the installation instructions. It says nothing about requiring the brew podman. It just says podman. It doesn’t say anything about not using the podman.io podman. It also lists some python libraries as required. Your system has python3 installed. You check the code and podman-compose is a python3 application. Of course you can’t install it with brew, because that overwrites the good podman with the bad podman, or at least the one that the podman contributors claim doesn’t have all the libraries for GPU acceleration, and to be fair, the warning in the Podman Desktop GUI about the missing krunkit, does suggest they are not wrong. The other installation methods are: pip or manual install. Every single time I have tried to install python and used pip, I have eventually ended up regretting it. And it has happened a lot. So that leaves manual install. Not great, but the instructions look relatively straight forward. There’s a script that builds a container which is then used to build podman-compose including all the python libraries and the other bits and pieces it needs. But the script fails. I have a look inside the script and find the commands it runs and run the commands myself, and it works! The python image gets downloaded and the build starts and a few minutes later the image is created and in podman vm. I eventually manage to get the build binary out of the container and onto my mac. I run it and it tells me “cannot execute binary file”. I go through the process again, same issue. I guess it’s because it‘s because it has compiled it for the container OS rather than your host machines’s mac os.

Blocked, blocked, blocked. Life is just a long series of being blocked.

The only remaining way is to install pip, and go the python route, but it’s 4:33am and I’ve got to sleep a bit.

Update: I wrote this blog post and it’s now 6:29am. Hello daylight. #

I only had about 5 minutes to write this post, so I cheated and gave Gemini my git commit history, asked it to write the post. It was really dull. So I asked it to write it in the style of Hunter S. Thompson. This is what it came up with:

Right, let's get down to the bone and gristle of it. The week began with a righteous and noble goal: to cage the beast, to get this Ollama AI monstrosity running in a container where it couldn't do too much harm. A clean setup, a moment of pure, crystalline satisfaction. But the calm was a lie. The silence from the machine was deafening. What was it thinking in there? What dark, twisted packets was it sending into the ether? The paranoia began to creep in, a cold sweat on the back of the neck that only a true fiend for the truth can understand. There was no choice but to turn the tools of the hunter on the hunted. tshark became my weapon, a digital scalpel to peel back the layers of the network and stare into the abyss.

This descent into the wire wasn't the only madness on the docket. While one eye was fixed on the network traffic, the other was dealing with the nagging demons of the frontend—bloated Vite builds screaming for a fix. A quick bit of code-splitting surgery to shut them up. Then, a pivot to the backend, a place of shadows and secrets, to install some proper logging and a panic button to dump the whole database if things went sideways. And in a final, desperate grab for sanity, I wrestled with the macOS security apparatus to make Touch ID bend to the will of sudo. A small victory, yes, but in this line of work, you take what you can get before the next wave of insanity crests the hill.

So that‘s been the past week. #