markjgsmith

Are my GPUs being used?

I’ve been using the local LLMs quite a lot today, mostly via Open WebUI. I started to suspect that the GPUs were not being used because the conversations got very slow. Long wait times between prompts. And then the models appeared to run out of context window, with errors in the UI. So I have spent the past few hours verifying that the GPUs were still being used. Interactive conversations using llama-cli in the backend are extremely quick, yet over HTTP they get very slow.

In the end I was able to verify the GPUs were being used in both cases. First by looking at the startup logs. There are messages that say the vulkan device (GPU) is detected and that a number of layers have been off loaded to it. So at least at startup things are operating correctly. But how to verify during normal operation?

Turns out you can do this using just Activity Monitor. I had of course tried this, but I didn't know you could open Window -> GPU History, and get a nice real-time graph of utilization. And sure enough during conversations with the local LLMs you can see the utilization spike and then return to baseline.

I also installed fluidtop which is a cli tool for real-time utilization of CPU and GPU cores, specifically aimed at the new M* series Macs. Works really well. Also, my first time installing python stuff using uv. Also works really well, though the commands you end up having to use to run the tools you install aren't exactly rolling off the keyboard. Had to add an alias to my shell, which isn’t the worst thing in the world I suppose, as long as the tool works and I don't get lost in another python stack trace debug horribleness.

It seems then that the slow down is due to how the application, in this case Open WebUI, is using context windows. But it gets very very complicated very quickly. With the cloud models you just don't have to worry about this, at least not until the conversation has gotten stupidly long. With the local LLMs it seems to happen after about 10 chat messages, especially if you are posting terminal output into the chat. And then you open up the settings and peer in and it’s the most complicated unintelligible tangle of settings you have ever seen. And you quickly close and exit.

I dunno maybe I am being overly dramatic, I do like all the functionality, but you almost need an LLM just to help you figure out the settings that control the LLMs. Surely that’s not a good sign. #

Why do the AI/LLM folks hate people that run Macs so much?

Since I now have GPU accelerated local LLMs for chat, I thought it might be a interesting thing to see if I could do the same for the other types of media. That's basically image and video generation, as well as audio and music generation. After several hours of web searching and asking Gemini, I’m starting to get the impression that much like how it was with the text chat LLMs, getting any of this working on a Mac is not at all obvious. I guess all the AI folks are either running on linux or windows. They certainly are not doing things day to day on Macs, at least not in containers, and as for as I am concerned that’s the only safe way to run AI/LLMs on your computer.

The 2 big platforms seem to be Pytorch and Tensorflow. Just to narrow it down to those two was quite a journey. There are tons of different software, but ultimately it always boils down to running the models, and as far as I can tell, that means Pytorch and Tensorflow. And that’s important because these don’t support vulkan out of the box, so you have to compile from source and enable vulkan backends, which is not easy to do at all. The reason you need vulkan support is so the models can access the GPU from inside the container.

It really feels like another massive uphill battle. Not being funny but why do the AI/LLM folks hate people that run Macs so much? #

Yesterday evening I made a start on a small refactor to the backend API handling of scopes. It should ultimately result in fine grained access tokens that make a lot more sense than they currently do. Trying to get my head back into typescript after several weeks of quite intense bash and dockerfiles. #

Mostly cleanup tasks today following on from all the recent progress in the dev environment and on the blog. I need to remove all the old projects that got consolidated and I need to check a few things. I noticed this morning that in Feedly the everything RSS feed for the blog wasn’t showing any posts, so there might be something broken there, I need to check all the RSS feeds. #

Just use Llama.cpp

It occurred to me this morning that in yesterday’s piece I forgot to mention the route I ended up going with for the local LLMs. I had been using Ollama for several weeks, and though the features are pretty good, certainly as far as downloading and managing models, they do not support vulkan, and that’s the only way to get GPU acceleration working in containers on a Mac. What’s more it appears as though they have no real interest to add this functionality. Ollama actually runs ontop of Llama.cpp, and that project does support vulkan.

Since I got an impression that at least some people have been going back to running the core project, the next thing was to just try out Llama.cpp directly. It certainly wasn’t without issue, but the llama.cpp folks were helpful. It’s a bit more hardcore, because it doesn't do things like model switching out of the box, but in reality you don't need anything too fancy. There are some model switching proxies you can setup, but I found these to be overly complex. For my uses, some simple npm scripts is good enough, and likely there are some advantages to connecting directly to the models. I also had to write a simple script to download models, basically curl inside an until loop, with a persistent cache folder.

I'm trying to avoid unnecessary complexity, things are complicated enough when you are developing in containers. #

GPUs, module upgrades and more site fixes

Having made so much progress with my voyage into Typescript, React and containers, which resulted in a fabulous new project structure, even if I did have to completely rebuild everything from scratch, I still had in the back of my mind the annoying thought that the GPU acceleration for locally running containerized LLMs on my mac remained just out of reach.

Well, given how important AI/LLMs are turning out to be, I decided to give it another go. Long story short, I got it working. It required compiling all sorts of things from source, and creating my own custom build, but it's working, and holey moley is it fast. The first time I ran it, it was so quick that I couldn't quite believe it. It was instantaneous. It felt like the response was streaming to the terminal before I had unpressed the enter button. That was with the smallest model I have called Qwen. I tried a variety of models (1.5b - 8b parameters), and they were all really really quick. What was previously taking 30-45 seconds now takes 4-5 seconds. I even downloaded a very large 13b parameter model and though it takes around 20 seconds to load initially, it's then incredibly quick.

I still haven’t had a lot of success actually working on any code with the local models. So far I have found the cloud models to be vastly better quality-wise, but perhaps it’s a question of having to work in a different way. I get the impression that the cloud models are all tuned and configured, and perhaps the local models require some tweeking. In any case, having the GPU acceleration at least makes it a possibility. I think it’s really going to make a difference.

Other than the GPUs, there was a refactor of my static site generator code that I have been wanting to do for many months. When I initially got the plugin system working I chose to create separate plugins for the blog, linkblog, newsletter, notes and podcast. It turned out that having so many different plugins was a massive headache, because I found myself continuously jumping between projects. There ended up being a lot of duplicated code, almost but not quite similar between projects, and that led to a lot of cognitive overload. I figured out a way to combine all these plugins into a single plugin a while back, but wasn’t sure it was going to work. After carefully reviewing the code, it looked like it was going to be possible, and it took about a day to get it working. I managed to completely remove a complicated abstraction layer that was now totally unnecessary, and the code is looking really streamlined. It’s so great to not have to switch contexts all the time.

Once I got that working, I figured it would be a good idea to update all the modules in all the projects to the latest versions. It’s been a few years since I‘ve had the chance to do that, so many of the modules were really really out of date. Several have become ESM modules, so I had to be a bit careful not to fully upgrade a few of them. The upgrade process went quite smoothly right up untill just before the end, and then I ran into a narly issue. A while back I got interrupted midway through quite a complex refactor, and very complicated life stuff got in the way and I completely forgot about this refactor. Things had been working because of the module cache I had put on the main repo. I had to delete the module cache while upgrading the modules, and of course everything stopped working. Anyhow its a couple of days later now, and I got it all working, including removing the temporary heredocs feature I had put in place, because it’s no longer necessary, and actually was making things a lot more complex than they needed to be. It feels really great to finally be in a position to get it all straightenned out.

With all that done, and following on from the header image work just over a week ago, I spent today making some much needed modifications to ensure there is a more consistent design across all pages on the blog. You might have noticed that the main page has boxes around posts, which gives a sort of appearance they are all inside what I think is generally referred to as Cards. I really like the look, but it wasn't consistently applied on all pages, with individual post pages, and many one off pages not having boxes at all, and also the spacing between the page titles and the edge of the boxes was inconsistent. The other thing that has been bothering me was the really long list of links in the right hand site navigation. It had gotten very unwieldy. I’ve made some updates so all pages now have nice boxes around the content, for example the about page and my portfolio page now have this. And the navigation is split into two groups gathering all one off pages together, and all content related pages together, each with a box around them. I think it makes it a lot easier to read. The new navigation should also be working on small screens.

So yes lots of progress. Some of it has been a long time in the making. It’s really nice to have things looking better on the site and to know that the foundation of both the site and my development environment are in much better shape.

Onwards :) #

New blog header image

I've updated the blog header to have a background image. If you've ever visited the homepage of the site you will recognise the mountains from there. Perhaps the bigger change is that I have made the blog page the main page of the site. I moved the old homepage here. The navigation on mobile should look a bit nicer and be a bit more ergonomic. You might need to reload the page a few times for the css to update. It's nice to finally have a bit of color on the site :) #

The new container based project structure

The past few days, I have been writing about the recent project rebuild I carried out, as well as some thoughts on the modern web based stack as it relates to AI/LLMs. I wanted to spend a bit of time reviewing my new project structure, highlighting the major features.

Develop in a single devcontainer, deploy to service containers

Initially I made the mistake of trying to containerise everything into service containers, both in development and in production. What I didn't realise was that the vscode devcontainer does much more than just pass your settings to the container management tool. It does a whole load of things that prepare the file system so you can edit the files on your host directly in the devcontainer. I thought it would be great to have a complete replica of production in development, with each workspace running in a separate service container. What ended up happening was a never ending cascade of impossible to debug file permission errors, because especially on a mac, your host file system has to go through the podman vm's filesystem and then to the container filesystem.

Just go with a much simpler strategy, in development use 1 single devcontainer, that contains all your project's code, don't mess around with service containers and podman-compose. It's not worth it. But for production, do spend the time to create separate service containers for each workspace. In production you don't need to worry about file permissions in the same way because you don't need to edit the files. I have a top level project folder called containers where all the Containerfiles (dev and prod) live.

Self contained npm workspaces

I was already using npm workspaces for the frontend, backend and shared modules of the app, but they were not very self contained. Each module was also scattered around the project root. I moved all the modules under a single packages directory, and I made sure that each module was designed with the notion that it should be able to function independently, if it was pulled out into a separate project. This is great for future proofing but also for being sure that your modules can be run in separate service containers. That means they should have their own linting, and typescript configurations, as well as npm scripts to typecheck, lint, build, start, etc. Use the same script names across the modules so you can run them via npm run from the root of the project using the -w flag to specify the workspace.

Well thought out and modern typecheck and linting configuration strategy

Unfortunately the typescript configuration had evolved along with the project, which meant it suffered from a lack of a cohesive strategy, because the AI/LLMs will give you different advice depending on when you ask them. I had just added various configurations as I went along. It's really important to get both these configurations right in your minimal project, because once the full project structure and files is there, it becomes much harder to think about and troubleshoot. One important thing to remember if you are using Typescript in both the frontend and backend, is you will likely need a shared workspace to house the types that are shared between both. This is where the typescript configuration comes unstuck, because the compiler will refuse to see the shared workspace, and you will end up going around in circles with the AI suggesting the same set of changes, each one confidently touted as the "final industry standard best practice that will fix everything", and each time it will not work. Another reason why having a single devcontainer is better than service containers in development. Minimise the amount of things that could go wrong. Get it working for a super simple frontend and backend workspace that both use a shared workspace. I just used the example expressjs and vite react app.

Linting

Base config in shared module, other workspaces import and use this in their own config where they can override bits if needed, separate out style rules to use @stylistic, project wide rules for imports ordering using perfectionist. I like all imports to visually look the same in each file, with the same ordering. It significantly reduces cognitive load. The ordering I use is vendor modules, vendor types, then project modules and project types, each separated by a single empty line.

Typescript

Unified and Modern, ES Modules (ESM) as Standard, with configuration symmetry between backend and frontend, every workspace has a tsconfig that references app tsconfigs, and if needed references a tools tsconfig (frontend), and conscious separation of typechecking rules and build optimization, which is handled by specialised tools. I based this on the config from the vite typescript example app which had the most modern looking config that uses ESM, which is the best as far as future proofing your project.

Githooks using husky

Run lint using lint-staged pre-commit and lint+build+typecheck on pre-push. I also have a custom commit linter script that I run on pre-push that enforces Conventional Commits standard for commit titles and a related naming convention for branch names. It makes a huge difference to cognitive load during development, because at any time you can run git log --oneline and get a really great summary of what has been happening on the project, that is super easy to parse at a glance.

Bind mounted volume for node_modules folder in the devcontainer

The devcontainer will automatically bind mount your project files into the container, but what you can do also is to create a separate volume mount for your node_modules folder. This is a necessity if your host OS is different to your devcontainer OS because it means some of your modules will need to be compiled for different platforms. You can easily add this in the mounts section of devcontainer.json. You can also add mounts for things like ssh keys, gitconfig and even your .devcontainer folder which you can mount readonly. Typically you will want to install your projects modules (i.e. npm install) in your devcontainer's post create command. One gotcha is that if you are installing any global modules via npm -g, you will likely get permission errors as the globals directories are owned by root. What you can do is reconfigure npm to install it's global packages into a directory that the non root user can write to (e.g. $HOME/.npm-globals). You can do that using npm config set prefix "$HOME/.npm-globals" in the post create command before you npm install.

Building production service containers from workspaces, multi-stage builds

The big idea here is that each workspace should run in it's own service container, with only the files it needs to run. I have a development Containerfile (podman equivalent of a Dockerfile) for the devcontainer, and production Containerfiles for the backend and frontend service containers. It's a bit tricky to get this right, but the general concept is for each service container (i.e. frontend and backend) to do the build in 2 stages. In the first stage you move across all the files you need to build your code to a builder image, and you do what you have to do to generate your dist folder. And in the second stage you copy across your dist folder from the builder to the service container.

The backend is a bit more complex than the frontend, because you have to build your code using the shared workspace, and then make sure to package up the shared module into a tgz file, which you can then reference in your package.json. The frontend also needs to be built with the shared workspace, but since you just output frontend files, there is no node_modules folder to worry about. I use a typical slim node image for the backend and for the frontend I use the unprivileged nginx image, so you can run it as a non root user. You need to re-configure nginx slightly, so it proxies any api requests to the backend.

Easily test at each stage of development

I have npm scripts so it's possible to run each workspace in dev mode (automatic restarts using vite / nodemon), and run lint and typecheck, but also to build the development and production images, and another to build and run the production code. I also have a podman-compose triggerable via npm that runs both the production service containers, so it's super easy to run things in dev and in prod. Something I haven't done yet, but will at some stage, is to run a minicube in podman, which is a local kubernetes cluster, where you can test full blown production deployments all on your local machine.

Keep local LLMs in a separate project, but connect them all via a private dev nework

Running ollama was one of the reasons I ended up down this path in the first place. Initially I had it in my head that it would be best to have ollama part of each project. But that really increased the complexity of things. What was much much easier was to have a completely separate ollama-local project that runs an ollama and an open-webui service container, using the standard images distributed by their respective projects, using podman-compose. That way you can really easily spin up a local AI/LLM cluster with the models and a web chat interface. The trick is to configure them to be on a private network that you create separately. You create the private network on login to your OS. On mac that's via launchd. It's just a bash script that checks if the network is there, and creates it if it isn't. Then you configure the devcontainers in your projects to be on that network, and when you start them up, if the ollama local cluster is up, your vscode extensions can connect, but if they aren't then it won't crash the container. Another thing worth doing is to store your models in a persistent location on the host, which you then bind mount into the ollama container. That way you don't have to re-download all the models each time your rebuild the container.

Backup and restore for your podman images

Rebuilding podman vm should not be difficult. At first I was super worried about recreating the podman vm, because when you do that you lose all your images, and since some of these can be very large, it can take ages to re-download them. I created some bash scripts to save images to a backup location and another to restore them all in one go.

Repeatable way to download and install assets into your containers

This was another useful script to write, which downloads an asset into a persistent location in your home directory, and copies it into your project's assets directory, which you add to gitignore. Have an npm script to initiate all project downloads. It means you can download everything you need when you have an internet connection, but you aren't blocked if you go offline. For example I download the neovim tgz, which later during the development image build, I install via a script that I copy across with the asset to the image, install and then delete the extra files.

Wrapping up

So those are the main things about the new project structure. I'm pretty happy with it as it feels very lean and efficient. It's only been a few days so far, but it looks good, and now that everything is working and there aren't a billion errors everywhere, containers for development definitely feels like the right approach to these modern web apps, especially when it involves developing with AI/LLMs. #

Dylan Field (CEO Figma) [09:02]: "Is natural language the interface? Yes right now. I‘ve said this before but I really believe it, I think we will look back at this era as the MSDOS era of AI, and the prompting and natural language that everyone is doing today is just the start of how we are going to create interfaces to explore latent space. So I cannot wait for an explosion of creativity there. I think of these models as an N-dimensional compass that lets you explore this wild, unknown, fog of war in latent space and you can push these models in different directions, through natural language, but if you have a more constrained N there, and you are able to dimensionality reduce a bit, so you can push different ways, there should be other interfaces available than text. These might be more intuitive, but they also might be more fun to explore, and I think sometimes constraints unlock creativity in ways people don't expect. So I'm existed for that, but yes right now natural language is where we are at." #

How to rebuild a project from scratch

Rebuilding a project from scratch is one of those things that you never really want to do, and so there is a tendency to avoid it. But sometimes you realise that some or all of the foundations are compromised in some way, and the only way to fix them is a complete project rebuild. That just happened to me, and though it was pretty scary, and a lot of effort, I got through it, and it actually wasn't as bad as I thought it would be once I made the decision to do the full rebuild.

Keep in mind that I was the only person working on this repo, if you are working on a repo with others, definitely discuss with them before undertaking anything like this.

I think this is a situation that will become more common place now that many of us are using powerful AI/LLM tools, because as much as you can gain tremendous ground in build outs, it's very easy for instability to creep in without you noticing. This is something that would have happened anyway without these tools eventually, it's just that you reach that point a lot faster. Anyhow you will likely find most of this is just common sense, but when you are in the thick of it, seeing the woods for the trees isn't always obvious.

So what happened? Well the past month or so I have been building an Oauth 2.0 REST API backend with a React frontend. And I have been using AI/LLMs and devcontainers. Quite a few ups and downs, but I was making some seriously great progress, at least from the perspective of the application code. Both the frontend and backend were super solid. I was on solid ground, or so I thought. While experimenting with running Ollama locally on my laptop I uncovered some problems with my dev environment. The devcontainers I had been using started crashing on startup. I tested various configurations, and that uncovered some misalignment in my typescript and linting configurations, and before I knew it, I was in a giant tangle of errors.

I am big into writing minimal projects. Over the years, I have written many of these, I have a special section dedicated to them in my portfolio. When you get into difficulties it's often a great way to find stability again. There is a real art to it. In this specific case, things were so tangled that it wasn't obvious at the outset what exactly the minimal example should be. I thought initially that what I needed was a minimal example of an ollama based webdev environment, and spent a while doing that. Eventually I realised that ollama was complicating things, and that actually what I needed was a NodeJS Typescript Express React minimal example. A minimal project all built on containers. That was a bit of a voyage in itself, but once this new well structured project was working, the big challenge was to backport it into the original, much more complex, REST API + React project.

I thought initially that I would be able to move over pieces of it bit by bit into the old structure, reshaping as needed. It became obvious that was just never going to happen. I spent hours just pushing small pieces around, testing and seeing everything was still broken, and being totally overwhelmed by the enormous mountain of files, all out of shape, plagued by lots of false starts and bits sticking out all over the place. It was a monstrosity. I knew I now had a known good structure, but I didn't want to start a fresh project because I didn't want to lose the project history. There was a lot of valuable information in there. Here's what you do.

• First of all, backup the main branch, call it something like main-backup. Also backup the entire project into a tar ball.
• Ok you backed everything up. That's good, but also very very important, backup the project git directory (.git). It's a hidden dotdirectory. You will only be able to see those with ls -la.
• I decided to not use AI in Vscode for any of this. It has a tendency to try and take over and lead you down terrible paths. I did use AI but only via web browser. I was using Gemini.
• Create a new feature branch, call it something like refactor-new-project-structure.
• Make sure you are on your new feature branch, and then delete everything! Literally rm -rf * in your project directory. You read that correctly. This is the scary part. Remember you have backups. If you don't have backups, go back to the start of this list and make backups! Remember to delete all the dotfiles and dotdirectories in the project directory too. I just did that one by one, after deleting all the regular non-hidden files. One important exception: Don't delete the .git directory. That's where all your project git history is stored!
• Recreate / copy across your minimal working project to your new feature branch.
• Get that working, test it's all working. There is no reason it won't work since you already got it working in a separate project.
• Now pull bits in piece by piece from the old main branch into your feature branch. It's a bit tedious, but you can do this relatively easily. Take your time. Don't miss anything. List, view and checkout files and directories direct from the main branch using these commands:
  • git ls-tree --full-tree -r main
  • git ls-tree main:some/folder
  • git show main:path/to/some/file.js
  • git checkout main -- some/folder
  • git checkout main -- path/to/some/file.js
• Add and commit each piece with a descriptive title. Keep testing it's still working along the way. Ditch anything you no longer need from the old main branch.
• Eventually you will have recreated your project, and it will all work again!
• You will have a long series of rebuild commits. I had about 30 commits.
• Backup the rebuilt branch, call it something like backup-rebuilt-project.
• At this stage, things were still kind of complex, so I asked AI to help me group these into 6-7 groups of related commits that describe the rebuild process, so I could squash them into a more focussed history. I copied and pasted into the chat the relevant commits listed from running git log.
• The AI came up with a great rebase plan, with the commit titles and the commits all grouped. I just had to copy and paste into the rebase in neovim. It also prepared titles for each of the 6-7 squashed commits, which I copied and pasted into the rebase as it progressed. I rebase squashed the slightly haphazard 30 commits into 6-7 tidy and focussed commits.
• At this stage, make another backup of the rebased branch, just in case!
• Now change to main branch, merge in the squashed branch. In my case there were literally no conflicts, which was a relief but not too surprising.
• Backup the merged main branch into your final branch that you will merge in on Github, call it something like refactor-new-project-structure-final.
• Reset your local main branch to the state of the Github remote main branch.
• Push your final merged branch to Github and create a PR.
• Merge the PR into main. I got the AI to write me a comprehensive PR message that summarized the rebuild.
• Pull latest changes to your local main branch from remote, test it's all working on main branch locally.
• Now that everything is merged into main, and tested, cleanup by deleting all your backups and the feature branch.
• You are done! Do a retrospective on the nightmare you just completed. What did you learn?

You see it actually wasn't that bad, mostly common sense.

In my case the biggest structural change I had made was to move several top level folders which were all npm workspaces under a single packages folder. Git is quite clever and can mostly keep track of where files have moved to. Apparently having a focussed series of commits makes that easier. Aside from that most of the changes were to do with typescript and eslint configurations. The bulk of the code was completely unchanged. The only code that had changed was some of the npm scripts and Dockerfiles/Containerfiles. But there wasn't that much to merge in, just a few things that were in the original files that I wanted to keep, like installing neovim and networking tools for the devcontainer image. I just copy and pasted those across manually, and tested it was all working.

There were a few odds and ends to smooth out, like updating to a more recent development base image, so neovim had all the libraries it needed, and a bit of a refactor of how the backend production image assembles the final production node_modules folder to include the shared workspace. On the whole though, it was a lot smoother than I expected, even if it was kind if scary.

If you are in a similar situation, I know it sure isn't that much fun, but hopefully reading this will help somewhat. Sometimes it just helps to see something similar mapped out and to have the knowledge that it's possible. #

Containers, Typescript, React and AI/LLMs

I wrote a while back about how Typescript, React and Gemini was a pretty great combo. I still stand by that post, but I want to make add a slight refinement: Containers.

Yes, in a way it’s much more complex, at least initially, and you can definitely fall down some very dark debugging and configuration holes, but once you get the hang of it, it's kind of night and day compared to developing the old school way. With containers you have these neat little worlds all bundled and isolated that you can move around as a unit, much like an actual container. There is quite a lot to think about, but once you have your build process figured out, everything is very seamless. At any stage of development you can run a script and a few moments later the app that you were working on is neatly packages up, and you can test it, as if it were deployed, and then of course deploy it.

This becomes really interesting when you are working with AIs because you can create these environments where you can rapidly experiment and explore new things in a safe way, knowing that if anything goes wrong, you won't have trashed your base host machine.

The way I have things setup now, it's sort of like a digital version of a mechanics garage. I have several pretty solid dev tools like an IDE, and all sorts of CLI tools, and now with containers I can whenever necessary spin up a cluster of locally running AI/LLMs which I can connect to directly from within any of my projects in VScode or via a web browser chat interface. It's like there is this big mechanical robot attached to the ceiling, that I can pull down anytime I need to do something requiring something specialist to handle particularly tricky or complex situations.

It's not just local LLMs, I can connect up to cloud LLMs too, which are even more powerful.

React makes building dynamic frontends way easier, and the Typescript acts like scaffolding, giving you lots of structure to help you navigate difficult situations with the AIs. You are able to spot when things go off the rails way sooner than you would otherwise, and as long as you have some well thought out best practices, you can achieve some incredible things.

Things like guarantied end-to-end data integrity. By sharing types between the backend and frontend, creating your types by inferring them from zod templates, which are used to validate data on the backend, while simultaneously strongly typing all objects that you pass to the backend service layer, and doing the same with all responses to the frontend, you can get to a place where your app is incredibly robust. And I‘ve found that the AIs perform much better too, because they have a lot more information to go on as to your intent.

Lots of other things are possible too, here are a few that I wrote about previously in my post about recent project milestones.

And with containers you now control the entire environment within which your apps run, both in development and in production. When I built linkblog.io, I made a conscious choice to build without containers because at the time I felt they weren't ready for prime time. Back then I wrote about the robust NodeJS architecure you could build using open source software and cloud services, I knew at some stage I would eventually be moving to using containers, and there is little doubt in my mind that now is that time. #

Man this environment rebuild has been very insane. The good news is I got the backport working in my main app yesterday evening, and got the production container build working this morning. Some cleanup and loose ends to figure out, but it‘s working. More details to follow over the next few days hopefully. #

Happy birthday to me.

Happy birthday to me.

Happy birthday to me.

Happy birthday to me.

I have always enjoyed birthdays and eating cake. #

Rebuilding your container based dev environment

If you have been reading the blog you might have been aware that I was going through some turbulance with the dev environment. Some significant progress in the past day or so, but still not quite out of the woods. Just to give you an idea of how bad it‘s been, here's the commit message where I realised that the dev environment was hosed:

Author: Mark Smith [email protected] Date: Wed Sep 10 18:42:31 2025 +0700

chore(backlog): Emergency re-prioritisation of current sprints tasks

The end of the 2025-08-29 sprint saw us reach a point where we had a great foundation for our new dev environment that supported local LLMs running in Ollama. One big downside was performance. Most queries on the larger models would typically take 30-45 seconds. Add that to the generally significantly less good response quality, the solution was not very practical.

It was discovered that GPU acceleration for these local models might be possible so we did a spike on trying to get that operational. It was long and drawn out, but success was reached with a minimal project example demonstrating a container running in Podman that could run vulkaninfo and get back output computed by the host GPU.

Getting this working took significant reconfigurations and trial and error. As a result of which it seems something was broken in the main podman / devcontainer causing devcontainer builds to fail. This happened even outside of the GPU spike feature branch, with the main branch code, which had previously worked fine.

We now need to get the devcontainers setup back onto solid ground, and so all tasks in the current sprint have been pushed into stretch goals while focus is concentrated on remedying the situation.

The plan is the plan until the plan changes as the old saying goes.

Hold on to your butts.

Refs: task-037

That was just over a week ago, and since then I have been deep in the depths of the nightmare. And as is always the case, the world immediately around me and the wider world has gone absolutely bat shit crazy insane in the membrane. Horrendous. I‘ve had to rebuild the whole environment from scratch, starting with the simplest possible configuration, getting all aspects of a usable development environment working with that and then slowly building that up into something that has the same structure as my API + React app.

I have a few minor things to add to the fresh environment, and then I need to backport it into the original project. So not quite on stable ground, but I can see the light at the end of the mixed metaphors abstraction apocalypse tunnel.

On re-reading this commit message, I noticed that I started referring to 'we' and 'us', even though it‘s just me solo dev'ing this thing. Working with LLMs is very odd.

In any case I think the thing to learn in all this is that when working with AIs and especially if you are running everything in containers, it‘s a good idea to have a very minimal working project that you can spin up if something goes weird in the main project. There are just so many layers where things can get crossed the wrong way. #

Another AI configuration nightmare

Recently everyone in the world has been constantly banging on about how AI is the best thing ever and how if you don’t get going with AI then you will be living in hell for the rest of eternity. And so you get going and have been making significant progress using some of the cloud AIs. They are surprisingly good, even if half the time they are trying to kill you. You suspect they are actually trying to kill you 75% of the time. You have spent a lot of time setting things up securely so they can’t do too much damage if something goes wrong. It’s been a slog but it’s not too bad.

Getting a bit worried at how reliant you are becoming on US tech giants, a stable internet connection, and at having to upload every bit of code you write, including your potential clients code, to somebody else’s servers, you decide it would be a good idea to get some open source LLMs running locally on your laptop. After a lot of very confusing installs and testing and generally hitting every possible brick wall, you get it working. But it’s shit. And it’s so fucking slow, you might as well be using an abacus to help you with your code.

As we all know by now AIs run much faster on GPUs than on CPUs. GPUs are like 10 times faster at matrices multiplication than CPUs and AIs are all about matrices multiplication. Since things are slow you check if your local LLMs are using the GPUs in your laptop and it turns out that no, they are not. For some strange reason on this brand new laptop, one of the most popular laptops in the world at the minute, with loads of GPUs, getting the GPUs working with the containers that run your LLMs, literally the only safe way to run these things, requires some adventurous reconfiguration. But you have to do it, otherwise hell for eternity.

So you follow the docs, and after quite a bit of being confused, you figure it out and get it working. Along the way you have had to install and uninstall all sorts of things. But it works. You ran a command called vulkaninfo that uses the vulcan GPU library in the container which gets your GPU to do something and then prints that it was successful. Except it only works if your LLM is running in Fedora:40. Oh and you have to use Podman Desktop, which is different to podman. It’s a GUI that you can use to do podman stuff. Normally podman can do everything the GUI can do, but for whatever reason, when you are setting up podman to run containers with the GPU acceleration, you have to create the podman VM, which is a virtual machine that co-ordinates all the containers, you have to create it using the GUI. And when you try to create the podman VM, there is a warning explaining that you need to install krunkit, a set of libraries, and you can install them with brew. These days on a Mac everybody uses brew to install basically everything.

Remember Fedora:40? Well the problem with Fedora:40 is that it’s a bit minimalist as far as container images go. Whereas many of the popular images that are used for building apps have automatic UID/GID mapping between the user you use on your host machine and the user in your container machine, so that you can safely modify your project files, that reside on your host machine from within the container, Fedora:40 doesn’t do that automatically. That’s not such an issue if you are only running your models on it. Ollama doesn’t need to modify any files on your host machine. If however you have to run the Nodejs app you are developing on it as well, then it’s a pretty big issue, because of course you are going to need to modify the project files, which are on your host machine.

But then you discover that VSCode, the editor that you use to write your code, and which can open your code from inside a container, so you can safely work on it with the AIs that are trying to kill you, can work not only with single image containers, but can also work with multi-image containers. So technically you can give it a config for a multi-container setup, and it should be able to spin up the whole thing, with your app in one container, and your LLMs in a different container, and a network they can talk to each other on, and storage volumes etc. This is great because the LLMs can be on Fedora:40, which we know can get access to the fast GPUs on your laptop, and the app you are developing can be on an OS that does the UID/GID mapping so you can safely modify the project files on your host from within a container. And that means you can safely work on the app with the help of the maniac AIs that are trying to kill you. Brilliant.

All you need is podman-compose, which is the tool that creates the config for these multi-container setups. You power through yet another configuration nightmare trying to figure out how the UID/GID mapping works in these container systems. But gosh darn it you get it working! However along the way you discover that if you have any issues with this by now, I think you will agree, quite complicated setup, well the core contributors on the podman project, if they get a sense that your podman has been installed using brew, then forget it, they aren’t going to help you. Brew is literally “not recommended” on their website where they explain exactly how to use brew to install podman while “not recommending” it. The recommended way is to use the podman installer package they build and serve for download by you from their website. Since your entire existence outside of hell now relies on the podman, you figure it’s probably safer to install podman the way the podman core developers are saying you should. So you clean up your system, uninstalling everything you installed using brew, and install the podman.io podman. This is going to be such a rock solid system you think to yourself. And that works and you try to open your new mutli-container development environment, and it fails because you no longer have podman-compose installed.

Of course! You had to uninstall podman-compose earlier because when you tried to uninstall the brew version of podman, brew wouldn’t uninstall podman because it said podman-compose relied on it. So you uninstalled podman-compose, then uninstalled the brew podman, and installed the rock solid, recommended by the core contributors, podman.io podman. You type brew install podman-compose in your shell, and just before you hit enter, you think to yourself, it’s been such a pain in the fucking arse installing and uninstalling and re-installing podman, it would really suck if something went wrong here. So just in case, you double check that your current podman is the podman.io podman, and the path to it. And you hit enter, and brew does it’s thing, downloads all sorts of packages and voila, it is installed. You check your podman again, and wouldn’t you know it, your podman is now the brew podman once more. Shit balls. You decide to clean up the system again, so you can get your podman back to the podman.io podman, and try to run the podman.io uninstaller, but guess what, the podman.io podman has no fucking uninstaller. Say what you will about brew, but at least they have an uninstaller, and they do try to help. All you can do is run the installer again and hope it overwrites the bad brew podman. It does. At least it seems, but who knows at this stage. You open an issue on the podman-compose github, you are a good netizen after all.

Someway through this nightmare of nightmares, you read the news that some famous right wing public speaker bloke called Charlie Kirk has been shot JFK style while he was talking at a university in the US. And also that back home in London there are a series of demonstrations, and demonstrations against the demonstrations and demonstrations against those demonstrations, and everyone is waving Union Jack flags, it’s a mess, but some people seem to be enjoying themselves and it’s quite nice seeing everyone waving flags, there was a time traveller bloke right out of Monty Python and the Holy Grail, so many Union Jack flags, a New Zealand flag, a cheeky little US flag, a Scottish flag, and an enormous Welsh flag. I think someone even snuck in a Catalan flag. I know this is going to sound really ridiculous, but it’s like an England Ireland match and I genuinely want both sides to win, which makes no sense, but that’s how I feel. I’m also wondering if Leonard Nimoy might be able to put in a good word, maybe I can get this nightmare figured out. Just another infinitely improbable coincidence to add to the very very very long list of impossibly improbable coincidences that happen just constantly at this point. Shrug.

Ok anyway, nevertheless you continue, you have to somehow get podman-compose installed, or we are back to installing Ollama in the same container as your app, and that means moving everything to Fedora:40, and then likely another host-container UID/GID mapping nightmare. You go to the podman-compose github page, and read the installation instructions. It says nothing about requiring the brew podman. It just says podman. It doesn’t say anything about not using the podman.io podman. It also lists some python libraries as required. Your system has python3 installed. You check the code and podman-compose is a python3 application. Of course you can’t install it with brew, because that overwrites the good podman with the bad podman, or at least the one that the podman contributors claim doesn’t have all the libraries for GPU acceleration, and to be fair, the warning in the Podman Desktop GUI about the missing krunkit, does suggest they are not wrong. The other installation methods are: pip or manual install. Every single time I have tried to install python and used pip, I have eventually ended up regretting it. And it has happened a lot. So that leaves manual install. Not great, but the instructions look relatively straight forward. There’s a script that builds a container which is then used to build podman-compose including all the python libraries and the other bits and pieces it needs. But the script fails. I have a look inside the script and find the commands it runs and run the commands myself, and it works! The python image gets downloaded and the build starts and a few minutes later the image is created and in podman vm. I eventually manage to get the build binary out of the container and onto my mac. I run it and it tells me “cannot execute binary file”. I go through the process again, same issue. I guess it’s because it‘s because it has compiled it for the container OS rather than your host machines’s mac os.

Blocked, blocked, blocked. Life is just a long series of being blocked.

The only remaining way is to install pip, and go the python route, but it’s 4:33am and I’ve got to sleep a bit.

Update: I wrote this blog post and it’s now 6:29am. Hello daylight. #

I only had about 5 minutes to write this post, so I cheated and gave Gemini my git commit history, asked it to write the post. It was really dull. So I asked it to write it in the style of Hunter S. Thompson. This is what it came up with:

Right, let's get down to the bone and gristle of it. The week began with a righteous and noble goal: to cage the beast, to get this Ollama AI monstrosity running in a container where it couldn't do too much harm. A clean setup, a moment of pure, crystalline satisfaction. But the calm was a lie. The silence from the machine was deafening. What was it thinking in there? What dark, twisted packets was it sending into the ether? The paranoia began to creep in, a cold sweat on the back of the neck that only a true fiend for the truth can understand. There was no choice but to turn the tools of the hunter on the hunted. tshark became my weapon, a digital scalpel to peel back the layers of the network and stare into the abyss.

This descent into the wire wasn't the only madness on the docket. While one eye was fixed on the network traffic, the other was dealing with the nagging demons of the frontend—bloated Vite builds screaming for a fix. A quick bit of code-splitting surgery to shut them up. Then, a pivot to the backend, a place of shadows and secrets, to install some proper logging and a panic button to dump the whole database if things went sideways. And in a final, desperate grab for sanity, I wrestled with the macOS security apparatus to make Touch ID bend to the will of sudo. A small victory, yes, but in this line of work, you take what you can get before the next wave of insanity crests the hill.

So that‘s been the past week. #

Openvibe all the things

I installed Openvibe.social earlier. It looks kind of cool.

Here’s the blurb:

One App For The Open Social Web All decentralised social networks—single timeline

I’m not really using social media at the minute. I think it’s cause there are just too many places to go to. I don’t have enough time to be checking in all these places. I can only just about do it with one place.

Anyway it was easy to install, loads of posts from the people I follow on Bluesky and Mastodon showed up in the timeline. I posted a couple of messages and like some sort of strange echo they appeared in triplicate in my timeline, one for Bluesky, one for Mastodon, and since I also connected up my Nostr, one for that too.

I'm kind of confused about Nostr, during sign-up you have to enter your nsec, which is your private key. It makes sense in some ways, because they need to be able to create posts as you so they need to have your private key. No way around that as far as I can see. What's confusing is that literally next to the box you enter your private key, it says something like "never share your private key". WTF.

Doesn’t Nostr have a built in way to grant authorization, that you can revoke? If you have to give your private key out to 3rd party services, what happens if one of them turns out to be bad? You have to create a new Nostr account. Seriously?

I'm not really using Nostr much so it doesn’t make that much difference to me. But I do want these services to work well. I might use them more if they were easy and secure to use. #

Haseeb Qureshi on Kanye West releasing $YZY [59:05]:

It’s like the same cabal, it’s the same people [...], just like every villain, it's kind of like an Avengers movie, every villain from a previous cycle is also back, is part of this thing. It is a nice way to wrap up the saga, if this is the last celeb coin we are going to have to deal with".

LLMs running locally

I finally got working the thing I have been configuring the past few days: a collection of LLMs of various different sizes, running on my local machine, all accessible via terminal and through a chat interface in VSCode. I fell down all sorts of strange rabbit holes, but it‘s working. I now have a much better appreciation of how all these AI models work, but still lots to learn.

This was the first time I've worked with Claude. I found Claude to be very efficient, most of the time, refreshingly concise, and dare I say it, surprisingly honest about the current state of AI tooling. There are still some strange things though.

When I asked each of these newly configured models separately what their name was, they all initially insisted that they were an AI language model created by OpenAI called GPT-4. Of course since I downloaded, installed and configured them, I know that none of them are in fact GPT-4 created by OpenAI. Interestingly when I disconnected from the internet, they all started insisting that they were Claude made by Anthropic. None of the models are actually Claude made by Anthropic.

The VSCode extension that I was using to connect to the models, has a website where they have a chat interface, and I was chatting quite a bit with that AI, who also claimed to be Claude.

So yes, it’s working, but I‘m not exactly super confident it‘s working particularly well. One of the models was initially very sure that 2 + 2 = 3. Based on the conversations I had with Claude, it’s quite apparent that most developers that are using AI aren‘t exactly overly worried about security. It seemed like I was in the minority to try and configure things to run in devcontainers.

The big thing here is that I wanted to be setup so that if needed I could work with AIs on coding projects without needing to upload any code to 3rd party servers. That‘s the downside of cloud based AIs, they are very quick and sometimes quite clever, but they require that you upload all your code to their servers, and I know that that isn't always an option for some folks.

I’m hoping to work on some coding features over the next few days. I‘m very interested to see how they perform compared to Gemini. #

The past few days I seem to be stuck in a bit of a configuration nightmare. Hopefully I'll have the thing I'm configuring working soon. It's partially working now, but practically speaking not functional.

I also added an anti-corruption layer to the frontend code, which is a fancy way of describing a standard way to unwrap the API payload, which theoretically makes the application more portable. #

The world is really getting very synchronicity overload the past few days for me. These sorts of times rarely end particularly well. Thought that was worth mentioning, not that it will make any difference at all, world is going to world. #

I was very happy when I started out writting this post, having a morning ice coffee, after a nice breakfast. The ordeal I had to go through to publish this little post has left me not quite as happy.

Such is life sometimes. #

Miljan Braticevic [38:46]: “When you create an account on X let's say, you are given this account by this corporation, this is something that they give to you, and it's something that they can take away from you at any point. In contrast, when you are on an open protocol like Nostr, you claim your key/pair. You take this massive number out of the universe, and that's you're private key. So you have claimed some territory in this massively large namespace of numbers and then you've claimed something and this is yours and then you defend it. What a stark contrast between these two, and then everything is downstream from there.”

I thought this was a great explanation of one of the central paradigms of key/pair based systems. Somewhat surprisingly, because I really am all for independence and self reliance and the rest of it, I found that the new knowledge that it unlocked, was accompanied by a roughly equal amount of unease. Like somehow there might be a side to this that is not being covered. Might be a bad idea to push to get rid of the centralised systems so completely, at least for a little while. I guess that has been my opinion for a while now. It seems to me like it might be one of those hills that isn't worth dying on, like an asymptote stretching off into infiniti that you can never quite figure out.

I'm making a mental note that it might be worth revisiting the tractability vertigo of this whole thing some time again in the future. Analysis paralysis is a real so and so. #

Some recent project milestones

I have a short list of blog posts I want to write covering some of the key things I have learnt getting my API & React frontend project to a pretty great place as far as code quality. It's been quite an interesting few weeks development-wise. I'm way more impressed with both Typescript and React than I expected to be, and though it's been kind of a roller-coaster ride working with Gemini, I'm still impressed with that too. I'll hopefully get to those posts over the next few weeks, but in the meantime, I thought I would do a quick review of the major project milestones from the past few weeks.

Without further ado, and roughly in the order they occurred, here they are:

1. Zod based DTOs - Otherwise known as Data Transfer Objects, these are Typescript type definitions for all data that gets passed back and forth between the frontend and the backend. The cool thing about using Typescript in both the frontend and the backend, is that you can setup a shared workspace and put them in there. The backend and frontend still have their own type definitions, but in addition they can use the shared ones. The key thing is to define the data shape using Zod schemas, then create the types by inferal off of them. That way there is a single source of truth for the shape of all data across the frontend and backend, and as a bonus you can use the schemas to do data validation on the backend in a single reusable middleware on all routes. It standardizes all "In-flight" data.
2. Shared public entities - Another types related thing that works really well with DTOs. These are types for all "At-rest" data that contains some fields that you don't want exposed externally like passwords and private keys. You create the public version of the entity in the shared workspace and then create the internal version based on the public entities, with the sensitive fields in addition. It just ensures that the sensitive data never leaves the secure backend location.
3. Linting & typechecking - This was huge, especially when working with Gemini, because it has such a tendency to drift, and is constantly leading you down confusing and wrong paths. Having a way to automate the checks means you can mostly eliminate the possibility that style stuff you don't like and typescript errors ever make it into the codebase. I setup pre-commit hooks for linting and pre-push hooks for linting + typechecking. I also set up some custom bash script based pre-push hooks to enforce a specific branch name format and commit message format. It really makes reading the commit history so much easier. Also as far as linting and typechecking config, it's very important to get the config in place, test it is working, and then NEVER change it. Gemini will constantly attempt to change the config instead of fixing the errors, and get you into never ending config change loops. You should almost never change the config.
4. Secure devcontainers setup - I spent quite a bit of time setting up two different devcontainers, one has read access to the repo but no write access, the other can read and write. The idea being that I should be able to run Gemini in agent mode in the restricted setup and then verify the code myself before switching into the other devcontainer to checkin the code. Similar to the linting and typechecking config it's a good idea to not change this from inside the container. You can set the config to be on a read only mount, so you can always be sure nothing important gets changed without being aware of it. The trick is to clone the repo using https for the read only setup, and use a script launcher script that passes a fine grained access control personal access token (PAT), but for read + write use ssh keys mounted read only to the devcontainer. You will also need to run some post connect commands to ensure you switch correctly between the setups.
5. Security providers and contexts - Use these React idioms to setup authenticated parts of the site, making the authenticated user available wherever it's needed. I really like this feature of React, once you figure out how it works, it's a really elegant way to make sure certain hooks are only ever used inside parts of the DOM that are wrapped in a specific component. So you always know for sure that, for example, the authenticatedUser is available inside the part of the site that is wrapped in a security context. Spend time understanding how the pattern works and name things well, because Gemini will often suggest ambiguous names for things.
6. Backlog.md task based workflow - When working with Gemini this is essential IMHO. You need some way of keeping track of work, because you will constantly be getting cutoff and each time you will need to get Gemini up to speed. Pretty much mirrors how humans should do it, specification driven development, but it's much easier because the AI can write much of the spec plans and acceptance criteria. Of course you will need to review it and make sure it's not full of weirdness, which happens quite a lot. The other thing that happens a lot is Gemini suddenly starts inserting backtics into documents, as a sort of sabotage because it breaks the chat window, and then there is no way to accept the changes. It's usually a sign that Gemini is confused about something. You can often get around it by asking it to oneshot a new version of the doc with _NEW appended to the filename, and manually do a diff or copy paste yourself.
7. React router, queries and loaders - Another really great React feature, that enables you to pre-fetch all the data you need on page load, and when combined with Tanstack query, you get caching thrown in for free. It might actually be a feature of Tanstack, I can't quite remember. In any case it's cool and makes apps feel really fast. Might have been easier to start with these from the outset, depending on your React / javascript fluency. I got it all working without it, only for Gemini to suddenly tell me about it after initially forgetting to tell me about it. The conversion took quite a while.
8. Me endpoints for added security - I realised that the initial architecture was leaking user information via the urls used to access the API. It meant that it was trivial to identify admin users just by the urls they were loading. This was a pretty great solution, suggested by Gemini, apparently it's very standard these days, but I hadn't been exposed to this pattern. Basically you have dedicated /me routes (i.e. /me, /me/blah etc) that specifically only loads the data of the logged in user, and that's hardcoded into the route, so privilege escalation is impossible, but also since all users can use these routes regardless of their role, it makes it much much harder to identify say admin users based just on the urls they are requesting during normal usage.
9. Weekly sprint planning process - The backlog task setup has been crucial, but it became a bit unwieldy just because it was so good. I setup a simple weekly process based around the Agile development methodology idea of weekly or bi-weekly sprints. I do it weekly, and have a standard document that gets generated during a planning session with Gemini where we identify which tasks I will attempt to complete in the upcoming week. This works really well with the already in place process of retrospectives, for documenting major decisions. Gemini is always reminding me of very important things we have discussed previously while working on features. Really helps with general AI guidance.
10. End to end data integrity pattern - Probably the most important, and is really a sort of supply chain type concept, where a series of things you put in place in such a way as to guaranty that the data sent in the frontend application layer is pretty much always exactly the same data received by the backend service layer. There are all sorts of little crevices and weird unexpected places things can get screwed up as you pass data through all the frontend and then backend layers, but I've found a way that is very very effective. Relies heavily on shared types.
11. API response refactor - This was a follow up bit of refactoring to the previous, end to end pattern. You have got to remember to standardise the responses. Again using some clever shared types, you can ensure all the responses have a standard response envelope, for single object responses, lists of objects responses, and failures and error responses. Gemini was super useful to do a quick survey of the top 10 API platforms response envelopes. You can then craft one that works best for your purposes. With enough thought you can design something that is flexible enough to accommodate more complex features like paging, even if you don't implement that initially.
12. User details pages - For admin views, you need to be able to search for a user and display their resources, just in case something goes wrong. Only give admin fine grained modifying capability once an activity log is in place, so you can see what actually happened. The search feature was yet another place where React really shines. We had a ResourceList component used across many pages to list the various resources. Well one of the props it accepts is a way you can pass in an HTML snippet and it inserts it into the top right of the table. So you just pass in an input box and you can wire it up really easily to which rows get displayed in the table as you type in the input box.
13. Make Oauth endpoints RFC 6749 compliant - When I overhauled all the response types, the Oauth 2.0 flow stopped working, because Oauth has some very specific requirements as to the shape of the data, and the new response envelope was causing issues. Just remember you might have some endpoints that have different envelope requirements, depending on what you are doing.

So that about covers the main things from the past 2-3 weeks. It's been very productive, although very tiring, and at times quite confusing. Working with an AI agent is awesome until it innevitably isn't. But it's possible to put in place some best practices, tools, and development methodologies to get the codebase back into great shape. #

About a week ago I thought I'd gotten the codebase of my new project to solid ground. I knew I still had quite a few features to add to get the migration to a React frontend finished, but I soon discovered that there was considerable drift from the established project best practices.

I ended up having to go back over all the routes and standardise the API response envelope, and put in place some type checking to ensure that all the responses were consistent. It was rather a big effort with losd of knock on things that needed fixing and updating, but I got it done a couple of days ago. I was completely exhausted, both mind and body. I'm starting to feel a bit better now that I have had a bit of propper rest.

In any case the code is looking great, with a new react router, queries and loaders, end-to-end data integrity, and now a standardised response envelope. Hopefully I'll get some time to write a blog post about it soon. #