markjgsmith

Some recent renovation successes

Over the past few months I’ve had quite a few successes in various renovation projects. It can be very chaotic going through these periods, and it’s not always obvious seeing the woods for the trees, but looking back I can see some definite themes. I wanted to spend a moment reviewing things, it’s important to celebrate the wins.

1. New blog redesign

I’ve been blogging for something like 20 years. The first few blogs are no more, long since dissolved into the sands of time, but the current incarnation has been around for a little over 10 years. Actually, according to the date calculator, it’s been 14 years, 3 months, 3 days.

There was a long period where it was just a hand rolled linkblog, but eventually I added a blog, newsletter, podcast and notes sections, all interwoven on the front-page. This year I finally got to redesign the whole thing, still keeping it’s minimalist aesthetic but adding a bit of color.

2. New container based dev environment, Typescript+React, safe collab with AIs

I have in the past built quite serious projects, including a social media SaaS, but I had stayed away from containers, Typescript and React. This year that all changed, brought on somewhat by the new realities of using AI/LLMs in web development. I now have first hand experience of how these technologies work extremely well together having built an Oauth 2.0 REST API with React frontend. It certainly hasn’t been plain sailing, but I have settled on a really solid new container based typescript-react project structure, that I am very happy with.

3. New locally running LLMs, GPU accelerated running in containers

I’ve spent a lot of time experimenting with AI developer tools, and I’ve been blogging about all my experiences with AI. There have been many ups and downs, but it’s obvious that these tools are going to have a big impact, and realising that I was using the frontier cloud models more and more, I decided it was important to at least get some experience running local AI open source models. Even if they aren’t as powerful, hopefully that will improve over time.

After a voyage of discovery I got some LLMs running locally, but then ran into performance difficulties. In ended up going down a very very deep rabbit hole looking into how to get GPU acceleration working in containers. That was at the same time I was trying to build the new Typescript/React/Containers project structure, which was all very confusing, but I got through it and finally got GPU accelerated local LLMs running in containers accessing my mac’s GPUs with massive speed improvements.

4. New social media auto-poster

After all the container based AI GPU Typescript React craziness, I decided to switch focus back to my blogging, noticing that my posts were not syndicating particularly well across the various social medias I use. After researching and testing several commercial auto-poster products, I realised that what I really wanted was a solution that I could plug into my blog's static site generator, something I could run in a Github Action. So I built a social media auto-poster! I'm very excited about this. It’s looking really promising, hopefully rolling out at the end of the month.

5. New blogging scripts

With all this momentum in my blogging, it felt like the right time to overhaul my blogging setup, refactoring all the scripts, aliases and functions I use to publish the blog. This was an incredible success, working with Gemini I was able to drastically simplify and make my setup much more robust and added several features including managing images, which is a huge win. I am already seeing big benefits in my day to day blogging. Massive time saver.

6. New dotfiles for multi-platform bash config

The final renovation project was a complete overhaul of my dotfiles, something that I had been wanting to do for literally years, but had been in a state of complexity paralysis. With Gemini’s help, I was able to cut through years of cruft and somewhat tangled scripts, and by deciding to drastically simplify my approach, I now have a repeatable multi-platform setup for managing my shell configuration. This is a huge win for stability and robustness. It’s deeply reassuring to know that I can rebuild any of the core components of my setup with a minimum of fuss. That I can be back up and running very quickly should I need to do a complete system rebuild.

There are likely a few others, but these are the main ones.

Looking back at it all, it’s turned out to be an exercise in improving robustness at every level of my development stack, while exploring the emerging world of AI assisted development. It’s been quite a ride. I’m super glad that I’ve been blogging through it all, because it makes these sorts of retrospective posts possible. It can feel like you aren’t making progress at times, the world just keeps throwing endless hurdles in your way, but actually slowly but surely things are getting a little bit better every day. #

Vibe civilization engineering

I did a bit of civilization engineering with Gem while I was listening to some podcasts yesterday. Kind of interesting, and wild, and scary. #

I'm seeing and hearing a lot of people in early adopter circles getting very interested in AI generated music the past few weeks. It seems to be moving from this is just a silly joke to hold on there might be something interesting here territory. #

Overhaul of my dotfiles

I did an overhaul of my dotfiles!

For those of you not familiar with the concept of dotfiles, how to explain them? Well on linux/unix systems many applications store their user settings inside 'dot' files and / or folders. These are files who’s filename starts with a dot, for example .bashrc, .bash_profile, or .vimrc. The reason for this is that by default when you list a folder's contents, the items that start with a dot are not listed. This keeps things uncluttered, because most of the time you don’t need to be seeing these files.

Since programmers typically use a lot of different software tools, in a variety of ways and on a variety of different systems, it’s worth spending the time to get good at organising these files. Hence the idea of dotfiles, which are collections of dot files that are organised in such a way that you can easily make changes to them and deploy them onto different systems. Typically via git, with them all stored in a git repository like on Github. People get very carried away with them. Check out awesome dotfiles for some idea.

The thing is, as awesome as they are, it’s all too easy to fall down a very big rabbit hole. That’s what had happened to me with my first attempt. I figured it would be great to use my dotfiles to install and configure the entire system, and it would have been, but the reality is that system installation is something most of us do so infrequently, that these complicated setups rapidly get out of date, and they are so tedious and annoying to test, that in practice they just gather dust and you get more and more scarred to touch them because you can't remember all the reasons you made them the way you made them, and they just become a thing that you open up every now and then, start panicking and quickly close and move onto something else.

The thing is though, they are quite important, because when you do have to setup a new system, you either use them or you have to do the whole thing from scratch, which is also a total pain. A few days ago, hot off the back of several small overhaul successes, I decided to give it another go. Surely there was a middle ground between automated install of the entire universe and do everything manually?

As it happens I did recently have to install everything again from scratch, first my Android phone, and then my new mac laptop. When I did that I basically started with a fresh empty config and just added the minimum I needed as I went along. It has been working quite well, but a bit disorganised and also lots of annoying copy and pasting, via email, between both devices. They share a lot of configuration but there are also notable platform specific differences. The biggest insight I had before embarking on this overhaul was that all the fancy installation scripts were totally unnecessary. In the event of re-installation, I'm okay doing that by hand. But the one thing I really do rely on day to day is my bash shell configuration. All the aliases, exported environment variables, shell functions and bash scripts. I use many of these every single day.

With that in mind I asked Gemini what suggestions it had for structuring my dotfiles to do just that, and only that, nothing too fancy. I'm very glad I asked because the response I got back was formidable. In seconds Gemini cut through years of cruft and tangled code, suggesting a very clear and simple structure, and within a few hours working together we were able to come up with a setup that incorporated all my current configuration, while adding lots of very useful things that I wouldn’t have thought of. Now I don’t claim these are the best dotfiles ever, I was mainly trying to replicate the setup that I had developed over time, there are likely refinements that I will make over the next few months, but I’m pretty happy with it so far.

I’m still somewhat surprised it was that easy, and both laptop and android phone have been updated to the new setup and are so far running very smoothly.

It’s really great to know that should I need to re-install either of them, I will at least relatively quickly have a configured bash shell and be able to code and blog without to much interruption.

The whole repo is released under MIT license. Feel free to read it, experiment with it and get inspired to build something similar for your own situation. #

Who are the suppliers?

A note to remind myself that in future tech build outs, do some research at the start who the big suppliers are. Seems completely obvious now but I totally didn't think to figure out who would be supplying the materials and core components for the current data centre boom we are seing as a result of the AI industry build out. Several Japanese engineering companies have been ripping, between 30% to 500% added to their value.

The companies mentioned are:

• Advantest Corp [OTCMKTS: ATEYY]
• Tokyo Electron [TYO: 8035]
• Softbank Corp [TYO: 9984]
• Fujikura [TYO: 5803]
• Mitsui Kinzoku Co Ltd [TYO: 5706]
• JX Advanced Metals Corp [TYO: 5016]

New blogging scripts

Feeling quite under the weather today. Rough day yesterday, and today I’ve got a sneezy cold that’s been steadily gaining momentum the entire day. One presses on nevertheless. Some progress in my blogging setup.

I decided to do an overhaul of all the bash scripts I use for day to day blogging. I use a static site generator to build the blog, but separate to that I previously had a very basic bash script combined with some aliases that enabled me to very quickly create new empty markdown files for each type of post, with all the right frontmatter, and name the files correctly. It sort of worked okay but there were some weirdnesses. Actually quite a few weirdnesses, and the aliases had become ridiculously complicated.

I figured maybe this was a good task to work on with Gemini. I've been looking at these scripts for years, so couldn't really see any obvious way to improve them. I knew there would be, but I needed a fresh pair of eyes. So I cut and pasted the mess of scripts and aliases into the chat, and briefly explained how it all worked. To my surprise Gemini fully understood how everything connected together, totally understood what I was trying to achieve and made some very impressive suggestions.

Before long we had a 4 phase plan to refactor everything, and within a few hours we had everything working. There is a new master script that has absorbed much of the logic that was scattered across the aliases and the updated aliases are neat and tidy, and look great.

What’s more while we were working on the plan I mentioned that one thing that would be great was to streamline the workflow I use for adding images to posts, which was really convoluted. Gemini figured out a really great way to cut out almost all of the hassle, so now adding an image is super simple.

The other thing that is loads better is adding links. The script automatically adds and commits the changes to the git repo, so lots of the tedium and choreiness has evaporated. #

Spent several hours today on the social media auto-poster. It’s starting to look really awesome. The project structure is just right and everything is easily repeatable. I showed Gemini the existing setup with a bit of explanation, and basically just said, do the same sort of thing for some other social networks, and it obliged, and synthesized new code based loosely on the existing structure, but with relevant variations to accommodate the different APIs. Gemini is shockingly great when the structure exits and the task is very clear. #

Beauty in journalism

It occurs the me that the main reasons I read so much from the Guardian is that:

1. It's basically free, though of course you should donate if you can.
2. The writting is a good read most of the time.
3. It just looks so much nicer that any of the other MSM news sites.

However, I find that a lot of the opinions I either don't agree with, or cause my eyes to roll. It's astonishing to me how much of an effect beauty has. I very much like reading the Guardian, but also a lot of the time I very much do not.

I wish there was another MSM newspaper with a different political leaning that had a website that was just as pretty so I could AB test my stupid brain.

Oh well what to do? I guess I'll read another Guardian article. #

Is it simple yet?

It occurred to me today that in yesterday’s piece about how to build a social media auto-poster using Github Actions, although I did mention that Gemini was doing it’s best to go down every possible dead end, I wasn’t very specific about what those dead ends were. Gemini absolutely loves making things more complex than things need to be. We were at one point exploring using sqlite databases and all sorts. But you don’t need any of that.

You have to keep pressing for the simplest possible solution, but you do have to explore some of the more complex things first, and then circle back and say hey Gem, this all seems way way too complex, how can we drastically simplify it. And then it does actually suggest some good solutions.

All that to say, if you just want to post a few things to social media you really don’t need to save anything complicated. You can do it with just the features available in Github Actions. #

Building a social media auto-poster

Lots of progress on the Github Actions RSS to social media auto-poster I’ve been working on. I started out building the workflow primitives, just simple reusable workflows that enable you to post a message to social medias. The first two I’ve implemented are Mastodon and Twitter (X). Got those working and then went up a level and implemented a reusable workflow that posts to all socials, which triggers the first two.

At this point things were quite complicated and debugging became tricky. When you are running a workflow that calls another workflow things can get quite confusing. And initially I wasn’t sure I could have these all in the same repo, so I had them in separate repos, which added to the complexity. So I spent quite a bit of time implementing a repeatable way to easily and safely log workflow inputs and secrets (i.e. masked). It was quite cool getting to this point.

With that working, I embarked on the most complex of all, a workflow that reads an RSS feed, pulls out the newest posts, and for each one launches an instance of the post-to-all-socials workflow. The two tricky things with this workflow are how to launch many parallel jobs, and how to store state between runs. The level of complexity at this stage was considerable.

Gemini was of course being very helpful then being a complete liability, leading us both down several quite dangerous dead ends, forgetting why it had decided to go a particular route and then confidently announcing that the best final solution that would fix everything was to migrate to a new architecture which was the exact architecture that it had initially said was not up to the task. This happened at least 3 or 4 times. Thankfully I had been making good use of git feature branches, so never ended up in a situation that wasn’t recoverable, though it was certainly a slog.

I eventually figured out the right architecture and got it working, which is awesome, except that the workflows were spread across several repos, so I consolidated them all into one repo. That’s where I’m at now, and I’ve had to put everything on hold because I’m about to run out of build minutes. Testing this whole thing has been quite difficult, especially because there were loads of minutes wasted with many workflows that simply would not cancel. Something that should have taken a few seconds would eat up 5 minutes. This happened a lot.

So anyway, I’m waiting for the build minutes to reset which should happen at the end of the month, and I’m also waiting on Github support because for some reason the main workflow has stopped being recognised properly and won't start. I’ve debugged it using all the tools I know, including a special Github Actions linter called actionlint, and I can’t figure it out. You get all the way to the end and right before the last hurdle and the whole thing stops working. I’m getting a strange sense of deja vue, it’s almost like something like this has happened before.

Anyway, hoping to have something working soon :) #

Upgrading my social medias

The tweeks I made to the Mastodon auto-poster I use have started to filter through, and it’s looking a lot better. It‘s way easier to see the difference between blog posts, links and notes and the cards that appear no longer contain ugly looking unrendered markdown. Pretty happy about that.

One of the other things that I think will make a difference is that I’ve decided to change the way I format the linkblog links. The social media sites tend to have post character limits, so the links weren’t fitting. In recent years I had been using the link’s title and a short comment, based around how many javascript newsletters I subscribe to tend to do things. Well that’s not going to work if I’m posting the links to the socials, so the links will now have much smaller amount of text. Still getting used to it.

I updated the aliases and scripts I use to post links, so it’s much quicker to post a link now. Previously I had to rename files after creating them, but that’s all automatic now. It’s all just a oneliner on the command line. Perhaps that will mean I can post more things. We’ll have to see how things go.

The bigger news is that a few hours ago, I got the first version of my RSS to social media auto-poster working. It runs in a Github Action, is built from reusable workflows, and uses the matrix feature to run a separate job for each target destination. It’s not live yet, still quite a bit of things to test, but it’s looking very cool. #

The combination of the Nominative Determinism and Last Mile Synchronicty of this piece about a duo successfully rowing across Pacific Ocean is quite something. #

It occured to me that one of the advantages of having gone down the route of having many plugins for my ssg, even though that became a bit of a headache, was that my github actions made very good use of reusable workflows. I spent a bit of time the past few days seing if I could repurpose the same structure to build an social media auto-poster. I’ve made a ton of progress, and have the core primitives working.

In some ways it feels like I am very close to get it working, yet I know from experience that sometimes getting that last 20% working is deceptively much more work than first appears. All that to say, that I am quite happy that is seems like I might have something working soon, but a bit tentative because I know there are likely a lot of small details that need to be figured out. I’ve been here many times before, and I can feel the trauma of past experience holding me back. #

Github Actions RSS to social media auto-poster?

I‘ve had my head out of social media for some time now, but I checked in on Mastodon a few days ago, looking into some inconsistencies in the blog’s RSS feeds, and I discovered that the auto poster I had been using has been posting but all the posts looked horrendous. It’s a combination of several things, including changes I have made to the blog over the past few months, changes I made to the RSS feeds, and having the auto poster settings that no longer made any sense. It was a complete mess.

I have spend a few hours updating the auto-poster so at least it has the right feeds and settings now, but I feel like I need a more cohesive strategy, because once I have it solved for Mastodon, I will have to do the same for Twitter, and Bluesky, and Nostr etc etc. I had a quick look for a commercial product, but I couldn't find any that had all the features I wanted, and that were affordable. I don’t actually want anything too complicated. It’s starting to feel like I will need to roll my own.

What I would really like is to have a Github Actions workflow that runs every so often, read an RSS feed, and posts the latest things in the feed to social media. The simplest would be some sort of CLI tool I could run, but I haven’t found anything yet. I did find several community made actions like this one, and this one, and this one. None of these feel very maintained. There is of course Textcasting, but it’s not at all obvious how I could turn that into something I could run in a Github Action.

I feel like this was something we used to do much more easily in the early days of blogging. Maybe I’m not remembering clearly, perhaps it was a mess back then too. Anyway, just blogging about it in case the universe is feeling like going in this sort of a direction.

It’s completely wild that so many millions of people apparently use these social media platforms, and have been for decades, and that there are so many millions of folks apparently writing open source software, yet something as basic as an RSS auto-poster isn’t a solved problem. You would think that this would be almost the first thing that would get solved. What is up with that? #

AI agents vs AI models

I thought this bit from a recent Changelog Interviews Podcast [31:26] was very interesting.

Deepak Singh: "What you are using in Kiro is not Sonet 4. You are using is an agent that uses Sonet 4, for the core thinking part of it’s work. There is a set of scaffolding that goes around it. Everything from your system prompts to the way you handle caching, to the way you are handling any guard rails that might be in the picture. Sonet 4 between lets take two editors, they both say Sonet 4. They are not equivalent. It’s how you drive context into the LLM, and the scaffolding you have around it that makes it useful or less useful or more."

This is something I had suspected for some time now, but it’s great to finally hear it from someone working in the AI/LLM sector. I first noticed it when using the Continue Vscode extension, which connects to LLMs you have running on your local machine. There were pretty wild differences between what I was getting back from the LLMs via the extension and what was coming back from the CLI. After a bunch of testing I concluded that there must be some sort of entity, some sort of intelligence inside the extension. This is one of the reasons I spent quite a bit of time putting together some network monitoring tools and a way to visualise external connections that are being made. Also it’s yet another reason you want to be running all these AI/LLM softwares in containers.

I find it really strange that this is not common knowledge. I have been working with LLMs for several months now and it’s the first time I have had this confirmed from somebody working in the industry. And it’s not like this is just developing, when you look at the way for example the Continue extension is structured, it’s whole point is to create these agents, which are completely distinct from the models. Yet the way people talk about it, this fundamental architectural reality is completely glossed over. It’s bizarre IMHO.

BTW, this was why I mentioned in my post earlier in the week: "likely there are some advantages to connecting directly to the models".

If you are getting strange responses, be sure you aren't being LLM-in-the-middled. #

Are my GPUs being used?

I’ve been using the local LLMs quite a lot today, mostly via Open WebUI. I started to suspect that the GPUs were not being used because the conversations got very slow. Long wait times between prompts. And then the models appeared to run out of context window, with errors in the UI. So I have spent the past few hours verifying that the GPUs were still being used. Interactive conversations using llama-cli in the backend are extremely quick, yet over HTTP they get very slow.

In the end I was able to verify the GPUs were being used in both cases. First by looking at the startup logs. There are messages that say the vulkan device (GPU) is detected and that a number of layers have been off loaded to it. So at least at startup things are operating correctly. But how to verify during normal operation?

Turns out you can do this using just Activity Monitor. I had of course tried this, but I didn't know you could open Window -> GPU History, and get a nice real-time graph of utilization. And sure enough during conversations with the local LLMs you can see the utilization spike and then return to baseline.

I also installed fluidtop which is a cli tool for real-time utilization of CPU and GPU cores, specifically aimed at the new M* series Macs. Works really well. Also, my first time installing python stuff using uv. Also works really well, though the commands you end up having to use to run the tools you install aren't exactly rolling off the keyboard. Had to add an alias to my shell, which isn’t the worst thing in the world I suppose, as long as the tool works and I don't get lost in another python stack trace debug horribleness.

It seems then that the slow down is due to how the application, in this case Open WebUI, is using context windows. But it gets very very complicated very quickly. With the cloud models you just don't have to worry about this, at least not until the conversation has gotten stupidly long. With the local LLMs it seems to happen after about 10 chat messages, especially if you are posting terminal output into the chat. And then you open up the settings and peer in and it’s the most complicated unintelligible tangle of settings you have ever seen. And you quickly close and exit.

I dunno maybe I am being overly dramatic, I do like all the functionality, but you almost need an LLM just to help you figure out the settings that control the LLMs. Surely that’s not a good sign. #

Why do the AI/LLM folks hate people that run Macs so much?

Since I now have GPU accelerated local LLMs for chat, I thought it might be a interesting thing to see if I could do the same for the other types of media. That's basically image and video generation, as well as audio and music generation. After several hours of web searching and asking Gemini, I’m starting to get the impression that much like how it was with the text chat LLMs, getting any of this working on a Mac is not at all obvious. I guess all the AI folks are either running on linux or windows. They certainly are not doing things day to day on Macs, at least not in containers, and as for as I am concerned that’s the only safe way to run AI/LLMs on your computer.

The 2 big platforms seem to be Pytorch and Tensorflow. Just to narrow it down to those two was quite a journey. There are tons of different software, but ultimately it always boils down to running the models, and as far as I can tell, that means Pytorch and Tensorflow. And that’s important because these don’t support vulkan out of the box, so you have to compile from source and enable vulkan backends, which is not easy to do at all. The reason you need vulkan support is so the models can access the GPU from inside the container.

It really feels like another massive uphill battle. Not being funny but why do the AI/LLM folks hate people that run Macs so much? #

Yesterday evening I made a start on a small refactor to the backend API handling of scopes. It should ultimately result in fine grained access tokens that make a lot more sense than they currently do. Trying to get my head back into typescript after several weeks of quite intense bash and dockerfiles. #

Mostly cleanup tasks today following on from all the recent progress in the dev environment and on the blog. I need to remove all the old projects that got consolidated and I need to check a few things. I noticed this morning that in Feedly the everything RSS feed for the blog wasn’t showing any posts, so there might be something broken there, I need to check all the RSS feeds. #

Just use Llama.cpp

It occurred to me this morning that in yesterday’s piece I forgot to mention the route I ended up going with for the local LLMs. I had been using Ollama for several weeks, and though the features are pretty good, certainly as far as downloading and managing models, they do not support vulkan, and that’s the only way to get GPU acceleration working in containers on a Mac. What’s more it appears as though they have no real interest to add this functionality. Ollama actually runs ontop of Llama.cpp, and that project does support vulkan.

Since I got an impression that at least some people have been going back to running the core project, the next thing was to just try out Llama.cpp directly. It certainly wasn’t without issue, but the llama.cpp folks were helpful. It’s a bit more hardcore, because it doesn't do things like model switching out of the box, but in reality you don't need anything too fancy. There are some model switching proxies you can setup, but I found these to be overly complex. For my uses, some simple npm scripts is good enough, and likely there are some advantages to connecting directly to the models. I also had to write a simple script to download models, basically curl inside an until loop, with a persistent cache folder.

I'm trying to avoid unnecessary complexity, things are complicated enough when you are developing in containers. #

GPUs, module upgrades and more site fixes

Having made so much progress with my voyage into Typescript, React and containers, which resulted in a fabulous new project structure, even if I did have to completely rebuild everything from scratch, I still had in the back of my mind the annoying thought that the GPU acceleration for locally running containerized LLMs on my mac remained just out of reach.

Well, given how important AI/LLMs are turning out to be, I decided to give it another go. Long story short, I got it working. It required compiling all sorts of things from source, and creating my own custom build, but it's working, and holey moley is it fast. The first time I ran it, it was so quick that I couldn't quite believe it. It was instantaneous. It felt like the response was streaming to the terminal before I had unpressed the enter button. That was with the smallest model I have called Qwen. I tried a variety of models (1.5b - 8b parameters), and they were all really really quick. What was previously taking 30-45 seconds now takes 4-5 seconds. I even downloaded a very large 13b parameter model and though it takes around 20 seconds to load initially, it's then incredibly quick.

I still haven’t had a lot of success actually working on any code with the local models. So far I have found the cloud models to be vastly better quality-wise, but perhaps it’s a question of having to work in a different way. I get the impression that the cloud models are all tuned and configured, and perhaps the local models require some tweeking. In any case, having the GPU acceleration at least makes it a possibility. I think it’s really going to make a difference.

Other than the GPUs, there was a refactor of my static site generator code that I have been wanting to do for many months. When I initially got the plugin system working I chose to create separate plugins for the blog, linkblog, newsletter, notes and podcast. It turned out that having so many different plugins was a massive headache, because I found myself continuously jumping between projects. There ended up being a lot of duplicated code, almost but not quite similar between projects, and that led to a lot of cognitive overload. I figured out a way to combine all these plugins into a single plugin a while back, but wasn’t sure it was going to work. After carefully reviewing the code, it looked like it was going to be possible, and it took about a day to get it working. I managed to completely remove a complicated abstraction layer that was now totally unnecessary, and the code is looking really streamlined. It’s so great to not have to switch contexts all the time.

Once I got that working, I figured it would be a good idea to update all the modules in all the projects to the latest versions. It’s been a few years since I‘ve had the chance to do that, so many of the modules were really really out of date. Several have become ESM modules, so I had to be a bit careful not to fully upgrade a few of them. The upgrade process went quite smoothly right up untill just before the end, and then I ran into a narly issue. A while back I got interrupted midway through quite a complex refactor, and very complicated life stuff got in the way and I completely forgot about this refactor. Things had been working because of the module cache I had put on the main repo. I had to delete the module cache while upgrading the modules, and of course everything stopped working. Anyhow its a couple of days later now, and I got it all working, including removing the temporary heredocs feature I had put in place, because it’s no longer necessary, and actually was making things a lot more complex than they needed to be. It feels really great to finally be in a position to get it all straightenned out.

With all that done, and following on from the header image work just over a week ago, I spent today making some much needed modifications to ensure there is a more consistent design across all pages on the blog. You might have noticed that the main page has boxes around posts, which gives a sort of appearance they are all inside what I think is generally referred to as Cards. I really like the look, but it wasn't consistently applied on all pages, with individual post pages, and many one off pages not having boxes at all, and also the spacing between the page titles and the edge of the boxes was inconsistent. The other thing that has been bothering me was the really long list of links in the right hand site navigation. It had gotten very unwieldy. I’ve made some updates so all pages now have nice boxes around the content, for example the about page and my portfolio page now have this. And the navigation is split into two groups gathering all one off pages together, and all content related pages together, each with a box around them. I think it makes it a lot easier to read. The new navigation should also be working on small screens.

So yes lots of progress. Some of it has been a long time in the making. It’s really nice to have things looking better on the site and to know that the foundation of both the site and my development environment are in much better shape.

Onwards :) #

New blog header image

I've updated the blog header to have a background image. If you've ever visited the homepage of the site you will recognise the mountains from there. Perhaps the bigger change is that I have made the blog page the main page of the site. I moved the old homepage here. The navigation on mobile should look a bit nicer and be a bit more ergonomic. You might need to reload the page a few times for the css to update. It's nice to finally have a bit of color on the site :) #

The new container based project structure

The past few days, I have been writing about the recent project rebuild I carried out, as well as some thoughts on the modern web based stack as it relates to AI/LLMs. I wanted to spend a bit of time reviewing my new project structure, highlighting the major features.

Develop in a single devcontainer, deploy to service containers

Initially I made the mistake of trying to containerise everything into service containers, both in development and in production. What I didn't realise was that the vscode devcontainer does much more than just pass your settings to the container management tool. It does a whole load of things that prepare the file system so you can edit the files on your host directly in the devcontainer. I thought it would be great to have a complete replica of production in development, with each workspace running in a separate service container. What ended up happening was a never ending cascade of impossible to debug file permission errors, because especially on a mac, your host file system has to go through the podman vm's filesystem and then to the container filesystem.

Just go with a much simpler strategy, in development use 1 single devcontainer, that contains all your project's code, don't mess around with service containers and podman-compose. It's not worth it. But for production, do spend the time to create separate service containers for each workspace. In production you don't need to worry about file permissions in the same way because you don't need to edit the files. I have a top level project folder called containers where all the Containerfiles (dev and prod) live.

Self contained npm workspaces

I was already using npm workspaces for the frontend, backend and shared modules of the app, but they were not very self contained. Each module was also scattered around the project root. I moved all the modules under a single packages directory, and I made sure that each module was designed with the notion that it should be able to function independently, if it was pulled out into a separate project. This is great for future proofing but also for being sure that your modules can be run in separate service containers. That means they should have their own linting, and typescript configurations, as well as npm scripts to typecheck, lint, build, start, etc. Use the same script names across the modules so you can run them via npm run from the root of the project using the -w flag to specify the workspace.

Well thought out and modern typecheck and linting configuration strategy

Unfortunately the typescript configuration had evolved along with the project, which meant it suffered from a lack of a cohesive strategy, because the AI/LLMs will give you different advice depending on when you ask them. I had just added various configurations as I went along. It's really important to get both these configurations right in your minimal project, because once the full project structure and files is there, it becomes much harder to think about and troubleshoot. One important thing to remember if you are using Typescript in both the frontend and backend, is you will likely need a shared workspace to house the types that are shared between both. This is where the typescript configuration comes unstuck, because the compiler will refuse to see the shared workspace, and you will end up going around in circles with the AI suggesting the same set of changes, each one confidently touted as the "final industry standard best practice that will fix everything", and each time it will not work. Another reason why having a single devcontainer is better than service containers in development. Minimise the amount of things that could go wrong. Get it working for a super simple frontend and backend workspace that both use a shared workspace. I just used the example expressjs and vite react app.

Linting

Base config in shared module, other workspaces import and use this in their own config where they can override bits if needed, separate out style rules to use @stylistic, project wide rules for imports ordering using perfectionist. I like all imports to visually look the same in each file, with the same ordering. It significantly reduces cognitive load. The ordering I use is vendor modules, vendor types, then project modules and project types, each separated by a single empty line.

Typescript

Unified and Modern, ES Modules (ESM) as Standard, with configuration symmetry between backend and frontend, every workspace has a tsconfig that references app tsconfigs, and if needed references a tools tsconfig (frontend), and conscious separation of typechecking rules and build optimization, which is handled by specialised tools. I based this on the config from the vite typescript example app which had the most modern looking config that uses ESM, which is the best as far as future proofing your project.

Githooks using husky

Run lint using lint-staged pre-commit and lint+build+typecheck on pre-push. I also have a custom commit linter script that I run on pre-push that enforces Conventional Commits standard for commit titles and a related naming convention for branch names. It makes a huge difference to cognitive load during development, because at any time you can run git log --oneline and get a really great summary of what has been happening on the project, that is super easy to parse at a glance.

Bind mounted volume for node_modules folder in the devcontainer

The devcontainer will automatically bind mount your project files into the container, but what you can do also is to create a separate volume mount for your node_modules folder. This is a necessity if your host OS is different to your devcontainer OS because it means some of your modules will need to be compiled for different platforms. You can easily add this in the mounts section of devcontainer.json. You can also add mounts for things like ssh keys, gitconfig and even your .devcontainer folder which you can mount readonly. Typically you will want to install your projects modules (i.e. npm install) in your devcontainer's post create command. One gotcha is that if you are installing any global modules via npm -g, you will likely get permission errors as the globals directories are owned by root. What you can do is reconfigure npm to install it's global packages into a directory that the non root user can write to (e.g. $HOME/.npm-globals). You can do that using npm config set prefix "$HOME/.npm-globals" in the post create command before you npm install.

Building production service containers from workspaces, multi-stage builds

The big idea here is that each workspace should run in it's own service container, with only the files it needs to run. I have a development Containerfile (podman equivalent of a Dockerfile) for the devcontainer, and production Containerfiles for the backend and frontend service containers. It's a bit tricky to get this right, but the general concept is for each service container (i.e. frontend and backend) to do the build in 2 stages. In the first stage you move across all the files you need to build your code to a builder image, and you do what you have to do to generate your dist folder. And in the second stage you copy across your dist folder from the builder to the service container.

The backend is a bit more complex than the frontend, because you have to build your code using the shared workspace, and then make sure to package up the shared module into a tgz file, which you can then reference in your package.json. The frontend also needs to be built with the shared workspace, but since you just output frontend files, there is no node_modules folder to worry about. I use a typical slim node image for the backend and for the frontend I use the unprivileged nginx image, so you can run it as a non root user. You need to re-configure nginx slightly, so it proxies any api requests to the backend.

Easily test at each stage of development

I have npm scripts so it's possible to run each workspace in dev mode (automatic restarts using vite / nodemon), and run lint and typecheck, but also to build the development and production images, and another to build and run the production code. I also have a podman-compose triggerable via npm that runs both the production service containers, so it's super easy to run things in dev and in prod. Something I haven't done yet, but will at some stage, is to run a minicube in podman, which is a local kubernetes cluster, where you can test full blown production deployments all on your local machine.

Keep local LLMs in a separate project, but connect them all via a private dev nework

Running ollama was one of the reasons I ended up down this path in the first place. Initially I had it in my head that it would be best to have ollama part of each project. But that really increased the complexity of things. What was much much easier was to have a completely separate ollama-local project that runs an ollama and an open-webui service container, using the standard images distributed by their respective projects, using podman-compose. That way you can really easily spin up a local AI/LLM cluster with the models and a web chat interface. The trick is to configure them to be on a private network that you create separately. You create the private network on login to your OS. On mac that's via launchd. It's just a bash script that checks if the network is there, and creates it if it isn't. Then you configure the devcontainers in your projects to be on that network, and when you start them up, if the ollama local cluster is up, your vscode extensions can connect, but if they aren't then it won't crash the container. Another thing worth doing is to store your models in a persistent location on the host, which you then bind mount into the ollama container. That way you don't have to re-download all the models each time your rebuild the container.

Backup and restore for your podman images

Rebuilding podman vm should not be difficult. At first I was super worried about recreating the podman vm, because when you do that you lose all your images, and since some of these can be very large, it can take ages to re-download them. I created some bash scripts to save images to a backup location and another to restore them all in one go.

Repeatable way to download and install assets into your containers

This was another useful script to write, which downloads an asset into a persistent location in your home directory, and copies it into your project's assets directory, which you add to gitignore. Have an npm script to initiate all project downloads. It means you can download everything you need when you have an internet connection, but you aren't blocked if you go offline. For example I download the neovim tgz, which later during the development image build, I install via a script that I copy across with the asset to the image, install and then delete the extra files.

Wrapping up

So those are the main things about the new project structure. I'm pretty happy with it as it feels very lean and efficient. It's only been a few days so far, but it looks good, and now that everything is working and there aren't a billion errors everywhere, containers for development definitely feels like the right approach to these modern web apps, especially when it involves developing with AI/LLMs. #

Dylan Field (CEO Figma) [09:02]: "Is natural language the interface? Yes right now. I‘ve said this before but I really believe it, I think we will look back at this era as the MSDOS era of AI, and the prompting and natural language that everyone is doing today is just the start of how we are going to create interfaces to explore latent space. So I cannot wait for an explosion of creativity there. I think of these models as an N-dimensional compass that lets you explore this wild, unknown, fog of war in latent space and you can push these models in different directions, through natural language, but if you have a more constrained N there, and you are able to dimensionality reduce a bit, so you can push different ways, there should be other interfaces available than text. These might be more intuitive, but they also might be more fun to explore, and I think sometimes constraints unlock creativity in ways people don't expect. So I'm existed for that, but yes right now natural language is where we are at." #

How to rebuild a project from scratch

Rebuilding a project from scratch is one of those things that you never really want to do, and so there is a tendency to avoid it. But sometimes you realise that some or all of the foundations are compromised in some way, and the only way to fix them is a complete project rebuild. That just happened to me, and though it was pretty scary, and a lot of effort, I got through it, and it actually wasn't as bad as I thought it would be once I made the decision to do the full rebuild.

Keep in mind that I was the only person working on this repo, if you are working on a repo with others, definitely discuss with them before undertaking anything like this.

I think this is a situation that will become more common place now that many of us are using powerful AI/LLM tools, because as much as you can gain tremendous ground in build outs, it's very easy for instability to creep in without you noticing. This is something that would have happened anyway without these tools eventually, it's just that you reach that point a lot faster. Anyhow you will likely find most of this is just common sense, but when you are in the thick of it, seeing the woods for the trees isn't always obvious.

So what happened? Well the past month or so I have been building an Oauth 2.0 REST API backend with a React frontend. And I have been using AI/LLMs and devcontainers. Quite a few ups and downs, but I was making some seriously great progress, at least from the perspective of the application code. Both the frontend and backend were super solid. I was on solid ground, or so I thought. While experimenting with running Ollama locally on my laptop I uncovered some problems with my dev environment. The devcontainers I had been using started crashing on startup. I tested various configurations, and that uncovered some misalignment in my typescript and linting configurations, and before I knew it, I was in a giant tangle of errors.

I am big into writing minimal projects. Over the years, I have written many of these, I have a special section dedicated to them in my portfolio. When you get into difficulties it's often a great way to find stability again. There is a real art to it. In this specific case, things were so tangled that it wasn't obvious at the outset what exactly the minimal example should be. I thought initially that what I needed was a minimal example of an ollama based webdev environment, and spent a while doing that. Eventually I realised that ollama was complicating things, and that actually what I needed was a NodeJS Typescript Express React minimal example. A minimal project all built on containers. That was a bit of a voyage in itself, but once this new well structured project was working, the big challenge was to backport it into the original, much more complex, REST API + React project.

I thought initially that I would be able to move over pieces of it bit by bit into the old structure, reshaping as needed. It became obvious that was just never going to happen. I spent hours just pushing small pieces around, testing and seeing everything was still broken, and being totally overwhelmed by the enormous mountain of files, all out of shape, plagued by lots of false starts and bits sticking out all over the place. It was a monstrosity. I knew I now had a known good structure, but I didn't want to start a fresh project because I didn't want to lose the project history. There was a lot of valuable information in there. Here's what you do.

• First of all, backup the main branch, call it something like main-backup. Also backup the entire project into a tar ball.
• Ok you backed everything up. That's good, but also very very important, backup the project git directory (.git). It's a hidden dotdirectory. You will only be able to see those with ls -la.
• I decided to not use AI in Vscode for any of this. It has a tendency to try and take over and lead you down terrible paths. I did use AI but only via web browser. I was using Gemini.
• Create a new feature branch, call it something like refactor-new-project-structure.
• Make sure you are on your new feature branch, and then delete everything! Literally rm -rf * in your project directory. You read that correctly. This is the scary part. Remember you have backups. If you don't have backups, go back to the start of this list and make backups! Remember to delete all the dotfiles and dotdirectories in the project directory too. I just did that one by one, after deleting all the regular non-hidden files. One important exception: Don't delete the .git directory. That's where all your project git history is stored!
• Recreate / copy across your minimal working project to your new feature branch.
• Get that working, test it's all working. There is no reason it won't work since you already got it working in a separate project.
• Now pull bits in piece by piece from the old main branch into your feature branch. It's a bit tedious, but you can do this relatively easily. Take your time. Don't miss anything. List, view and checkout files and directories direct from the main branch using these commands:
  • git ls-tree --full-tree -r main
  • git ls-tree main:some/folder
  • git show main:path/to/some/file.js
  • git checkout main -- some/folder
  • git checkout main -- path/to/some/file.js
• Add and commit each piece with a descriptive title. Keep testing it's still working along the way. Ditch anything you no longer need from the old main branch.
• Eventually you will have recreated your project, and it will all work again!
• You will have a long series of rebuild commits. I had about 30 commits.
• Backup the rebuilt branch, call it something like backup-rebuilt-project.
• At this stage, things were still kind of complex, so I asked AI to help me group these into 6-7 groups of related commits that describe the rebuild process, so I could squash them into a more focussed history. I copied and pasted into the chat the relevant commits listed from running git log.
• The AI came up with a great rebase plan, with the commit titles and the commits all grouped. I just had to copy and paste into the rebase in neovim. It also prepared titles for each of the 6-7 squashed commits, which I copied and pasted into the rebase as it progressed. I rebase squashed the slightly haphazard 30 commits into 6-7 tidy and focussed commits.
• At this stage, make another backup of the rebased branch, just in case!
• Now change to main branch, merge in the squashed branch. In my case there were literally no conflicts, which was a relief but not too surprising.
• Backup the merged main branch into your final branch that you will merge in on Github, call it something like refactor-new-project-structure-final.
• Reset your local main branch to the state of the Github remote main branch.
• Push your final merged branch to Github and create a PR.
• Merge the PR into main. I got the AI to write me a comprehensive PR message that summarized the rebuild.
• Pull latest changes to your local main branch from remote, test it's all working on main branch locally.
• Now that everything is merged into main, and tested, cleanup by deleting all your backups and the feature branch.
• You are done! Do a retrospective on the nightmare you just completed. What did you learn?

You see it actually wasn't that bad, mostly common sense.

In my case the biggest structural change I had made was to move several top level folders which were all npm workspaces under a single packages folder. Git is quite clever and can mostly keep track of where files have moved to. Apparently having a focussed series of commits makes that easier. Aside from that most of the changes were to do with typescript and eslint configurations. The bulk of the code was completely unchanged. The only code that had changed was some of the npm scripts and Dockerfiles/Containerfiles. But there wasn't that much to merge in, just a few things that were in the original files that I wanted to keep, like installing neovim and networking tools for the devcontainer image. I just copy and pasted those across manually, and tested it was all working.

There were a few odds and ends to smooth out, like updating to a more recent development base image, so neovim had all the libraries it needed, and a bit of a refactor of how the backend production image assembles the final production node_modules folder to include the shared workspace. On the whole though, it was a lot smoother than I expected, even if it was kind if scary.

If you are in a similar situation, I know it sure isn't that much fun, but hopefully reading this will help somewhat. Sometimes it just helps to see something similar mapped out and to have the knowledge that it's possible. #

Containers, Typescript, React and AI/LLMs

I wrote a while back about how Typescript, React and Gemini was a pretty great combo. I still stand by that post, but I want to make add a slight refinement: Containers.

Yes, in a way it’s much more complex, at least initially, and you can definitely fall down some very dark debugging and configuration holes, but once you get the hang of it, it's kind of night and day compared to developing the old school way. With containers you have these neat little worlds all bundled and isolated that you can move around as a unit, much like an actual container. There is quite a lot to think about, but once you have your build process figured out, everything is very seamless. At any stage of development you can run a script and a few moments later the app that you were working on is neatly packages up, and you can test it, as if it were deployed, and then of course deploy it.

This becomes really interesting when you are working with AIs because you can create these environments where you can rapidly experiment and explore new things in a safe way, knowing that if anything goes wrong, you won't have trashed your base host machine.

The way I have things setup now, it's sort of like a digital version of a mechanics garage. I have several pretty solid dev tools like an IDE, and all sorts of CLI tools, and now with containers I can whenever necessary spin up a cluster of locally running AI/LLMs which I can connect to directly from within any of my projects in VScode or via a web browser chat interface. It's like there is this big mechanical robot attached to the ceiling, that I can pull down anytime I need to do something requiring something specialist to handle particularly tricky or complex situations.

It's not just local LLMs, I can connect up to cloud LLMs too, which are even more powerful.

React makes building dynamic frontends way easier, and the Typescript acts like scaffolding, giving you lots of structure to help you navigate difficult situations with the AIs. You are able to spot when things go off the rails way sooner than you would otherwise, and as long as you have some well thought out best practices, you can achieve some incredible things.

Things like guarantied end-to-end data integrity. By sharing types between the backend and frontend, creating your types by inferring them from zod templates, which are used to validate data on the backend, while simultaneously strongly typing all objects that you pass to the backend service layer, and doing the same with all responses to the frontend, you can get to a place where your app is incredibly robust. And I‘ve found that the AIs perform much better too, because they have a lot more information to go on as to your intent.

Lots of other things are possible too, here are a few that I wrote about previously in my post about recent project milestones.

And with containers you now control the entire environment within which your apps run, both in development and in production. When I built linkblog.io, I made a conscious choice to build without containers because at the time I felt they weren't ready for prime time. Back then I wrote about the robust NodeJS architecure you could build using open source software and cloud services, I knew at some stage I would eventually be moving to using containers, and there is little doubt in my mind that now is that time. #