GPUs, module upgrades and more site fixes
2025-10-11 23:34:35 +01:00 by Mark Smith
Having made so much progress with my voyage into Typescript, React and containers, which resulted in a fabulous new project structure, even if I did have to completely rebuild everything from scratch, I still had in the back of my mind the annoying thought that the GPU acceleration for locally running containerized LLMs on my mac remained just out of reach.
Well, given how important AI/LLMs are turning out to be, I decided to give it another go. Long story short, I got it working. It required compiling all sorts of things from source, and creating my own custom build, but it's working, and holey moley is it fast. The first time I ran it, it was so quick that I couldn't quite believe it. It was instantaneous. It felt like the response was streaming to the terminal before I had unpressed the enter button. That was with the smallest model I have called Qwen. I tried a variety of models (1.5b - 8b parameters), and they were all really really quick. What was previously taking 30-45 seconds now takes 4-5 seconds. I even downloaded a very large 13b parameter model and though it takes around 20 seconds to load initially, it's then incredibly quick.
I still haven’t had a lot of success actually working on any code with the local models. So far I have found the cloud models to be vastly better quality-wise, but perhaps it’s a question of having to work in a different way. I get the impression that the cloud models are all tuned and configured, and perhaps the local models require some tweeking. In any case, having the GPU acceleration at least makes it a possibility. I think it’s really going to make a difference.
Other than the GPUs, there was a refactor of my static site generator code that I have been wanting to do for many months. When I initially got the plugin system working I chose to create separate plugins for the blog, linkblog, newsletter, notes and podcast. It turned out that having so many different plugins was a massive headache, because I found myself continuously jumping between projects. There ended up being a lot of duplicated code, almost but not quite similar between projects, and that led to a lot of cognitive overload. I figured out a way to combine all these plugins into a single plugin a while back, but wasn’t sure it was going to work. After carefully reviewing the code, it looked like it was going to be possible, and it took about a day to get it working. I managed to completely remove a complicated abstraction layer that was now totally unnecessary, and the code is looking really streamlined. It’s so great to not have to switch contexts all the time.
Once I got that working, I figured it would be a good idea to update all the modules in all the projects to the latest versions. It’s been a few years since I‘ve had the chance to do that, so many of the modules were really really out of date. Several have become ESM modules, so I had to be a bit careful not to fully upgrade a few of them. The upgrade process went quite smoothly right up untill just before the end, and then I ran into a narly issue. A while back I got interrupted midway through quite a complex refactor, and very complicated life stuff got in the way and I completely forgot about this refactor. Things had been working because of the module cache I had put on the main repo. I had to delete the module cache while upgrading the modules, and of course everything stopped working. Anyhow its a couple of days later now, and I got it all working, including removing the temporary heredocs feature I had put in place, because it’s no longer necessary, and actually was making things a lot more complex than they needed to be. It feels really great to finally be in a position to get it all straightenned out.
With all that done, and following on from the header image work just over a week ago, I spent today making some much needed modifications to ensure there is a more consistent design across all pages on the blog. You might have noticed that the main page has boxes around posts, which gives a sort of appearance they are all inside what I think is generally referred to as Cards. I really like the look, but it wasn't consistently applied on all pages, with individual post pages, and many one off pages not having boxes at all, and also the spacing between the page titles and the edge of the boxes was inconsistent. The other thing that has been bothering me was the really long list of links in the right hand site navigation. It had gotten very unwieldy. I’ve made some updates so all pages now have nice boxes around the content, for example the about page and my portfolio page now have this. And the navigation is split into two groups gathering all one off pages together, and all content related pages together, each with a box around them. I think it makes it a lot easier to read. The new navigation should also be working on small screens.
So yes lots of progress. Some of it has been a long time in the making. It’s really nice to have things looking better on the site and to know that the foundation of both the site and my development environment are in much better shape.
Onwards :) #