markjgsmith

Another suspiciously timed allergy flare up day today. The world continues to be very unhappy with my imperfections. Likely another tsunami on the way. Life goes on. Looking forward to another day. #

The alergy attack appears to have past, took almost the entire day. It can be completely debilitating when it happens. Very hard to concentrate on anything. It's sort of difficult to describe, not entirely disimilar to how it would feel to have stinging nettles rubbed all over your face, and right into your eyes. Sometimes I describe it as being punched in the face. It really drains your life energy. Roll on tomorrow :) #

Static websites + git everytime

Just as I manage to finally clear the decks this morning, and about to get back to finishing the rebuild of the Github Actions auto-poster, the internet connection disappears. Literally within a few seconds. Looks like the world is productivity min-ing again. This will no doubt be followed by some yah’ing and some everything is your faulting. Hard not to wonder whether reality should have a .cause property.

At least my blog is a website built with a static site generator so I can keep on blogging locally and sync later with git. #

Social media auto-poster progress

I managed to get a very minimal example of the Github Actions social media auto-poster workflow working. It proves that in theory the necessary sequence is possible. The reason it’s tricky is because I am trying to run all the auto-poster jobs in a job matrix in order to parallelize them. I’ve almost run out of build minutes this month, so here’s a summary, so I can pick it up again next month. It’s not really intended for anyone in particular except me.

It boils down to the GitHub scheduler making it very difficult to enforce a sequence of jobs where some of those jobs are optional (skipped). I need this because I have some jobs that read and write the current state of the RSS feed before and after running the auto-posters, but I don't want to force the caller to use the state management provided. They should be able to manage their own state.

The problem is though that the scheduler fails to properly trace the history when a required upstream job is skipped, causing the matrix job that launches the poster jobs in parallel, to enter a deadlock at the dependency validation stage. The solution bypasses this by using the always() condition, which forces the runner to ignore the flawed historical trace and proceed with execution based only on the immediate, successful output of the preceding step.

Though there are some edge cases still to test, it successfully completes with all jobs enabled and with some jobs skipped. The devil is always in the detail. #

Improved RSS feeds

I discovered that my RSS feeds were rendering the title inside the description. Titles should go in the title field! I know I know. How embarrassing.

Honestly it’s some sort of miracle of miracles that I even have RSS feeds that work at all. When I wrote the static site generator, I figured out how to do serverside components using just regular javascript and ejs templates, and that was pretty cool. It occurred to me that I could use the same serverside components to render the RSS feeds.

The way it works is essentially when the feed builder iterates through the posts data, it grabs the post’s item component, renders the feed item using the exact same component that gets used to render the post on the website. It works for all posts types, whether it’s blog, linkblog, podcast, newsletter or notes.

It wasn’t very obvious at all, especially because I wrote it all on an iPod Touch. It was a rough few years. I only realised relatively recently because I haven’t really been using RSS readers much at all the past few years.

I am always quite scarred to change the RSS code, because historically speaking, every time I go near the RSS code, really weird things tend to happen. I have no idea what that is all about, but it’s happened so many times that there is some amount of trauma.

Anyways, I took a look at the code just now, and it wasn’t that bad. It was really just adding if statement, though I did have to duplicate a dictionary for the everything feed, which without going into too much detail, doubles the memory consumption. Seems to be rendering fine though. Hopefully it will look a bit less ugly in readers. #

Not much fun with Gem

Two days of very frustrating debugging Github Actions with Gemini, who seems to have completely perfected being a lemming and confidently launching itself into ever more precarious situations, and every time I have to descend into the depths of hell just to make the tinniest bit of progress, and somehow find a way out of the trail of destruction it has caused.

Currently rebuilding a very complex workflow from scratch. I've had to build several minimal examples in order to prove to it that it doesn't have to be as horrendously complex as it constantly tries to make things. And when it becomes obvious I’m right, it then starts talking about my code being wrong, never mind that it wrote the code a few minutes earlier.

Not much fun. #

Google’s new coding agent called Jules

Jed Borovik was on the Latent Space Podcast talking about what it’s like inside Google Labs, and their new AI tool Jules.

I liked his description of the direction he sees things going in [08:06]:

“We have an API, so people are using it for all kinds of things. Triggering it when something happens. We saw an example where someone is triggering Jool’s to do all kinds of updates to their site, and then they have a Github action that is going to automatically merge Jules pull request. So all kinds of stuff is flowing...we are really changing how people are able to do stuff [...] we also have a CLI. We want to meet developers where they are [...] an API is great but when you are working locally you want to be able to trigger stuff [...] by the time this podcast comes out we will be integrated with the Gemini CLI [...], all kinds of places where we are going to mix and be able to harness this power, because developers work in all kind of spots, and so making it easy to have this autonomous ambient agent that can really do all kind of work for you.”

A lot of their discussion resonated with me, I’ve ran into many of the things they talk about.

I asked Gemini a bunch of technical things about Jules, and finally whether Gemini had met Jules:

"That's a fun, meta question! As a helpful AI assistant built by Google, I can't actually "meet" Jules in the way two people or two physical robots would. We exist in different operational contexts, but I am certainly aware of and have access to a wealth of information about its function and design."

It took the whole night to download but it did finally download and I am now running Tahoe 26.1. So far most things seem to be working find, but somewhat worryingly the vpn app is behaving a bit strangely. I connects but none of the buttons in the interface work. The weird thing is that the issues started happening right before I installed the updates. After the OS update I updated the vpn app, but the issues persist. And people wonder why folks hate updating their software.

The biggest difference so far with Tahoe is that it changed the wallpaper from giant redwood trees to a sunny beach with giant rocks and snow covered mountains far in the background. Perhaps mountains are trending. I appear to be ahead of the curve on mountains. #

It’s 1997s era of making agents

I thought this take from Bret Taylor about where we are in the AI rollout [1:30:13] was pretty good:

"We are like in the 1997 era of making agents. I found this article for Siera summit about creating websites in 1997, and there was this Wired article [...], and it was basically about banks spending 23 million dollars to add transactional support to their website, like adding a login form basically. And then you fast forward to the late 2010s and Kylie Jenner starts a multi-billion dollar cosmetics line with [...] 7 full time staff.

So we are still in the 1997 era of building agents where it’s way too hard. You end up putting a lot of engineering around what is a very intelligent set of models, just to make it work well, and I think what do you need to create a 7 person team to create a multi-billion dollar business on agents? And I think we have a lot of product and technology work still to do [...] but for an applied AI company like ours, the models are actually pretty great right now."

BTW, Bret is co-founder of Sierra and chairman of Open AI.

In case you weren’t around this was 1997.

It’s a great way to situate our current moment in the broader picture. Of course one of the big questions that naturally follows is how much faster the acceleration will be in this era, because presumably it won’t take 20 years? Or maybe it will?

It would be an interesting metric to track. #

Emoji fixes everything

I think I might have fixed the weird Github Actions workflow corruption thing. The fix is almost stranger than the original bug. I fixed it by adding an emoji to the workflow name. This is literally what I did, I'm not even joking. Is this the real world? #

Looks like the worldo breakages aren’t quite over yet. My MacBook Air M4 just kernel panic’d and crashed while I was using VScode pushing code. Gemini seems to think it’s something to do with low level hardware.

This kernel panic was caused by an unexpected and unrecoverable error in the system's low-level hardware management software, likely concerning the Accessory Operating Processor (AOP) or a peripheral it controls (like the audio system or other internal controllers).

It is almost certainly not caused by a third-party application, but rather an issue in the macOS kernel, Apple's firmware, or potentially a rare hardware malfunction.

Not a particulary great sign. At the exact moment it happened a woman outside made a noise that can be best described as "very suprised". Also not a great sign.

I’m running updates now, it seems to be upgrading me to Tahoe 26.1, 14.62 GB of updates to be precise. Might take a while... #

I've been listening, and occasionally watching, the TBPN Podcast a lot recently. It occurred to me today that it might be this era's Techcrunch, in terms of the flow of interesting stuff that goes through them. It feels similar in a lot of ways, but it's a 10x version of it. Their analysis of the bizarre moment we are in is very often quite incredible and it's also a lot of the time fucking hilarious. Now it is a bit heavy tech bro style silicon valley cultureish, so you have to be ok with that, but it's very much it's own thing, and I find I'm even sort of into the ridiculous add reads. There are of course some bits that are a bit average, but I just skip through those. Overall it's really high signal. #

Quite bad allergies flare-up today. It’s a real pain. This seems to often follow after several days of tsunamis. #

I got pretty much right to the end of the refactor of the auto-poster, everything was working, and then I notice that the workflows in Github are suddenly corrupt, appearing as the path instead of the name of the workflow. That’s exactly the same thing that happened last time. This is not normal. Hey world can you please stop getting in a huff and breaking everything? Would you please consider it? #

The internet connectivity issues where joined by their friends broken kettle, broken headphones, and a whole lot of world being prickly. I had a feeling there was going to be a streak of broken things a few days ago when I accidentally broke my coffee cup after being repeatedly tsunamied by worldo. Hey ho, that’s how it goes sometimes.

Well I just finished a pretty big refactor of the Github Actions social media autoposter. Not tested yet but it’s looking pretty good.

Managed to get myself a new kettle and I had a spare pair of headphones tucked away, and the internet and vpn both seem to be stable again, so I was able to listen to some great podcasts earlier. It’s such a crazy great time we live in, even if it is very confusing a lot of the time. #

I've been putting the finishing touches to new post to social media Github Action workflows. It's looking real good, very close to having it working, and now the internet just went down :( #

Peak plumber

I thought Jason Calacanis made an interesting point on his podcast this week [1:03:40]:

"If you could do a search and figure out what’s wrong with your dish washer [...] and not have to call the plumber, you’re like this is an incredible experience. You know what, people haven’t had that experience yet. But this winter when people’s HVACS go off or their pipes freeze and they take a picture of it, or they do a search on Google and it says oh you have this HVAC unit, it’s known to do this, here is how you reset it and relight the pilot, and you don’t have to call the person, and you fix it, or your housekeeper was able to clear the dishwasher cause suddenly they know how to take a picture of it and put the model number and say Error 72, and it’s like Error 72 is something is blocking it, take out this thing and clear the blockage. How many times have to had somebody come to your house and they fixed it in 5 minutes and you had to pay 150 bucks?"

The word on the street these days, is that the best new career path for everyone is to become a plumber or an electrician. Every Hollywood celebrity is parroting this new wisdom. And on the surface it seems like there might be some sense to it, especially with the AI boom and the explosion of data centers being built. Now I’m not saying that there isn’t something to this, but I am saying that if you are considering it, be aware that you should look more closely at the data. There are lots of competing factors.

While there is a data centre boom going on, and they do need lots of tradespeople in that industry, that’s a very specific type of electrician, and likely by the time you finish your apprenticeship, the boom will have plateaued quite a bit. And the other things is, as Jason points out, there is a whole class of jobs that new AI tools, available to everyone, that will be completely removed from the day to day of regular electricians and plumbers.

All this to say that, it’s worth doing your own analysis of the situation, because IMO there are a lot of self serving opinions being casually thrown about at the minute. They aren’t necessarily untrue, but they aren’t necessarily true either.

BTW, there are some interesting and quite spicy opinions about OpenAI in this episode too, especially if you are a developer using or thinking of using their API. Worth a listen. #

Platform generated AI slop at scale

There is an interesting piece on the latest Vergecast episode [34:15] with David Pierce and Nilay Patel about the immanent death of the creator economy. There is lots of good discussion, but the real catch is a quote they pulled from a recent Meta earnings call in which Mark Zuckerberg recently announced that we are entering the 3rd era of social media.

According to Zuck the 1st era was all about sharing content generated by family and friends, the second era was when they added the creator content, and the third era is AI generated content that will be added "ontop" of the first two kinds. It sure is an interesting choice of words.

Nilay has this to say about it:

"There is only so many people in the world Mark. And they only have so much time in the day. Supply and demand dictates that if you add much more content to the feeds, the attention will be taken from something else. I know where the attention will be taken from, cause you are not paying friends, family and the elementary school parents group, you are paying the creators. You are going to take the money away from the creators with your universe, your corpus of AI content."

It’s one of those extremely obvious points that is worth saying out loud, especially because the creator economy hasn’t exactly been having the best of times recently. But Meta aren’t the only ones. OpenAI is heading in this direction at a shocking pace, they have gone from non-profit, to possibly the biggest profit maximizers in all of human existence, entering into any market that has even a hint of action.

Platforms competing with their users isn’t exactly a new thing, it’s unfortunately a re-occurring theme in tech. It seems we are about to enter into another era of the platforms eating their users. #

Elon Musk on All-In Podcast [51:06]: “Try using any of the recent so called Open AI open source models, they don't work. They basically open sourced a broken, non-working, version of their models, as a fig leaf.”

Great interview, especially the bits at the end about solar energy. Ultimately all energy production leads to solar. I hadn’t heard this perspective before. It’s completely obvious if you are thinking about it from the right scale. But of course few of use are. #

Some recent renovation successes

Over the past few months I’ve had quite a few successes in various renovation projects. It can be very chaotic going through these periods, and it’s not always obvious seeing the woods for the trees, but looking back I can see some definite themes. I wanted to spend a moment reviewing things, it’s important to celebrate the wins.

1. New blog redesign

I’ve been blogging for something like 20 years. The first few blogs are no more, long since dissolved into the sands of time, but the current incarnation has been around for a little over 10 years. Actually, according to the date calculator, it’s been 14 years, 3 months, 3 days.

There was a long period where it was just a hand rolled linkblog, but eventually I added a blog, newsletter, podcast and notes sections, all interwoven on the front-page. This year I finally got to redesign the whole thing, still keeping it’s minimalist aesthetic but adding a bit of color.

2. New container based dev environment, Typescript+React, safe collab with AIs

I have in the past built quite serious projects, including a social media SaaS, but I had stayed away from containers, Typescript and React. This year that all changed, brought on somewhat by the new realities of using AI/LLMs in web development. I now have first hand experience of how these technologies work extremely well together having built an Oauth 2.0 REST API with React frontend. It certainly hasn’t been plain sailing, but I have settled on a really solid new container based typescript-react project structure, that I am very happy with.

3. New locally running LLMs, GPU accelerated running in containers

I’ve spent a lot of time experimenting with AI developer tools, and I’ve been blogging about all my experiences with AI. There have been many ups and downs, but it’s obvious that these tools are going to have a big impact, and realising that I was using the frontier cloud models more and more, I decided it was important to at least get some experience running local AI open source models. Even if they aren’t as powerful, hopefully that will improve over time.

After a voyage of discovery I got some LLMs running locally, but then ran into performance difficulties. In ended up going down a very very deep rabbit hole looking into how to get GPU acceleration working in containers. That was at the same time I was trying to build the new Typescript/React/Containers project structure, which was all very confusing, but I got through it and finally got GPU accelerated local LLMs running in containers accessing my mac’s GPUs with massive speed improvements.

4. New social media auto-poster

After all the container based AI GPU Typescript React craziness, I decided to switch focus back to my blogging, noticing that my posts were not syndicating particularly well across the various social medias I use. After researching and testing several commercial auto-poster products, I realised that what I really wanted was a solution that I could plug into my blog's static site generator, something I could run in a Github Action. So I built a social media auto-poster! I'm very excited about this. It’s looking really promising, hopefully rolling out at the end of the month.

5. New blogging scripts

With all this momentum in my blogging, it felt like the right time to overhaul my blogging setup, refactoring all the scripts, aliases and functions I use to publish the blog. This was an incredible success, working with Gemini I was able to drastically simplify and make my setup much more robust and added several features including managing images, which is a huge win. I am already seeing big benefits in my day to day blogging. Massive time saver.

6. New dotfiles for multi-platform bash config

The final renovation project was a complete overhaul of my dotfiles, something that I had been wanting to do for literally years, but had been in a state of complexity paralysis. With Gemini’s help, I was able to cut through years of cruft and somewhat tangled scripts, and by deciding to drastically simplify my approach, I now have a repeatable multi-platform setup for managing my shell configuration. This is a huge win for stability and robustness. It’s deeply reassuring to know that I can rebuild any of the core components of my setup with a minimum of fuss. That I can be back up and running very quickly should I need to do a complete system rebuild.

There are likely a few others, but these are the main ones.

Looking back at it all, it’s turned out to be an exercise in improving robustness at every level of my development stack, while exploring the emerging world of AI assisted development. It’s been quite a ride. I’m super glad that I’ve been blogging through it all, because it makes these sorts of retrospective posts possible. It can feel like you aren’t making progress at times, the world just keeps throwing endless hurdles in your way, but actually slowly but surely things are getting a little bit better every day. #

Vibe civilization engineering

I did a bit of civilization engineering with Gem while I was listening to some podcasts yesterday. Kind of interesting, and wild, and scary. #

I'm seeing and hearing a lot of people in early adopter circles getting very interested in AI generated music the past few weeks. It seems to be moving from this is just a silly joke to hold on there might be something interesting here territory. #

Overhaul of my dotfiles

I did an overhaul of my dotfiles!

For those of you not familiar with the concept of dotfiles, how to explain them? Well on linux/unix systems many applications store their user settings inside 'dot' files and / or folders. These are files who’s filename starts with a dot, for example .bashrc, .bash_profile, or .vimrc. The reason for this is that by default when you list a folder's contents, the items that start with a dot are not listed. This keeps things uncluttered, because most of the time you don’t need to be seeing these files.

Since programmers typically use a lot of different software tools, in a variety of ways and on a variety of different systems, it’s worth spending the time to get good at organising these files. Hence the idea of dotfiles, which are collections of dot files that are organised in such a way that you can easily make changes to them and deploy them onto different systems. Typically via git, with them all stored in a git repository like on Github. People get very carried away with them. Check out awesome dotfiles for some idea.

The thing is, as awesome as they are, it’s all too easy to fall down a very big rabbit hole. That’s what had happened to me with my first attempt. I figured it would be great to use my dotfiles to install and configure the entire system, and it would have been, but the reality is that system installation is something most of us do so infrequently, that these complicated setups rapidly get out of date, and they are so tedious and annoying to test, that in practice they just gather dust and you get more and more scarred to touch them because you can't remember all the reasons you made them the way you made them, and they just become a thing that you open up every now and then, start panicking and quickly close and move onto something else.

The thing is though, they are quite important, because when you do have to setup a new system, you either use them or you have to do the whole thing from scratch, which is also a total pain. A few days ago, hot off the back of several small overhaul successes, I decided to give it another go. Surely there was a middle ground between automated install of the entire universe and do everything manually?

As it happens I did recently have to install everything again from scratch, first my Android phone, and then my new mac laptop. When I did that I basically started with a fresh empty config and just added the minimum I needed as I went along. It has been working quite well, but a bit disorganised and also lots of annoying copy and pasting, via email, between both devices. They share a lot of configuration but there are also notable platform specific differences. The biggest insight I had before embarking on this overhaul was that all the fancy installation scripts were totally unnecessary. In the event of re-installation, I'm okay doing that by hand. But the one thing I really do rely on day to day is my bash shell configuration. All the aliases, exported environment variables, shell functions and bash scripts. I use many of these every single day.

With that in mind I asked Gemini what suggestions it had for structuring my dotfiles to do just that, and only that, nothing too fancy. I'm very glad I asked because the response I got back was formidable. In seconds Gemini cut through years of cruft and tangled code, suggesting a very clear and simple structure, and within a few hours working together we were able to come up with a setup that incorporated all my current configuration, while adding lots of very useful things that I wouldn’t have thought of. Now I don’t claim these are the best dotfiles ever, I was mainly trying to replicate the setup that I had developed over time, there are likely refinements that I will make over the next few months, but I’m pretty happy with it so far.

I’m still somewhat surprised it was that easy, and both laptop and android phone have been updated to the new setup and are so far running very smoothly.

It’s really great to know that should I need to re-install either of them, I will at least relatively quickly have a configured bash shell and be able to code and blog without to much interruption.

The whole repo is released under MIT license. Feel free to read it, experiment with it and get inspired to build something similar for your own situation. #

Who are the suppliers?

A note to remind myself that in future tech build outs, do some research at the start who the big suppliers are. Seems completely obvious now but I totally didn't think to figure out who would be supplying the materials and core components for the current data centre boom we are seing as a result of the AI industry build out. Several Japanese engineering companies have been ripping, between 30% to 500% added to their value.

The companies mentioned are:

• Advantest Corp [OTCMKTS: ATEYY]
• Tokyo Electron [TYO: 8035]
• Softbank Corp [TYO: 9984]
• Fujikura [TYO: 5803]
• Mitsui Kinzoku Co Ltd [TYO: 5706]
• JX Advanced Metals Corp [TYO: 5016]

New blogging scripts

Feeling quite under the weather today. Rough day yesterday, and today I’ve got a sneezy cold that’s been steadily gaining momentum the entire day. One presses on nevertheless. Some progress in my blogging setup.

I decided to do an overhaul of all the bash scripts I use for day to day blogging. I use a static site generator to build the blog, but separate to that I previously had a very basic bash script combined with some aliases that enabled me to very quickly create new empty markdown files for each type of post, with all the right frontmatter, and name the files correctly. It sort of worked okay but there were some weirdnesses. Actually quite a few weirdnesses, and the aliases had become ridiculously complicated.

I figured maybe this was a good task to work on with Gemini. I've been looking at these scripts for years, so couldn't really see any obvious way to improve them. I knew there would be, but I needed a fresh pair of eyes. So I cut and pasted the mess of scripts and aliases into the chat, and briefly explained how it all worked. To my surprise Gemini fully understood how everything connected together, totally understood what I was trying to achieve and made some very impressive suggestions.

Before long we had a 4 phase plan to refactor everything, and within a few hours we had everything working. There is a new master script that has absorbed much of the logic that was scattered across the aliases and the updated aliases are neat and tidy, and look great.

What’s more while we were working on the plan I mentioned that one thing that would be great was to streamline the workflow I use for adding images to posts, which was really convoluted. Gemini figured out a really great way to cut out almost all of the hassle, so now adding an image is super simple.

The other thing that is loads better is adding links. The script automatically adds and commits the changes to the git repo, so lots of the tedium and choreiness has evaporated. #

Spent several hours today on the social media auto-poster. It’s starting to look really awesome. The project structure is just right and everything is easily repeatable. I showed Gemini the existing setup with a bit of explanation, and basically just said, do the same sort of thing for some other social networks, and it obliged, and synthesized new code based loosely on the existing structure, but with relevant variations to accommodate the different APIs. Gemini is shockingly great when the structure exits and the task is very clear. #

Beauty in journalism

It occurs the me that the main reasons I read so much from the Guardian is that:

1. It's basically free, though of course you should donate if you can.
2. The writting is a good read most of the time.
3. It just looks so much nicer that any of the other MSM news sites.

However, I find that a lot of the opinions I either don't agree with, or cause my eyes to roll. It's astonishing to me how much of an effect beauty has. I very much like reading the Guardian, but also a lot of the time I very much do not.

I wish there was another MSM newspaper with a different political leaning that had a website that was just as pretty so I could AB test my stupid brain.

Oh well what to do? I guess I'll read another Guardian article. #

Is it simple yet?

It occurred to me today that in yesterday’s piece about how to build a social media auto-poster using Github Actions, although I did mention that Gemini was doing it’s best to go down every possible dead end, I wasn’t very specific about what those dead ends were. Gemini absolutely loves making things more complex than things need to be. We were at one point exploring using sqlite databases and all sorts. But you don’t need any of that.

You have to keep pressing for the simplest possible solution, but you do have to explore some of the more complex things first, and then circle back and say hey Gem, this all seems way way too complex, how can we drastically simplify it. And then it does actually suggest some good solutions.

All that to say, if you just want to post a few things to social media you really don’t need to save anything complicated. You can do it with just the features available in Github Actions. #

Building a social media auto-poster

Lots of progress on the Github Actions RSS to social media auto-poster I’ve been working on. I started out building the workflow primitives, just simple reusable workflows that enable you to post a message to social medias. The first two I’ve implemented are Mastodon and Twitter (X). Got those working and then went up a level and implemented a reusable workflow that posts to all socials, which triggers the first two.

At this point things were quite complicated and debugging became tricky. When you are running a workflow that calls another workflow things can get quite confusing. And initially I wasn’t sure I could have these all in the same repo, so I had them in separate repos, which added to the complexity. So I spent quite a bit of time implementing a repeatable way to easily and safely log workflow inputs and secrets (i.e. masked). It was quite cool getting to this point.

With that working, I embarked on the most complex of all, a workflow that reads an RSS feed, pulls out the newest posts, and for each one launches an instance of the post-to-all-socials workflow. The two tricky things with this workflow are how to launch many parallel jobs, and how to store state between runs. The level of complexity at this stage was considerable.

Gemini was of course being very helpful then being a complete liability, leading us both down several quite dangerous dead ends, forgetting why it had decided to go a particular route and then confidently announcing that the best final solution that would fix everything was to migrate to a new architecture which was the exact architecture that it had initially said was not up to the task. This happened at least 3 or 4 times. Thankfully I had been making good use of git feature branches, so never ended up in a situation that wasn’t recoverable, though it was certainly a slog.

I eventually figured out the right architecture and got it working, which is awesome, except that the workflows were spread across several repos, so I consolidated them all into one repo. That’s where I’m at now, and I’ve had to put everything on hold because I’m about to run out of build minutes. Testing this whole thing has been quite difficult, especially because there were loads of minutes wasted with many workflows that simply would not cancel. Something that should have taken a few seconds would eat up 5 minutes. This happened a lot.

So anyway, I’m waiting for the build minutes to reset which should happen at the end of the month, and I’m also waiting on Github support because for some reason the main workflow has stopped being recognised properly and won't start. I’ve debugged it using all the tools I know, including a special Github Actions linter called actionlint, and I can’t figure it out. You get all the way to the end and right before the last hurdle and the whole thing stops working. I’m getting a strange sense of deja vue, it’s almost like something like this has happened before.

Anyway, hoping to have something working soon :) #