markjgsmith

The latest big refactor

I want to make sure everything is still working as I‘ve just deployed the latest refactor of my publishing software. I previously wrote about the rendering tool, but this is the publishing side of things, it‘s the software that manages the content through a neat publishing workflow once you‘ve pushed it to Github.

As far as refactors go, this was kind of a big one. There had been a apparent misunderstanding between me and the LLM for quite a while. I thought we were on the same page about the architecture, so did the LLM, but we kept running into strange dislocations in the code. It‘s not always obvious immediately that this is happening, because sometimes the LLMs do come up with some really novel ways of architecting things, they have access to basically all of programming through all of time after all. Anyway the same sort of thing kept happening, but in different ways.

I eventually got to the bottom of it, and when I pointed out the ridiculousness of two entities that were clearly supposed to be different things, in the mind of the AI they were the same thing. Anyhow perhaps a story for another blog post.

For now just hoping this makes it onto the blog later tonight. Should know by morning if it worked. #

Gemini pretty down

Barely 24 hours after I wrote about Apple and Gemini and that I had been using Gemini a lot recently and now Gemini has had a makeover and looks great but is requireming me to click 10 times for each query :(

Strange. #

Apple and Google in an AI

Interesting to hear the latest from this years Apple developer conference. Apple‘s AI future for now will be with Gemini models. I‘ve been working a lot with Gemini the past few months, and it‘s very good a lot of the time, including for coding, which is what I mostly use it for.

I am struck, especially today actually, by how much Gemini likes to refactor code. In this world where all code is apparently one shotted, it‘s really quite odd how much refactoring one has to do to get anything worth having. I suspect one shotting is the new refactoring.

So Apple and Google...perhaps it is fitting. Always one more one-shottings away from the perfect software that will definitely be the most perfect software ever. This time is different. #

My blocked man switch worked

The Github Action I added to my blog to run at the start of every month worked. I added it because I got blocked again by Github last month, this time apparently for using hundreds of GB-hours (whatever that is) of storage even though I am using literally the smallest amount of storage possible.

One of the cool things about running a static site website, is I can continue to write offline and sync it back up when I get online again. I realized though that because I was only building the site when I push new content to Github, that once the new billing cycle came around that all the content I had written while offline wouldn‘t automatically get rendered into the site, because all the build triggers had been missed, and the new content would only make it to the site if I added some more content, which would trigger the site to rebuild.

With this extra little monthly workflow, I can be sure that any content I add while offline will definitely make it onto the site even without me writing something new at the beginning of the month. It worked last night, and all the content I added since I got blocked was rendered into the site.

The Github support team, who initially answered my support query, still haven‘t answered my question about what settings I need to use to not go over the storage quota limit. In the past Github support could take several days to reply to a support case but they wouldn't ever just go dark like that. That‘s quite disappointing, especially given how much time I have been spending developing on their platform the past few months. #

Vergecast going daily

The Vergecast Podcast is going daily. David Pierce interviewed Casey Neistat to get some advice. Casey is know for having run a daily vlog for years. I used to watch it.

I checked out Ze Frank, another one of the daily vloggers from the mid 2000s that got very popular. He‘s now doing nature related videos narrating in his style. His latest is about bees and wasps and ants.

I wonder what happened to Andrew Baron from Rocketboom. Another one of my favorite dailies from the mid 2000s, when video was just starting on the web. They used to do all sorts of things both indoors and outdoors.

Oh yeah there was also Mike Rugnetta and The Idea Channel. Actually not sure if this was a daily, but somehow in my mind it falls into a similar bucket.

As far as I know the Vergecast isn't video, but I‘m interested to see where they take it with an audio only daily pod. Potentially lot‘s of things to explore. #

Some great LTJ Bukem albums

I found myself needing some good music to code to the other day and after a bit of searching found an LTJ Bukem album I hadn't heard before:

• LTJ Bukem - Cosmic Twilight Vol.2 @ Tipper & Friends, Spirit Of Suwannee Music Park // 03.04.22

Seems like it was released in 2022. I haven't really been following the drum and bass scene for many years now, so it makes sense that I wouldn't have seen it. It's a great album. Really reminded me of the times I got into drum and bass around 1996. I feel like back then drum and bass was much lighter, things were very chill. Bukem uses the "Amen Break" a lot in his tracks. I thought he came up with it, but turns out folks had been using that sample for years.

Anyhow he's got several of his albums on youtube, Logical Progression being his most famous, and probably the best place to start:

• LTJ Bukem presents Logical Progression (CD-B mixed set, original 1996 version) Intelligent DnB

This live album is pretty great too:

• LTJ Bukem studio set '95 - Love Of Life "Master Mix" soulful dnb Cut 4 YT

Monthly catchup action

I‘m still trying to figure out exactly why the storage in my Github Actions has gone through it‘s monthly quota. Github support did get back in touch with me, but that got delayed because replying to the emails for some reason didn‘t add the message to the support case thread. I didn‘t realise initially so that wasted a few days. When I copied those messages into the thread, they replied with what looked like a pre-prepared reply with some links but not really answering my question.

I am genuinely confused by their storage quota. I have been using it for several years in the same exact way. Every day an action runs and builds the website if there was new content added, and during that build I create an artifact of the built website in order to pass it between two Github Action jobs. It‘s a pretty standard thing to do as far as I am aware. The artifact is around 100MB, and I set a retention policy of 1 day, which is the minimum amount. I don‘t even need the artifact to last a day, as soon as the build is finished it can be deleted as far as I am concerned.

Well Github have changed their billing system relatively recently, within the past few months, so I suspect it‘s got something to do with that, because I am literally using the least amount of actions storage as possible. The way it's always worked is as long as I was under the 500MB limit every day, and I always am because of the 1 day retention setting, then the quote is not exceeded.

But now when I look in the billing section they are talking about GB-hrs, and it lists 100s of GB-hrs. I have no idea what this means, but it just seems utterly ridiculous to me because I am as far as I am aware using the least amount of storage possible. I politely pointed this out to the support person, and they have gone dark on me. It's been several more days now and no reply. Anyway the billing cycle resets in 4 days.

I‘ve added a scheduled start to the build so that as well as building on push to main, it now builds once at the start of each month. That will hopefully ensure that any content I push during these weird periods were Github is blocking me for whatever reason they have decided to block me for this time, always makes it onto the website. #

Blocked by Github yet again

According to the Github account usage page I have used up all my Actions storage quota of 500MB, so it is blocking all my builds that use artifacts.The website's build uses artifacts to pass the built website between jobs, which is the Github recommended way to do things. Thus all builds of the website and thus all publishing to the website is completely blocked. I haven't even been publishing much the past few weeks!

This isn't the first time Github has blocked me completely but it is the first time it has done it on the basis of storage. In the past it has always been because of running out of free build minutes. The thing that is very strange about the storage quota is that as far as storage goes I am very conservative, setting a retention of 1 day on all artifacts I generate, and I only run 1 build per day, so it's very odd that I am over the quota suddenly. I have been publishing without storage problems in this, the recommended way, literally for years.

When I did some investigation on the command line using Github's gh tool I could see hundreds and hundreds of old artifacts on the system which would have totalled several GB in size. They were all "expired", because of the retension policy, but I would have expected these to get cleaned up. According to the LLM helping me I did everything correctly, and it looks like something on Github's side is suddenly trigering them to block.

Anyway I manually deleted all the expired artifacts, even though I shouldn't have to, so there are literally zero artifacts now, but even after a day the system is still blocking everything. The Github website says it will reset in 12 days.

They always seem to find some reason and way to block you, even when you spend years and years making everything perfect. I have emailed support, so hopefully I will hear back soon. This always seems to happen right when I'm putting the final fixes to software I'm building. Quite difficult to test a blogging tool when you aren't allowed to build your blog. The one small win is that since it's a static site powered by git I can still write posts offline, but right now it doesn't feel like much consolation. #

Bugs in mocking libraries

Bugs in code is not a lot of fun at the best of times, but finding a bug in a mocking library is the sort of thing that can really ruin your day, or week, or worse.

The past few days I‘ve been deep in a refactor, and I eventually hit a point where I was fighting the tools rather than the code. There were just too many moving parts, and it became impossible to see the woods for the trees. What appeared to be a straightforward bug in my integration tests turned into a two-day troubleshooting marathon. It’s a strange irony of working with LLMs: the ability to generate code so quickly means you can sometimes dig yourself into an even deeper hole than the one you are already in, with terrifying efficiency. You end up iterating on a flawed premise until you’re so far down you’ve lost all perspective on the original problem.

My way out of that cycle is to step away from the main project and build a "minimal". I’ve been doing this for years. I’ve even started a dedicated section in my portfolio to track them. The goal is to isolate the problem domain completely. By stripping everything back to a few lines of code and a single test case, the friction has nowhere to hide. In this case, it turned out that Node’s native fetch behaves just differently enough during a 204 handshake to throw Nock into a tailspin unless you explicitly override the transport.

I don't just build these repos to fix a single bug, they serve as a way to document the "why" so my future self can return to the problem and actually remember the details. It’s easy to forget the nuances of a library conflict six months down the line, but having a working, isolated example provides a path back to stable ground. It’s about being pragmatic, clearing the brambles in a small patch so you can see the through-line in your main project.

Here's the latest minimal: minimal-modern-node-http-mocking

The good news is that when I dove back into the main project I was able to fix the integration test I had been spinning on almost immediately. I feel like this latest minimal is a textbook example of when to use this technique.

Side note, a long list of very strange synchronicities for me this week, including what seemed like a name drop of the mocking library in a podcast I listened to moments after deciding to fix the bug. What's really weird, as far as I can tell they were talking about something completely unrelated to programming. It made no sense whatsoever in their context. #

Review and approve trending

Barely a single day after getting my review and approve publishing software fully working, and the White House annouces some sort of regime where they will review and approve AI models. I will file this away in the, at this stage, unbelievably large "what are the chances" folder. #

Recording integration test fixtures

Long day today, but managed to fix all the integration tests that had broken during the latest refactor. It‘s nice to finish the day with the latest build running smoothly. I had to re-record all the test fixtures, but that‘s been made a lot easier following recent changes I made. Previously I had to create all the test mocks by hand, which was really laborious and error prone, but now it‘s all automatic. Anyway, nice to have some progress :) #

More refactoring, more integration tests

This is another one of those test posts. I‘ve been busy with more refactoring of my blogging software. This time I‘ve been getting the integration tests working correctly. Thankfully the developer tools and libraries are pretty great. It‘s a bit laborious to get things setup initially and the LLMs have sent be down several pretty bad dead ends. But things are progressing well in general. #

Refactoring using LLMs

It‘s really quite amazing the sort of refactoring of your code you can get done when working with LLMs. They make it possible to see much much further into the future, and you can get a much better idea of whether something is possible or not, but also if you can't quite see the through line, but you know it‘s there, you can head off in the general direction, and work things out as you go.

This happened several times to me this week, I had a really good sense that it was possible, but it was really lot of very abstract changes, and I couldn't quite get a handle on it. Instead of fighting, I just said, what are the top 3 things we could do to the code base to put us in a position that would make the particular refactor possible. The LLM had a good sense of where we were heading as we'd been discussing it for a while, and so we charted a course and started on the top 3 things.

By the time we finished the top 3 things, it had cleared enough of the brambles and tangles that it was much much more obvious how the refactor could be done.

But don‘t fall into the perfection trap, you have to be pragmatic, there are times, where getting to stable ground is more important that having everything perfect. #

Another test post

Seems like the bug I fixed yesterday is properly fixed. It is at least with my links, this post tests it's also fixed in blog posts. #

Fixing bugs and weirdnesses

I hate to go on about it, but since I started blogging again it‘s been strange thing after strange thing after strange thing.

More bizarre and weird synchronicities than you can shake a stick at. I just fixed a bug in my review and approve workflow, the sort that you only find when you have it running in production. It took me all day yesterday to debug and write the fix. Anyhow I was just about to push the code to production and suddenly all my shells across my whole machine suddenly become unresponsive. All of them in Vscode, and all in iTerm2 and even the system Termnal. Had to reboot. That's two reboots in two days.

I was just about to write a test post to test things out, so this is the test post. I think I prefer the dull and boring test posts. Does everything have to be a high wire act? #