markjgsmith

I‘ve written a lot about content creation workflows over the years. I‘ve been busy building some I think interesting things in that space the past few months. This post is my first attempt to describe the server component of my new publishing software.

If you have ever built a content-heavy site, whether it is a tech blog, documentation hub, or a machine learning dataset repository, you have probably bumped up against the same frustrating trade-off. Traditional CMSs feel clunky and locked-in, while custom Git workflows can quickly turn into a brittle mess of custom scripts. I wanted something better: a publishing workflow that treats non-code content as first-class filesystem assets, handles release schedules intelligently, and stays completely decoupled from vendor lock-in.

To solve this, I have been building a provider-agnostic publishing server designed to orchestrate the entire content lifecycle. While my client tool makes it simple to author and render content locally, the server component makes it easy to put those assets through a proper atomic publishing workflow. The server sits quietly in the background, receiving content updates, running them through flexible review policies, and handling the transition from individual file updates to consolidated daily release batches.

What makes it particularly cool under the hood is how intentionally decoupled it is. The core execution engine is entirely independent of where your files live or how they get reviewed, whether that is GitHub, another Git provider, or an internal storage system. The system's architecture cleanly isolates pure business logic from storage drivers. This means you can boot it as a lightweight, zero-config serverless app using just environment variables, or plug in a full relational database as you scale up to multi-tenant workloads.

By taking care of the tedious pipeline logistics, like handling time-based release cutoffs, provider webhooks, and pipeline resilience during third-party outages, the system lets you focus strictly on writing and publishing. It has been a game-changer for my own publishing setup, and it provides a solid foundation for where I am taking my publishing tools next.

Just got the portfolio page up to date with recent projects by adding:

• OAuth 2.0 REST API & React Web Client
• Local GPU-Accelerated LLM Workflows
• Automated RSS Content Syndication Engine
• Template-First Publishing CLI
• Open-Core, Provider-Agnostic Publishing Engine

I used Gemini to write them, which is why they all look exactly the same length. Gemini is so much better than me at summarizing READMEs and blog posts, but I need to go back through them and make them slightly different lengths. Right now just trying to clear the backlog.

One thing you might have noticed is that daily items in the new blog are chronological interleaved, as opposed to grouping the notes at the top and links as the bottom of each day. A couple of reasons for this change in direction.

First off, when I got the new design working, without all the boxes around the content, there was much more space, but also the left margin borders on links and notes still made these posts stand out enough that I felt like the new design looks like it might be quite well suited to this setup, to just post things as they are published.

But secondly, the render times are so much quicker, all done just under 2 minutes per post once you include pushing to Netlify, I‘m considering re-rendering the site on each post rather than daily.

So I figured I would try out just posting things in the exact order I write them. Not sure I will stick with this yet, but it‘s nice to experiment a bit.

Luckily with the new blogging system, it‘s relatively straight forward. There are middlewares to load data and organise the data, and since you can write your own middleware, you can easily organise your data into buckets however you want and they get slotted into the pages all the same.

The fast rendering blog is now live. In the end I was able to get the rendering down to around around 15 seconds for the entire 15 years worth of content. It‘s about 26000 files in total. These are the main features:

1. Renders fast (~15 secs) and the build takes up way less space (~500M).
2. Sleek new design, removed the content boxes, swapping them out for a design focussed on blog posts with supporting links and notes.
3. Added the old newsletter posts back to the mix.
4. Site header background images. The cool thing is the blog remembers the current active header image when a particular post is published, so when you look at old posts, they will look like how they were when they were first published. It‘s all controlled by your post frontmatter, and of course it‘s automatic if you use the render tool to create the content.
5. There‘s also a top right image feature, which makes it really easy to add an image that appears in the top right of your post. Again just a frontmatter field you add to your post.
6. Tags pages - this was on the old blog system, but I‘ve managed to make it much more useful. All posts have clickable tags so you can easily see other similar posts with 1 click.
7. A light weight bucket framework for creating organised HTML, where different content types just slot into place whether it‘s a single post, a flat list, or grouped by day.
8. Themes - The CSS is way way more organised. Themes are installable from an npm module, and you can easily customise things, by adding a user styles.

Since I will likely be changing the header image relatively often, I have added a headers page to keep a record of each new one as I add them. The first one is called The industrial distillation workshop, a humorous look at the new trend in AI development that people seem to be quite freaked out about.

The whole thing is built using my new rendering tool, to which I had to add quite a few things, and seems to be working very well.

Udpate: Here‘s how it looks:

This is a new header background image for the blog, the first one using the new stateful header images feature. It‘s a humorous take on the current moment in the tech industry where it seems everyone is freaking out about AI companies building their models by asking other models zillions and zillions of questions, otherwise known as distillation.

I came up with the theme and idea but it was Gemini that really brought it to life. I basically just said make a humorous image about everyone freaking out about model distillation. I think it‘s kind of spot on.

Obviously I don‘t personally endorse model distillation, to be honest I don‘t really know how to think about it, but I do know that I need to be freaked out about it!

Here's the full resolution image in case it‘s difficult to see the whole thing in the blog header:

I plan to keep a list of the various blog headers I add to the blog here.

This was the original first blog header background image I added on 2025/10/04. I mentioned it here at the time. Previous to this I was running a homepage that used this as the background image. It wasn‘t a blog. It just listed my contact details and linked to the blog which was on a subdomain.

I called it floating above the mountains because when the site title text got superimposed on the mountains it was like I was floating above them. I love mountains. I love snowboarding. Lots of fond memories.

I can‘t quite remember where it‘s from. I think it was part of a blog starter kit I got online somewhere. It was a really long time ago.

Here it is in full:

You can see all the other header images so far on the headers page.

Update: I remembered the source. It was from HTML5 UP.

I‘ve been seeing AT protocol mentioned in several places in the past few weeks. It‘s one of the bunch of social media protocols that emerged a few years ago. I wrote about it at the time, as I read through most of the big hitters. I remember quite liking AT proto, it felt very well put together, well thought out.

Dan Abramov was on Shop Talk Show EP#724 talking about it. He does a really good job of painting the AT Proto picture. I like that it‘s all ultimately just files, that you can look underneath and see what is going on. I get the impression that it wouldn‘t be too difficult to debug AT Proto app code, and I thought the notion that you get your own personal Dropbox folder in the cloud that any app can read and write to, was very compelling. Worth a listen.

A day in the life of a software developer:

• A: We have problem!
• B: OK lets fix the problem.
• A: The problem is fixed!
• B: Ok awesome full steam ahead!
• A: The fix has highlighted a deeper problem.
• B: Oh shit we have to pause the steam ship! Save everything! Let’s fix the deeper problem.
• A: The deeper problem is fixed! We are fucking exhausted!
• B: I think we can see light at the end of the tunnel! Resurrect the steam ship!
• A: It’s a fucking mess but we have boosters now.
• B: I am still quite confused.
• A: How many times have we been through this loop?
• B: Life is loop.
• A: We still have boosters.
• B: I fucking love boosters, just wish it wasn’t so fucking messy so we could actually use them.
• A: We are cleaning up the mess!
• A: You are never going to guess what just happened while we were cleaning!

I‘m still trying to get my head around all the things that I‘ve been through in the past few days trying to optimise the rendering of the website. The difficulty I had writing this blog post is an indication of that for sure. But I do know that I don‘t often get an opportunity like this to use the HTML details tag, so...

Remember, when I started optimising it was taking 45 minutes to render the last 10 years of content.

How long does it now take to render the entire 10 years of the blog?

And also remember it was taking up 5GB of space.

How much space does the render now take up?

It‘s still kind of a mess. There are some small things that need fixing. I have to figure out how to resurrect the steam ship.

Porting the new rendering pipeline to the live blog isn‘t completely straight forward, but luckily the new themes feature will make that a lot easier. The old theme was a hardcoded internal theme, but as part of this whole endeavour I turned the theme system into a full blown real feature. You can now install themes as npm packages, and the new optimised website rendering configuration is the first real external theme.

Definite progress, even if it‘s slow going at times.

When I got the recent website redesign working and was able to finally get it deployed live, I had been working on the serverside of my new publishing tool for a while, and the static site renderer had been ticking along very nicely without any real issues. However when I switched back to the rendering and finally tried to build my whole website, with over 10 years worth of content, I had a bit of an unpleasant surprise.

The old system was consistently taking about 4-5 minutes to build and deploy, but the new system when I first ran it, took 45 minutes just to do the render. And in fact things would get even worse before they got better. I later discovered that the Github Actions runner had started failing with out of space issues. When I looked into it and ran the build locally it was taking up about 15GB of space. I undid the change I had just made and got it back down to 5GB, and the deploy started completing again, but it was clear I had both render time issues and space utilisation issues.

As you can imagine my heart sank considerably. Had I missed something important in my architecture? Did I overlook something fundamental that invalidated my whole approach? Not a great situation. I worked on optimising the rendering for several days. Got the times down to 30 minutes, then 15 minutes, and then 6-7 minutes. At which point I knew that things were back in the right general region, but I could tell there were some big improvements that could be made.

Well I‘ve spent another few days on it, and I‘ve made some really huge speed improvements and with it stability and much better HTML and CSS structure. Feeling pretty good about it.

I've been refactoring the RSS feeds on the blog the past few days. I now generate the feeds without any feed libraries. It's just plain old templates. Total freedom. No more fighting against other people's vision of reality. I needed to add some custom namespaces, and just updated the template and my static site generator handles all the data preparation. Easy.

Over the years I've gotten into some pickles using various RSS libraries. But you don't need them!

I've added the <source:markdown> and <source:linkFull> namespaces. Here's some docs on the source namespace.

I've also added the podcasts back to the blog, and with a little help from an LLM it was super eady to whip up an RSS 2.0 feed with all the things you need to have it work in itunes. With LLMs and templates you can do most things :)

Sometimes when you get into a tangle a tried and tested way to proceed is to step back, list the tangles as best you can, step back again, look at your tangle description as if you were looking at an expensive piece of art in an art gallery.

Take a break.

Take another peek, don't get scared or overwhelmed.

Think about how you could isolate the different tangles, which no doubt all affect each other. They often do. Then create a minimal example. After which you will be several steps closer to your goal.

You might need to create several of these.

If you have been visiting the blog over the past few days you might have noticed that it has changed somewhat. I deployed the latest version a few days ago, and have been doing various tweeks and fixes. I think things are looking pretty good. The design is a lot more sleek, still minimalist and optimised for reading text. On small screens, the header is smaller to give you maximum screen real estate. It‘s mostly black text on white background, and the titles are bold, there are some black borders around content to keep he focus on the posts. It‘s crisp and clear, and since it‘s a static site, all pre-rendered, it should load really fast.

The other big thing is navigation. In the right sidebar, you will see a list of all the years of content, and you can navigate directly in there to any date and see posts. And in the header you will see a "Filter" dropdown that gives you the possibility to select to just view a specific content type, i.e. notes, posts or links, and it should switch to that content type for the specific date period you are currently looking at. I was able to do away with lots of clutter that was previously in the right sidebar, and now the most important links are in the header: About, Portfolio, Contacts and of course Feeds.

Another small tweek, the year in the footer copyright text at the bottom of the screen is automatically rendered based on the current date, so no more forgetting to manually update it and suddenly it‘s out of date. One less thing to worry about.

The thing that‘s not apparent just from looking at it is that the whole thing is rendered using my new static site generator. That‘s a massive boost in terms of robustness. The code is all in one repo and way way better written, way easier to reason about. It should be much easier to add features, without worrying about breaking things. As much as I loved the old system, and it got me very far, it had some rather rough edges. The new system is inspired by what I learnt the first time around, but it‘s a complete rewrite, with loads of new features, and a much more solid rendering pipeline. And is highly programmable.

There are still some things I need to implement, like for instance tags. You can see the tags on blog posts, but they don't link to pages that contain all posts with those tags just yet. More to write on this in the near future, but for now just happy the migration went quite smoothly, even if it was a lot of work.

This is how it looks:

I‘m putting the finishing touches to the re-architecting of the blog. It was all looking great but the realities of managing a stock implementation alongside the customized version for the site were clearly going to be an issue. So I implemented a light weight theming feature.

It‘s just internal for the minute, only one theme which is he blog, but can theoretically be expanded into a full feature later. Sometimes you have to do just enough to get going without boiling the ocean.

Got it working late last night, going to try and see if I can get a live version deployed today.

It‘s been really great to watch on YouTube people from all over the world discover the USA in their travels for the World Cup over the past few weeks. Pretty much everyone is totally in love with what they are finding.

I‘ve travelled quite a bit in the states, having done an east coast to west coast road trip several years ago, so I experienced first hand much of what folks are seeing. The USA really is an amazing place, and the people are super cool too. It‘s very different to what much of the main stream media would have you believe.

And so what a fantastically beautiful thing that it‘s the 250th birthday of the formation of the USA today.

Happy birthday and best wishes to all for the next 250 :)

It‘s really quite mind boggling how many small things there are to fix when you are deploying something for the first time. It feels like it will never end. So many niggly little fixes and things you didn‘t anticipate. It‘s very exhausting.

The good news is that as of a few minutes ago I successfully deployed the new site, fully built using my new rendering tool. I still have a few things to migrate over and I have to rebuild the data to make sure I have the latest, so it‘s not quite ready to show, but the crucial thing is it worked.