markjgsmith

2024/05/29 #

• The tsunami of hate and harrassment continued all day yesterday and through much of the night via motorbike gang stalkers. Food, drinking water, washing water and electricity still completely cut off. This morning several low key driveby harrassments including 'your wah'. Pretty obvious water goading. Are you going to allow me to live today world? #

• cat << EOF > My static site generator series

  This short blog series is a first attempt at describing my latest development project. I've been working on the static site generator for a couple of years on and off, as life permitted, with very limited resources. It was born out of the ashes of my linkblogging SaaS project. Initially just to build my own personal website, but it's become much more of a generalised website builder over time.

  The personal website, by the way, is very minimalist. That's partly a personal preference, I feel like information based websites should focus on text. I also wanted to have a site that would render with or without javascript enabled in the browser, readable on desktop and mobile. The plan is to enhance this over time using all the glorious modern CSS features that exist. For the moment though it's very minimalist. The way to think about it is that underneath the minimalist outward appearance, there's a lot of cool things happening to generate all the pages. Making the website look cool will be possible relatively easily using things like over-riddable server side components. Things take time when you are building the tool you are using, you have to build the foundations first.

  In any case, before embarking on the static site generator project I did of course survey available tools. I had experience building my personal blog with Jekyll, but since I'm a Nodejs developer, I really wanted something written entirely in javascript. When I looked at existing javascript generators, though there were some great options available, I found that building the archives for my existing site to be overly complex. When I openned the hood to delve into the source code, things felt again very complex and somewhat convoluted.

  These were great tools, but I got the sense that whatever hard won battles they had been through had resulted in code that, at least for me, was very difficult to fully comprehend. I wanted to be using a tool that I fully understood, both as a user, but also as a developer. I wanted the ability to modify the tool should that be necessary. I felt the best way to do this was to build a static site generator from scratch, optimised for building my personal website, but general enough that it could be used for other websites too.

  That's basically the backstory, but I've also recently wrote a longer term vision which you might find interesting. In it I go into the some of the values I've developped over the years when it comes to developing software, and the general direction I see things going for this project into the future.

  Like I said in the opening paragraph, this is just a first attempt at describing the tool's high level functionality. It's not meant as documentation or as a user manual. It's a bit technical but it's more aimed at generating some interest in the project. The project is not currently released as open source, but that's absolutely part of the longer term plan. Sometimes life throws you a lot of curve balls, and that's certainly been the case for me the past few years. Also, knuckleballs. This is the best I can do at the minute with the severely restricted resources that I have available to me.

  So onto the static site generator features overview, let's start with some of the smaller but very useful stuff, these include clear readable code, rendering in batches, easily follow the flow of data through the render process, easily debug the render process, Github build/deploy/backup workflows, highly customisable render process, page layouts and configuration management.

  Now some of the more meaty stuff. Here's a non exhausted list of some of the more major features and concepts:

  • Renderers, middleware & renderer pipelines
  • Server side components
  • Incremental progressive rendering
  • Plugins and packaging
  • Data loading utilities

  I hope you enjoyed this overview. It's still early days for this project, but if this at all peaked your interest then get in touch with me.

  EOF

• cat << EOF > Renderers, middleware & renderer pipelines

  This post is part of the My static generator series. Follow the link to read the overview and get a complete list of posts.

  I used to work in feature film vfx. Specifically on the tech side of things, as System's Engineer building and maintaining the in-house infrastructure used by the various artists to create all manor of digital effects. Rendering is a core concept in that donain, much of which is done on large render farms of computers. It's the process of assembling, generating and compositing together many often very large image files. I wanted something similar but for websites.

  With that in mind, one of the core concepts of the static site generator is the idea of a renderer. They are pipelines of middlewares which take a source file, usually a template, as input and output a rendered file. Renderers enable you to define how the generator should render the source you provide to it. The static file generator can thus support a variety of templating languages. Out of the box, EJS, markdown and plain HTML files are supported. But you should be able to create custom middleware for most text templating languages that exist, which you can then use to create a renderer.

  The middleware that make up a renderer are chained one after the other so you can for instance pipe the output of one middleware into the input of another middleware. For instance out of the box the static site generator can render markdown files which contain EJS syntax. First the EJS in the source file is rendered by the EJS middleware, then the entire output is fed into the markdown middleware, the output if which is written to disk.

  Middlewares are quite generalised. Out of the box there are middlewares for rendering EJS/markdown templates, HTML files, plus ones for prettifying text and writing files to disk. It should be possible to create middlewares for many different types of tasks.

  The renderers are executed at runtime. Ultimately they output the rendered files which are written to disk.

  EOF

• cat << EOF > Server-side components

  This post is part of the My static generator series. Follow the link to read the overview and get a complete list of posts.

  Server-side components are a primitive that you can use directly in your render templates. They contain their own template but crucially they also have functionality associated with them in the form of functions. The component's template can use the functions to perform various transformations on the text of your templates and data passed to them at rendertime.

  The ability to keep functionality and form together like this is very beneficial when it comes to conceptualising the various pieces of the page you are trying to render. You can be sure that at rendertime the template will have available to it both the data and functionality it needs to render the templates.

  It also makes it possible to easily reuse pieces of pages in multiple places. What's more server-side components can be packaged up into plugins, which makes it easy to share things you build with others.

  The core of server-side components is all just regular javascript, no complicated libraries. However you can if course use libraries inside your components.

  EOF

• cat << EOF > Incremental progressive rendering

  Incremental progressive rendering is a feature I've been working on for the static site generator. It's a mode that you can enable which should enable you to drastically reduce build times if you have a content cache.

  It adds a check of each template's output locations, and only renders the output file if no output file currently exists or if the output file present has an older last modified time than the source template.

  When you have a content cache, and enable incremental progressive rendering, then the generator will only render out new content templates that have been added since your last build. A content cache is essentially the base output directory. By default this is the dist directory.

  If you are running locally, you don't have to do anything, just re-running the generator without deleting the dist directory. In automated build systems like Github Actions it's often possible to setup a cache. This will keep a backup of the dist dir and restore it before each build.

  If your site has many thousands of files, then it will speed up the build time since it skips all renders except for the latest templates added.

  At the time of writing this post the feature is work in progress and not merged into main yet.

  EOF

• cat << EOF > Plugins and packaging

  This post is part of the My static generator series. Follow the link to read the overview and get a complete list of posts.

  One cool thing about the static site generator is that it's very easy to bundle up various primitives you are using to build your site into a plugin. This would typically be a seperate repo. That repo can then be npm installed prior to running the build. Those plugins can then be added and used by the static site generator.

  Installing these plugins gives you the ability to quickly add prebuilt sets of pages and functionslity to your site. For example there are currently plugins written for blog, podcast, newsletter, linkblog and notes pages. They can be configured and customised via environment variables and component over-rides.

  There's a rudimentary over-ride system, so the static site generator's main primitives, like for instance server-side components, can be overridden by your own custom components. That means you can easily customise pieces of the blog generated by the blog plugin to look the way you want.

  Also some plugins are used by other plugins, so common functionality can be extracted and reused across plugins.

  I call the over-ride system rudamentary because there's room to improve this feature, but for most basic websites it actually works rather well, and it's simple to understand.

  EOF

• cat << EOF > Data loading utilities

  This post is part of the My static generator series. Follow the link to read the overview and get a complete list of posts.

  The static site generator had been built initially to generate content websites' like blogs, linkblogs, newsletters, podcasts and notes. The posts of such websites tend to be written in markdown files. Though you can obviously store these in many different ways, one tried and tested way is to use a calendar folder structure.

  An example of a blog post content markdown template showing the calendar folder structure:

  blog/2024/05/29/some-interesting-article.md

  Loading data from markdown files that are stored in such a folder structure is so common that I added utilities to the static site generator to load these files into memory. They are then made available to the templates being rendered at rendertime, sorted in various useful ways.

  This makes it very easy to iterate over, for example, all the blog posts of a specific year and create an index page.

  Frontmatter in markdown files is fully supported, and it's a great way to pass custom data into the render process.

  EOF

Today’s links:

• New Post: My static site generator series markjgsmith.com #

• New Post: Renderers, middleware & renderer pipelines markjgsmith.com #

• New Post: Server-side components markjgsmith.com #

• New Post: Incremental progressive rendering markjgsmith.com #

• New Post: Plugins and packaging markjgsmith.com #

• New Post: Data loading utilities markjgsmith.com #

• The reasoning behind publishers' desire to federate their sites is clear - The author talks to the Verge's Nilay Patel and 404Media's Jason Koebler to find out what federation has to offer smaller independant publishers, and what their plans are for this Fediverse centric future list23.com #

• Inside Bluesky’s Engineering Culture - Pretty interesting piece, at least the first half, that looks at what it’s like working at Bluesky as part of the engineering team. Also great description of the tech stack. Annoyingly the second half of the article is behind a paywall. newsletter.pragmaticengineer.com #

• Essays on programming I think about a lot - Great selection of programming essays that focus on specific aspects of the craft. For each item the author has extracted a poiniant quote from the essay, and written a short summary of the key ideas. I hope to find time to read some of these :) www.benkuhn.net #