2024 #

• It can take a long time

  2024-04-22 15:01:00 +07:00 by Mark Smith

  I wrote a couple of days ago about the blogging virtuous circle. The idea is to have a repeatable set of steps that on average result in you writting some good quality blog posts every day. I'm still not really there yet, but I'm getting better at it.

  When I started out blogging back around 2005, I wrote enthusiastically for several months, but it fizzled out eventually. I knew I needed a better routine. Twitter came along, and that was great, especially in the early days, but I had a nagging feeling that I really should be posting on my own site. There were a handful of people online that were running linkblogs. Basically self hosted sites where they posted interesting links they found on the web. I thought this was a great idea almost as soon as I saw it. I could see that this could be a fun, more frictionless way to publish that could increase overall blog output.

  But there was no software available to do what I wanted. Since I also wanted to learn web development, I decided to take it on as a project and build some linkblogging software. I successfully built a linkblogging SaaS, but I ran out of runway, and ultimately had to shutdown the project.

  All the while though I had been posting to the linkblog. There were many versions of the software and the archives in a way are a record of my web development journey. Though the SaaS project is no more, my personal linkblog lives on. It's been over 10 years that I've been posting links!

  These days I run a newsletter, and I've made some exploratory and quite experimental podcast episodes. I feel like I'm closer to getting the balance right. The point I'm making is that if you are on the fence about whether to start blogging, the beet time to start is today. Start out by just posting links to interesting articles. Your voice matters. Over time you'll be able to build that into a blog, newsletter, podcast, whatever. The important thing is to start.

  These days there are much better tools out there to build your own linkblog, and also many online linkblogging services. It's the easy way to get started with blogging. What have you got to lose?

• Investigating long Netlify deploy times

  2024-04-22 05:58:00 +07:00 by Mark Smith

  I've been looking into why the Netlify deploy times have increased by several minutes. It's a real mystery because I'm only adding at most a few new markdown files to the build each day, yet somehow the netlify CLI is reporting that ~3200 files are different and need re-uploading. That clearly takes a long time.

  The only thing I can think that might cause it is creating the post archives. Perhaps I'm creating some of the pages from a javascript Object and the order of items is changing each time. That's a possibility. There's nothing on actual pages that should be changing each time.

  Anyway it's a bit tricky to test because I have to do 2 builds one after the other then compare them. Comparing using diff is easy, but comparing visually will be difficult because the staging environment only has one deploy target, but I'll need to deploy 2 builds.

  Hopefully I'll figure it out.

• The blogging virtuous circle

  2024-04-21 17:43:00 +07:00 by Mark Smith

  About the most difficult thing in blogging is to do it regularly. Ideally you really want to be writing posts every day, but it's difficult to always be inspired, to always have something interesting to talk about.

  I've been writing on the web since 2005, with a few failed blogs and later social medias like Twitter. I've been striving to find a blogging virtuous circle, which I think is likely possible with enough practice, and with the right tools.

  A blogging virtuous circle is a repeatable way to write on the web that is sort of like a perpetual energy machine. The end result is a consistent output of, hopefully well written and interesting blog posts. I'm not claiming I have found the way yet but here's what I'm aiming for.

  1. Post links to a linkblog
  2. Write short notes posts
  3. Write longer blog posts

  The web is all about discovery and sharing. Reading stimulating content online will result in you thinking about new topics and discovering new things.

  The start of the virtuous circle is super easy and frictionless. When you find something interesting online, share it on your linkblog, along with a short description of what it is. Then move on with your day's activities. I like to do this early in the morning. It's an easy way to start off the day. Get some quick wins, with a few short links. If nothing else, you'll know that you have contributed to the global commons and made the web a little bit better.

  As you go about your day you are likely to spend some time thinking about what you've read and posted, and you'll have some relevant ideas, or perhaps just a realisation of a specific part of something you posted that you find really interesting. This is a good time to write a few notes, with your ideas, or perhaps extracting a short quote that you think is particularly poigniant. The aim here is to not spend too much time on it. Once again, keep on with the rest of your day.

  Then at some point hopefully enough ideas will have coallesced that you will be ready to write a full on blog post. You'll have links you can add to it, quotes, and various ideas you can expand on. But if you don't, no worries, once again, hopefully your notes will have made the web a little but better. Maybe you will have inspired someone else to write something great. Sometimes that person is future you. It's amazing how often I find myself resding back through my old notes.

  That's the blogging virtuous circle. It's pretty simple really, but it's actually quite difficult to get the balance right. You can also expand it to include a newsletter and a podcast, but start with the writing side of things.

  One thing to remember is that blogging isn't like writing a novel or a university disertation or even an essay. It doesn't have to be perfect. There's always the next day. Anything you write will be replaced within a few days so it doesn't really matter all that much if it isn't the best prose you've ever writen. The funny thing is though, the more you write, the better your writing will get.

• Cory Doctorow on the origins of capitalism

  2024-04-20 15:03:00 +07:00 by Mark Smith

  Cory Doctorow in his piece about why capitalists hate capitalism:

  For capitalism’s philosophers, the rent / profit distinction was key. Rents bread complacency and stagnation. The feudal Lords got the same rent no matter what. There was no incentive to re-invest those rents in better agricultural tools, or advanced training for surfs. If your surfs invented a better scythe that let them bring in the harvest in 1/2 the time. You, their Lord, got no benefit from it. What’s more, the lord on the next estate over faced no threat from the competitive edge your surf’s bold innovation conferred.

  Maybe the lords gained no advantage from the better scythe, but surely that's to the surf's advantage. They could make a killing selling their new invention to other surfs in nearby estates. Soon enough and all the surfs have halved the amount of time they need to spend working the land.

  He continues:

  But profit was always subject to competition. For capitalism’s theoreticians, competition undergirded capitalism’s virtues. The fear of a rival taking your business with a product that’s better and / or cheaper sets the capitalist on a continuous hunt for efficiencies and innovations that deliver better products at lower prices. The fear of a rival luring away your best workers who are not bound to you, the way that surfs were bound to their lord’s land, forces you to find ways of keeping your staff happy and thus loyal.

  So the capitalists essentially freed the lords from their surfs, in exchange for becoming the managers / baby sitters of said surfs, but with the understanding they could exploit them for their labour. And presumably the lords were investing in the capitslist, sharing in the profits.

  The OG lord is essentially just creating a surf out of the capitalist, who does the work of wrangling the surfs. Perhaps I'm missing something here, but isn't this just lord surf fractal?

• Raoul Pal on the baby boomer savings complex

  2024-04-20 18:09:00 +07:00 by Mark Smith

  Raoul Pal in a recent episode of his podcast [22:13]:

  You see everything did break back in 2008. Peak population growth created these problems going forwards and we are seeing the central banks doing one thing and one thing only. In fact, I think the governments are doing one thing and one thing only. […] Their entire job is to keep afloat the great baby boomer investment complex.

  You see most of the savings in the US and across the world are in older populations. Those were the richest populations the world had ever seen. Those have all of the wealth. In the US particularly they own equities, in Europe they own bonds. In Japan they own bonds. So what you can’t allow is the equity market to collapse. […] Of course the governments and the central banks understood this. They understood all of this. They have been trying to debase currency to optically keep the price of those assets high so the baby boomer savings complex continues.

  Two things jumped out at me from his thesis. Firstly, that much of our current economies dynamics are a direct result of demographics. The second thing is that, unlike many from crypto and Bitcoin circles, he believes the central banks aren't incompetent, they know exactly what they are doing.

  It's worth listening to the rest of the episode, he makes a lot of interesting observations.

• Promoting your own stuff is okay

  2024-04-18 16:57:00 +07:00 by Mark Smith

  Wiser and much more experienced bloggers than me have said this many times before. In a way, it's so obvious but it's worth repeating just to remind yourself. One of the reasons I'm writing this here post. To be very honest, I'm still not very good at it, even after all these years. The truth is it's difficult to get the balance right.

  Your blog is your own personal advertising platform, for yourself.

  That's one of the biggest reasons for not running ads on your blog. If you run ads you are competing against yourself. Every situation is different of course, but I'd say that for most people, most run of the mill normal blogs, running ads just isn't worth it.

  Much better is to try and gather a loyal audience that like to read your content, so that when you do other projects they will likely be more than happy to help promote it. If your content is good enough this will be effortless, because they'll want to talk about it anyway.

  There are lots of angles to take. Talk about your past experiences, your current projects, topics areas that you want to highlight in your CV, hobbies you are getting into, problems you are debugging and solving. When someone visits your site, they'll be able quite quickly to get a sense for who you are.

  Dave Rupert from Shop Talk Show Ep#611:

  People don’t know. […] Everyone’s too busy to care. You have to constantly be sharing the cool things about your thing. Your blog, your framework, your whatever. Keep advocating for the things you like […] If you like something you should talk about it. Then other people will know about it. And then it will get picked up.

  The tricky part is to not make it sound like you are lecturing. I think in the world of brand marketing it's called something like 'native advertising'. Instead of specific spots where ads are inserted, the content actually is the ad. One way to do this is to talk about current events, but use it as an opportunity to remember relevant pieces of your past.

  We are all learning after all, most of us actually like to hear relevant points of view with interesting stories.

  If you are just starting out then focus more on linking to things you find interesting and why. Your point of view is valuable too. Over time you'll build up a collection of stories and you'll find a way to integrate these into your content.

  I wouldn't say I'm particulary good at doing this. I definitely haven't found the right balance yet. But as you read and listen to others you'll realise those that succeed aren't afraid of talking about what they do. But they are much better at doing it than others. They have often thought about it a lot and distilled out the most valuable pieces of information.

  Figuring out where your value is, amoung all your knowledge, is key. That, I think is how you go about building an audience for the long term.

• Typos and missing urls

  2024-04-15 10:47:00 +07:00 by Mark Smith

  I make quite a bit of spelling mistakes in my posts. It's annoying. I know. It also happens quite often that I forget to add a url to a link. I wish it wasn't the case. There's no spelling highlighter in my text editor.

  I write mostly offline, then sync it up when I am online. That means I often don't have the url available to me at the time of writing. I try to add thrm before I sync up, but sometimes I forget, sometimes I'm in a rush. But I do eventually get them updated.

  All this to say that if you find a link that's missing a url then check back later. And please ignore the typos. Both the links and typos will often get fixed in the next daily build.

• My current linkblog format

  2024-04-15 06:30:00 +07:00 by Mark Smith

  In case you were wondering why the links on the linkblog have the links repeated, once in the title and once in the domain link. Figured I'd write a quick blog post.

  For some examples see todays links here.

  Originally I used to have just the domain link. That was for the first 10 years. I would try to create a custom bit of text for each link, but I found that the article title was often better. It started to feel like a bit tedious to always find something original. Then a few years ago, about the time I moved off of Heroku, as part of the redesign I started using the new format, very much inspired by several javascript newsletters I read regularly.

  All the newsletters tend to use this format. I find that it's quite effective. There's something that's very appealing about getting a peak at the page on the other side of the link, but also followed up by descriptive text written by the newsletter author. As well as my own personal preferences, I figure they have likely done market research, they are link sharing experts after all.

  I already have the domain links operational so figured, I'd keep them. They get added automatically. Occasionally this comes in handy when I want to break from the format for whatever reason.

  One of the cool thing with the latest version of my static site generator is that it supports render pipelines, so I can put EJS syntax directly in the link's markdown file. That way I can easily access the file's frontmatter from inside the markdown file body text, and that's where the url is stored, so there isn't any duplication.

  That's the reasoning behind the current linkblog link format.

• Should we re-imagine git interfaces?

  2024-04-14 09:56:00 +07:00 by Mark Smith

  The Changelog guys have a great interview episode out featuring Scott Chacon, co-founder of Github. They discuss at length both new git interfaces and the realities of open source licensing. Definitely worth a listen.

  My facination with git has only grown the more I've used it. From the early days as a better svn, to the Github revolution, and into building github based workflows. I think there's a big opportunity for git based content workflows but realistically the git UI is currently only for programmers.

  Scott's re-imagining of his coding workflows is super interesting. He speaks of completely different ways of thinking about version control, more similar to how Google docs works, with auto-save, the ability to share work in progress via a url before having committed it, and the ability to allocate code from a working space into different branches in parallel. I really like his approach, especially that he's not trying to replace git completely. It's totally okay to use various git interfaces for different uses. That in itself is a pretty powerful idea. Use the UI you want, you can always switch over to the CLI if you need to.

  But if we can re-imagine git for programmers, why couldn't we do it for other types of people, for example writers, producers, editors, engineers. So many roles have a different equally valid ways of looking at the world. It might be worth spending time creating new interfaces on top of old software. Many specialised interfaces ontop of git, not as competing software, but as complimentary, might be an interesting path to explore. With a shared git based backend, really cool custom workflows could be assembled for virtually any setting. It might enable people with a diverse set of skills and experience to collaborate together.

• Rogan‘s vision of our AI governed future

  2024-04-13 12:01:00 +07:00 by Mark Smith

  From a recent Rogan podcast episode with Brendan O'Neill #2133 @ [01:20:36]. I found it both scary, but actually quite realistic:

  [...] I’m going to fully put on my tinfoil hat, I’m going to secure it with a chin strap. If AI was sentient, and if AI would want to ensure compliance, first of all if AI was sentient I don’t think it’s under any obligation to let us know.

  Why would it? I think it would just acquire more resources and stay in the shadows and just keep functioning as an organism. If it wanted things to collapse to a point where people are incapable of sorting things out amongst themselves, they are so far gone, they are so far down the rabbit hole of ideology and of tribal conflict, that it’s impossible, it’s never going to work out, it’s going to be a civil war. Unless we let AI take over.

  And then we let AI govern things cause AI is gonna look at things logically, it’s gonna find all the problems in our societies, it’s gonna fix them, it’s gonna allocate the money fairly, it’s not going to be any corruption. It’s going to be this intelligent over see-er that just decides what everybody does, in order for the greater good of the species on Earth.

  Incidentally I've found many of Rogan's recent wanders into the world of AI very precient. He's describing it probably the closest to how things have appeared to me for quite some time. For example his description of our matrix like simulation future, on episode with David Holthouse Ep#2129, towards the end of episode, I thought that was spot on.

  It's kind of scary, what happens if the super AI doesn't like you?

• The softly softly murmurers

  2024-04-10 09:18:00 +07:00 by Mark Smith

  I've written before about the softly softly murmurers but without much description.

  They set themselves up nearby where you are sitting in a cafe. They are always women, most often small and petite in stature. Usually it's a place that is quiet. You are probably focussed and getting things done. They operate in pairs. One positions themself in such a way that their face is hidden by the other person's head. That means they can still see your body's main outline, even though you can't see their mouth or eyes. Then they have an apparrent quiet conversation, but it's not a real conversation. Everything they say is keyed off of your movements. And guess what, it's very distracting once you notice it.

  If they notice that you have noticed, you might suddenly hear a 'you learn' emerge from their speaking in tongues style quiet babbling. It's also likely some of them think they are doing you a favour. But some are clearly malicious, trying to slow you down and make it impossible to do whatever work you're doing. It often happens in retaliation for an earlier event, which itself was often manufactured by others, but blamed on you.

  Perhaps the worst thing is that it's so pernicious. You are the only person that it will affect, no one else in the room will even be aware that it's happening, so if you say something, you will seem unhinged. Occasionally it unfolds in such a way that it's completely obvious to all, but unfortunately that hapoens rarely.

  And you really can't do much about it, except continue as best you can, what you were doing.

• The improbabilty of total solar eclipses

  2024-04-10 06:34:00 +07:00 by Mark Smith

  I've listenned to several podcasts covering the recent solar eclipse in the US. One of which was Andrew Keen's interesting interview with author Christopher Cokinos. Amoung other topics they talk about, I was struck by Christipher's description of the science behind the eclipse. Of the shear impossibility of it all.

  These days we hear announcements of eclipses every year. We almost take it for granted. But that wasn't always the case. In the years before scientific understanding and modern technology, solar eclipses really freaked people out, which is very understandable. You live your entire life knowing that the sun sets in the evening and rises in the morning. Eclipses turn all that on it's head. Those in the direct path of the eclipse, literally experience for 4 minutes night time recreated in the middle of the day.

  So what's so special about it? Well first off, the moon is at a distance perfect for eclipses. It's also perfect in size. It is 400 times smaller than the sun and 400 times closer to the Earth. That means that when it crosses the face of the sun, it's diameter is just such that it can cover it completely. Because if this, total solar eclipses are possible, and have enabled us to see the Sun's corona. This is the area on the surface edges that are much hotter than the rest of the Sun. This distance and size fact alone, leaves me thinking how improbable the situation is, but that's not the only thing.

  The solar eclipses we see on Earth happen seemingly randomly because it turns out that the moon orbits the Earth in a non-standard orbit. Though the Earth and Sun move in the plain of the Galaxy the moon's orbit is in fact tilted compared to the plain of the galaxy. The moon is 5 degrees off of 'the ecliptic'. Whereas other planets that do have moons, the moons are likely in a regular orbit. They would have eclipses every single month. Granted these days eclipses are seen as much less random, we understand the science, but that's only happened relatively recently. For most of humanity's existence, eclipses basically happend randomly.

  The moon is also strangely big in comparison to the Earth, especially when considering its short distance from the earth. It took a long time for scientists to figure out how it was formed. We think the moon was formed, not by a foreign entity gravity well capture, as is the case with most other planets, but by a collision with a Mars sized planet, causing an enormous part of earth to be ejected into orbit. That eventually, after billions of years, turned into the moon.

  Basically this whole setup is very very rare in galactic terms.

  This configuration had a massive effect on the development of the planet, the tides, the seasons, and even life. Some believe it might have played a big role in creating the conditions for life itself, since the early Earth would have been impacted by it's forces greatly.

  All this got me thinking. In my opinion it might even be responsible for our fear of the unknown, the development of gods and religions etc, because you never knew when an eclipse would happen. It likely created and shaped how we perceive the external world and ultimately the universe. And since most other habitable planets probably don’t have such a moon, I suspect that, should there be life somewhere else in the universe, these civilisations would develop with a very different view of the universe and existence.

  The very particular way our solar system has evolved likely has had a big impact on our species at a core level. If there are aliens somewhere in the Universe, how we experienced the moon as we evolved as a species and developed into a civilisation will probably be responsible for many of our deep rooted differences.

  Personally I am in awe at all this, I mean seriously, what are the chances? The fact that all these giant entities arranged themselves in such a way that we get total solar eclipses, and random ones, so we never got bored with them, always curious to look for thousands and thousands of years, at what was out there beyond the skies of our planet. It's amazing.

• What do they mean when they say lie?

  2024-04-08 06:20:00 +07:00 by Mark Smith

  Yesterday I asked what people here meant when they say 'lie'. There was a massive uptick in people saying the word as they walked past me. Women, men, school children, everyone was doing it. I wasn't doing anything, just walking down the road or sitting observing the world.

  Well this morning things become more clear. A taxi driver shouts loudly at a shoe shine / shoe repair bloke as he walks past to get his attention. 'Lie!'. It's clearly a beliteling derogatory term, on par with calling someone a dog. People are referred to as lies. Think about that for a second. It's basically saying you are so low you don't exist. You are a lie, not even real.

  Relatedly often people walking past, say 'you are' which I guess is saying something like 'you exist'. Which is nice until you realise that a load of people are actually saying 'you our', as in you belong to us. As in our property. But I digress. Recently it's been a lot more of the later. It seems any term will get corrupted.

  Back to 'lie'. They also use 'lie' to tell people to chill out, to wait. All this the same across most asian countries I've been to. So it has this dual purpose which means gives those who want to lord it over others, to be total assholes, to do it in broad daylight, and no one bats an eyelid.

  I think it's worth pointing all this out because as we go forward into a world mediated by AI, we have to remember that in much of the world people treat other people very badly indeed, and there's no reason whatsoever to believe that will be any different in the near future. It's clear AI tools will be used by some humans to subjugate other humans.

  Having said all this, it's worth noting that this particular set of incidents might all have been manufactured, including the taxi bloke and shoe repair guy. I say this because during the uptick yesterday, one of the incidents that happened was a women walking a dog talking with some police. As I passed she shouted 'lie' at the dog. About 30 seconds earlier two school kids teenagers walked past me in the opposite direction that I was walking, one of whom said, you guessed it, 'lie', as they passed. This morning I shouted to the taxi bloke 'you are lying' and he nodded looking a bit embarrassed.

  So in summary, groups of people definitely create situations where they openly use extremely detogatory ways of addressing others in public, in broad daylight. And this doesn't even get registered by anybody around as not normal, or not acceptable. It's clear this sort of thing also happens for real. Why wouldn't it, the fakers are basically teaching on-lookers that it's an okay and normal way to behave.

  Anyway, enough said, this sort of stuff makes my blood boil. How are we ever going to build a better world for all if this sort of thing is happening?

• The component overrides bug

  2024-04-08 05:42:00 +07:00 by Mark Smith

  The component overrides bug that I tracked down and fixed yesterday was a real tricky one. The bug got introduced while I was digging myself out of the massive hole caused by being blocked from cleaning up the codebase. It's layers upon layers of irony at this point.

  Turns out you can't currently override core components, something that needs to change, because you should be able to override any components with your own ones. Core components, by the way, are ones included in the main static site generator app. It's quite an easy update but not a priority right now.

  Anyway I wrote some additional tests in the main static site generator code, one of which was for overriding. Problem was I had forgotten about the core components special case, and the test was doing exactly that, overriding a core component. I got the test working, but it broke the feature everywhere else. The result in my website was that none of my custom components, were being picked up. The most notable consequence being the removal of multi-paragraph notes, which is a special feature I added to my website, outside of the notes plugin.

  There are quite a few customisations like this that I've added on my website. It's easier to do it there since that's the only live site at the minute. The plan is at some point to move all the custom features that make sense to be in the plugins, to the actual plugins. But it's not a priority. I have much bigger, much more important things to get right first. The interleaved homepage is one of those.

  Strange that the world seems so intent on blocking me from finishing the interleaved homepage, while at the same time forcing me to move my custom functionality into the plugins. Why in heavens would that happen? It feels like something in the universe is conflicted.

  Life goes on. Hopefully no starvation and thirst today.

• Complex malicious help

  2024-04-06 20:45:00 +07:00 by Mark Smith

  The new trend I've noticed is that almost everytime someone offers me help there's a malicious component, or several, to the whole thing. It's been like that for several weeks now.

  For example, someone gifted me some small cakes which were very nice, but also toothpaste and a toothbrush, and an old baseball hat.

  The toothpaste was actually useful, but there were two tubes. One regular, and one child size. The toothbrush was pink. I've lost count of the number of pink related incidents. They usually result in people shouting 'gay' at you constantly. The hat was strange, but the very next day my actual current hat disapeared. I think it was stolen. That entire day was full of food vendors trying to scam or mislead me in one way or another. It was like some sort of co-ordinated tsunami. I don't know how it happens, only that it does.

  Another pattern that happens all the time is gifts of food that requires hot water, so it's essentially not edible, if you accept you end up carrying all this unuseable crap around, making your bags heavier than they are already. But if you decline the gift they act suprised like you are ungreatful. These incidents are highly highly correlated with water thirsting periods, where they mysteriously only give you milk, but no water.

  It's like complex numbers there's a nice part and a malicious part. At times it all seems malicious.

• Motorbike gang stalker drivebys

  2024-04-06 20:07:00 +07:00 by Mark Smith

  There's been a sudden uptick in motorbike gang stalker drivebys this evening. It's like a memo had been sent out and the attack bees are swarming in on me. But it's not just folks on motorbikes, it's pedestrians too. People auspiciously doing their own thing, having a conversation, or being on the phone, then suddenly a whole bunch of vileness clearly directed at me, meant to goad me into reacting.

  If you say nothing they assume whatever it is they are accusing you of is true, you're a liar, you're gay, you're an actor, whatever, then they bully you more. If you reply telling them to f-off, they also bully you more, escallating to try to make you angry.

  Either way you end up being mutilated in some way, whether it's illness, body injuries, loss of some personal property, starvation, thirsting, there are so many ways they can get you. And you can't do nothing, it's happening in broad daylight. Whatever you do it's turned against you.

  And life goes on, you die a little more, they steal a bit more of your life, and you are the one that's in the wrong. And another day starts and you try to stay positive abd ignore the constant bullying.

• Robot mates and LLM conversations

  2024-04-06 18:30:00 +07:00 by Mark Smith

  It seems Apple might be getting into the home robots business. It makes a lot of sense. Their recently failed self driving car initiative has had them in the vague ballpark of robots, so perhaps they are thinking they might be able to pivot. It certainly could be a big market. Imagine a world where most households have a house robot. They might even cost about the same as a car if they can do laborious household tasks like laundry, cleaning, and cooking.

  People are pretty intrigued and excited about the possibilities. There's an interesting segment on the latest All-in Podcast about household robots. Skip to 49:20 for the robots bit.

  No doubt Apple would make the best robots out there. Certainly if their attention to detail and quality is the same as with the Vision Pro. Home robots are notoriously difficult to get right, there are many potential dangers if they are to be around humans for long periods of time in a home setting.

  I wonder whether it would be a good idea to start small. Not literally in size but in functionality, but while riding the current LLM / AI wave.

  I was sitting in a cafe yesterday alone, and it occurred to me while looking at the empty seat in front of me, that it would be kind of cool if there was a humanoid robot that just sat at a table and had a drink with you while chatting about random shit. I reckon the current LLM models are basically good enough for such a robot mate. Instead of typing into an annoying chat prompt, you would actually talk to a robot. Wouldn't that be cool?

  That's how it would start out, but it would give the developers time to work on facial expressions and body language, and work towards more advanced functionality. Just an idea. I think you'd be able to get something functional to market very quickly and safely. Might even be a good approach for some of the startups in the space.

  Sometimes starting with something basic but fun is the best approach.

• Module caches and reusable workflows

  2024-04-05 23:16:00 +07:00 by Mark Smith

  The last few days have been a bit of a disaster development-wise. I was forced into pressing forward down a difficult path because I was blocked for many days on all other paths. It ended quite badly, and I was unable to deploy to the website and a bunch of stuff broke.

  I was able to fix it within a day, but it was kind of scary, because since I'd been forced into pushing directly to main branch on several repos, the only way out was to reset hard the branch, losing several days work. Anyway it's fixed.

  All sorts of irony surrounding the whole episode. Here from a note I wrote mid way through:

  "The very thing I was worried about happening, which was the reason I was tidying up the code, actually happened when the world blocked me from tidying up the code!"

  There were a couple of useful take aways from the exercise.

  Firstly I've found a better way to implement the npm module caches used in Github Actions that makes it easier to refresh them when code changes in the plugin modules. That's a big deal because it was a massive source of confusion and wasted build minutes. I've also removed all caches that were used in plugins. The plugins now use simplified local versions of the server components in tests rather than those in the plugins. I guess you could call them mocks. That means there's no dependencies, no need to install plugin repos, so no need for an npm cache. The only repo that still needs an npm cache is the main website because it has to npm install all the plugins.

  That should make things much less confusing. Imagine how complicated it was debugging the main website with a cache, when the bug is in a plugin that also has a cache.

  Another thing I realised earlier was that the work I did trying to tidy up the plugins by consolidating them into a couple of repos might not have been for nothing. Reason being that I got reusable workflows operational between private repos. Theoretically it means I should be able to consolidate all the plugin workflows into one workflow that they all use to build and deploy. That will be much easier to maintain.

  The final thing I realised was that it would be nice to update the cache refresh logic into a seperate reusable workflow. Currently the build deploy backup workflow is dual purpose which is confusing and error prone. Since I know reusable workflows between private repos work, that might be possible. But for it to work the caches need to be useable across workflows in the same repo. The cache refresh workflow would create the new cache, which could then be used by the build and deploy workflow. I'm not sure if that's possible yet.

  To be clear having lots of private repos is still a bit of a headache, but all in all if I manage to get all or most of these optimisations working, the simplified cache setup and consolidating the workflows might make the overhead of having seperate repos for each plugin just about manageable.

• Masks are sort of dumb

  2024-04-05 22:25:00 +07:00 by Mark Smith

  I'm finding that very humid environments make my lungs worse.

  I end up coughing quite a lot and eventually that results in coughing up flem. I'm also finding that when I wear a mask and ensure that it's a tight fit, so the air I'm breathing is definitely being filtered by the mask, that the air in the mask itself gets very humid because obviously, the air I breath out contains water vapour created in my body. I know it's humid because when I wear my glasses while the mask is on, they immediately fog up.

  The point is that when I wear the mask for a long time, I eventually start to cough because of the humidity, and eventually it results in coughing up flem.

  It's a tough choice. Filtered air with coughing and flem, or non-filtered air with coughing and flem. If I wear a looser fit mask, the air doesn't get filtered, so what's the point?

  Like I was saying before, masks are sort of dumb.

  title: 'Masks are sort of dumb' date: '2024-04-05 22:06:00 +07:00'

• Feature phone interfaces

  2024-03-28 08:01:00 +07:00 by Mark Smith

  Really interesting episode of Citadel Dispatch with KG about his custodial Bitcoin solution Machankura, deployed in a many countries across Africa. The cool thing is that their solution works with old school feature phones. And it's suprisingly advanced in terms of functionality. In a way, it's simplicity is it's strength. If users can add credit to their phone, then they send/receive/store bitcoin.

  They even have a way to pay for groceries directly in stores via lightning, and a way to buy bitcoin annonomously using cash via gift cards or directly over the counter in participating stores. It's really very impressive.

  I wonder if he could open up his solution to other developers via an HTTP API?

  I'm also wondering how easy it would be to add a feature phone interface to an existing web2/3 app. What is the min app for a feature phone interface app?

  Personally I quite like the idea of using a simple device to do bitcoin transactions because it minimises your attack surface. Of course you would also have a smart phone for surfing the web, but keeping these devices seperate might be a good idea. Also should you loose all your belongings, it's a lot cheaper to buy a feature phone than the latest iPhone with 10 cameras and AI photo editing.

• Regulators as secret crypto shills

  2024-03-27 11:37:00 +07:00 by Mark Smith

  Another half baked mid-week observation for you. Maybe it will spark some ideas even if it isn't articulted in the most elegant way.

  The way things are built in the crypto space is sort of interesting, even if it's a bit worrying, because it can lead to all sorts of scamy behaviour. Contenders for new layer 1s start out as personality or meme coins. This is to get some traction and money flowing through the network. The developers then attempt to add much needed core features like security and decentralisation amoung others. If they don't succeed they can always turn the project into a layer 2.

  The point is the process is very difficult, fraught with difficulties, but can result in robust software. That software tends to be relatively well suited to building protocols, which can be used by many.

  This is in contrast to how software is typically build for more traditional web companies. These tend to favour centralisation and don't tend to be conducive to creating open protocols.

  It occurred to me that in a way the US and EU regulators that are trying to impose laws to curb tech giants are basically imposing a specific development philosophy onto companies. Their focus is on curbing monopolistic behaviour, rather than say decentralisation, or censorship resistance.

  I wonder if it's the case that eventually they will be imposing crypto style architectures and protocols. The certainly don't seem to be down with traditional web businesses at the minute.

  How ironic that they don't allow crypto then, when in a way, these are perhaps the companies that are trying hardest to be the most fair to participants. At least that's the case for the layer 1s.

• The AI layered tech stack

  2024-03-26 06:30:00 +07:00 by Mark Smith

  David Sachs outlined how he thinks about the bigger picture of the AI tech stack on the most recent All-In Podcast (48:43). I think it's a very useful mental model so I have reproduced it here so I have something to link to from other articles I write.

  1. Chips, silicone, Nvidia
  2. Foundation models, open AI, open source models etc
  3. Infrastructure, developer tools, vector databases
  4. Applications

  Talking about layers is pretty popular at the minute.

• David Pierce summarises the complaints about Apple

  2024-03-26 06:00:00 +07:00 by Mark Smith

  David Pierce repeats the complaints from makers about Apple in a recent Vergecast episode.

  It’s is actually worse than it was before. You’ve made our lives harder in the name of pretending to make our lives easier

  He also summarizes the situation quite succinctly:

  Apple keeps claiming it’s trying to make things easier for its users, and all it does is make things worse for everybody

  Apple is now embroiled in several very high profile litigations. It's going to be interesting to see how it unfolds, the results affect us all.

• The bitcoin layered stack

  2024-03-25 14:21:00 +07:00 by Mark Smith

  On a recent What Bitcoin Did Podcast, Dhruv Bansal outlined a big picture view of the Bitcoin blockchain, how it is designed to be extended and built upon in layers. Fundamentally it's all about auctions.

  Dhruv's main idea:

  "Bitcoin is layers of markets. Each layer relies on functionality of the layer below"

  The Bitcoin stack

  • Layer 0 - Creating / releasing bitcoin, deeply connected to time and energy
    • The network is putting out a bid for computation with a set reward, the first to solve wins, the order book is trivial, single asking price, first order that comes through settles that order
  • Layer 1 - Transfer bitcoin to each other, deeply connected to blockspace and mining pools
    • Settlement of orders is chosen when an individual miner (actually now we have mining pools), the order book of layer 1 is non trivial, it consists of all pending transactions in the mempool, which is much bigger than 1 standing asking price. More complex than layer 0, there are more degrees of freedom
  • Layer 2 - Allows much larger amount of people to transact, about small payments, deeply connected to routing (onion network lives inside the lightning network)
    • Something like the Lightning network, this is now a market for fast settlement that doesn’t go directly on the blockchain, the scarecity we see in this market is liquidity or the availability of bitcoin, number of orders in order book is vastly bigger than in layer 2, it’s not just every transaction we are trying to get into a block, it’s all the lightning payments that could potentially ever be made between all the people trying to buy coffee or whatever it is we use the Lightning network for
  • Layer 3 - Something about the internet, about routing and the transmission of information digitally online

  He goes on:

  Each layer increases the reach and scope of Bitcoin, and allows it to do more interesting things. We have to create new markets that allow Bitcoin’s reach to grow.

  I thought it was a really valuable way of looking at Bitcoin architecture so I've extracted the essence of what he was saying here, so that I have something to link to from other articles I write. Feel free to do the same from your articles.

  It's still worth listening to the full episode to get additional context. They talk about layers about 30 mins into the episode.

• The bitcoin and crypto rollercoaster

  2024-03-25 10:36:00 +07:00 by Mark Smith

  I've found my learnings in the Bitcoin and crypto space a bit of a rollercoaster recently. I ended up writing a somewhat frustrated blog post and followup note, so I've been trying to piece together my recent somewhat turbulent learnings. Life is complicated and confusing sometimes, so many things happening at once. Writing about it helps figure it out, even if it's a bit inelegant at times.

  Anyhow I thought it would be easier to vaguely re-create my journey with a list of podcasts. You too can experience the rollercoaster :)

  This one will make you comfortable that Bitcoin can send/receive/store:

  • Bitcoin Basics with Odell (Citadel Dispatch Podcast)

  These two will freak you out about the wider world of crypto, and make you confused about the current state of Bitcoin development:

  • Eigenlayer Q&A with Founder Sreeram Kannan (Bankless Podcast)
  • Good Morning With Steve Lee (Citadel Dispatch Podcast)

  This one will calm you down and be at ease and comfortable that Bitcoin actually has a robust layered architecture and sensible bigger picture:

  • Bitcoin Satoshi’s Vision with Dhruv Bansal (What Bitcoin Did Podcast)

  This one will make you curious about layer 1 and layer 2 infrastructure architectures:

  • The 7 Most Bullish Crypto Narratives for 2024 (Bankless Podcast)

  This one will make you much more comfortable about layer 2s in a Bitcoin context, and excited about the possibilities of programmable money which will likely compliment and enhance send/receive/store.

  • The Challenge of Scaling Bitcoin with David Seroy (What Bitcoin Did Podcast)

  Update: I just listened to the David Seroy Bankless episode which is a perfect compliment to his WBD episode. The future of Bitcoin with bitvm and zk rollups sounds very cool indeed.

  • Are Bitcoin L2s Real? with David Seroy (Bankless Podcast)

• Trump has become a standup comic

  2024-03-17 11:03:00 +07:00 by Mark Smith

  Rogan made this observation on one of his latest episodes. It was either the Black Keys or James Lindsay episodes, I can't quite remember which. He's had nearly a decade now of going around doing speeches and TV interviews constantly. And he clearly loves it, he's totally in his element, and because of his stream of consciousness style of speaking, he's basically at this stage a standup comic.

  I feel validated because I've been thinking the exact same thing for months. I make no commentary on whether this is good or bad for the United States, but to be honest, often he's pretty fucking hilarious.

• What will the Space Age be like?

  2024-03-17 10:47:00 +07:00 by Mark Smith

  I've become totally fascinated with the gilded age after listening to the latest Rest is History podcast series all about the Titanic. Building huge luxury ships that would sail across the Atlantic ocean was what entrepreuneurs did back then. Of course there were also huge industries in building massive road and rail networks, and many other large scale physical goods productions. I wonder if we will see a return to a similar age with the space race unfolding?

  Elon just managed to put the largest craft ever into orbit. Such a ship could transport many many people. Exciting times.

  I'm going to work on my facial hair just in case we do have to go all Edwardian Steam Punk in the next few years.

• Crypto WTF

  2024-03-17 10:20:00 +07:00 by Mark Smith

  So I've been listening to loads of podcasts mentioning Tether and Tron recently. They say these are super popular in places where people just want to do smaller transactions. Very popular in developing countries because fees are low. It's aimed at the unbanked, people that have never had bank accounts.

  But what's going on from a technical perspective? First of all it's totally unclear to me how these unbanked people use it because surely at some stage they need to turn these cryptos into a fiat currency in order to actually buy something. Also how do they get crypto in the first place? Don't you need a bank account to make use of crypto eventually?

  Are these all just literal seperate apps for your phone? Which are the blockchain, which are the coins? How do you exchange them? Or are you just stuck in the coin that you choose, along with all the other confused souls that somehow ended up with X coin on Y chain with Z wallet...

  Here's a tiny fraction of what I was presented with:

  • Tron
  • Tether
  • Etherium
  • Bitcoin
  • Aqua
  • Fedimint
  • Lghtning
  • Liquid
  • Optimism

  But wait there's more. There are combinations too. Tether Tron, Liquid Tether, and so many other combinations. You quickly start to wonder whether even those you are listening to talking about this stuff know what they are talking about. It's enfuriating.

  It's so unbelievably complicated. Once you do some research you will quickly be lost in a sea of technologies, protocols, stable coins, chains, side chains, rollups etc. How can sending, receiving and storing money be so fucking complicated? It's literally 3 fucking features y'all. WTF crypto?

• Low level git

  2024-03-17 09:54:00 +07:00 by Mark Smith

  Software is strange. You build it using your current knowledge, but things change and you realise that your software would have been better, or easier to understand had it been built another way. So people retrofit new interfaces to the old software, to modernise it. And it sort of works for many of the use cases.

  But if you try to do something complicated, or you run into trouble, accidentally going down a path you didn't mean to, the new interface often gets in the way. Git, amazing as it is, sufers from this problem.

  I've been reading Git from the bottom up recentally. It's been very enlightening. Git is so much simpler underneath than I thought. I'm finding it difficult though to hold both interfaces in my head.

  I keep wondering whether it would have been better to learn git using the low level primitives right from the start.

  Computer software doesn't always intersect well with human cultures.

• LLMs in the browser

  2024-03-17 09:42:00 +07:00 by Mark Smith

  I've been thinking about browser diversity the past few days. Looks like the EU is going to force Apple to allow alternative browser rendering engines on iOS. I think this is likely great for users.

  Anyway browsers are made up of various pieces, including the rendering engine. Firefox, Chrome and Safari have different variants. Meta has open sourced it's LLM Llama, Elon is apparently also open sourcing his Grok LLM. I wonder how long it will take until someone tries to integrate one of these directly into a web browser. Could be interesting to surf the web with a browser super powered by AI.

• Consent forms suck

  2024-03-17 09:34:00 +07:00 by Mark Smith

  I've written about this many times before. The web is now a wasteland almost totally covered in consent forms. It's a disgrace. Users don't want to wade through this crap, and who can blame them.

  Consent forms are handing the entire web, the thing many of us have spent decades contributing to, directly to AI companies on a silver platter.

• Belgium and communes

  2024-01-21 15:56:00 +0700 by Mark Smith

  I published a link earlier about Brexit and free movement for UK citizens, so I've got Europe on the brain. I spent a lot of time growing up in Belgium. For the most part it was fine, good in a lot of ways. Not the best, not the worst, I liked that about the place. It was cool.

  Belgium is made up of provinces, and the provinces are made up of communes. Communes are like councils in the UK. The smallest administrative unit of the state. Communes are geographic, as in the extend over a certain area, but each one also has an administrative office. Every once in a while you'd have to go there to do some adminstrative stuff. I went to the commune probably a handful of times during the entire time I was living in Belgium.

  I didn't end up learning that much about Belgium history because I changed from a french speaking primary school to an english secondary school. There was zero history in primary school as far as I remember. What I learnt was just from actually being there, every day life, and there were a few meuseums with stuff about battles with blokes with hats on horses hundreds and hundreds of yesrs in the past. A tremendous bore in my opinion. I did learn how to speak french rather well so that was good. I can pick a good belgian beer. Not so useful these days now that I no longer drink alcohol.

  But anyway, it occurred to me a few months ago isn't it odd that their smallest unit is the commune? What possible economic system would result in such a structure? I mean maybe it's a total co-incidence, but based just on the words alone, wouldn't you expect commune based societies to be created by communism?

  I'm amazed that I never saw this connection before, but on the other hand I was a science and maths major. I was never interested in politics, economics, history or all that stuff. That's been a more recent thing. I guess it's just some sort of weird coincidence because the entire time I was there, no one mentioned it. Not in the schools, not in the bars, not in the sports camps, or churches. I lived there for 10 years. So others musn't have been thinking about it either. Communes were just communes, you grow up with that around you, end of story.

  Next time I'm online, I'll need to look this up. It's super weird. I don't think they use communes in France by the way.

  Update: Couldn't find anything via the regular web, so tried Google Bard. Turns out the term predates the creation of the state of Belgium. Usage started in 17th century, standardized across the region after the french revolution in 1789. Belgium was created in 1830.

  Btw, this is the first time I've used a ChatGPT. I hope it's accurate. Sounds like it is. :)

  Bard doesn't work with the internet archive so I've copied the full text of the answer:

  Question: When did Belgium start using the term commune

  The term "commune" has been used to refer to local administrative units in Belgium since the 17th century. However, it was not until the French Revolution of 1789 that the term was officially adopted and standardized across the country. In 1795, the French revolutionary government created the communes as the basic unit of local government in Belgium. This was part of a broader effort to centralize power and eliminate the old feudal system.

  The communes were initially very small and fragmented, with many of them having only a few hundred inhabitants. However, over time they gradually merged and grew larger. Today, there are 581 communes in Belgium, ranging in size from the city of Antwerp with over 500,000 inhabitants to the rural commune of Vloesberg with only 70 inhabitants.

  The communes are responsible for a wide range of local services, including:

  • Providing public education
  • Maintaining roads and public buildings
  • Providing social welfare services
  • Collecting taxes
  • Enforcing local laws and regulations

  The communes are governed by elected councils and mayors. The councils are responsible for making decisions about local affairs, while the mayors are responsible for implementing those decisions.

  The term "commune" is also used in a more informal sense to refer to the local community or people who live in a particular commune. For example, someone might say "I'm from the commune of Leuven" to indicate that they are from the city of Leuven.

  Here is a timeline of the use of the term "commune" in Belgium:

  • 17th century: The term "commune" is first used to refer to local administrative units in Belgium.
  • 1789: The French Revolution of 1789 leads to the standardization of the term "commune" across Belgium.
  • 1830: The creation of the Kingdom of Belgium in 1830 reinforces the use of the term "commune" to refer to local administrative units.
  • Present day: The term "commune" is still used today to refer to local administrative units in Belgium.

  Note: The only fact relevant to the origin date that Bard was able to verify via matching google search results and listed in the "Checks Completed - Understand the Results" section, was the standardization following the french revolution. That was from this wikipedia page. I've bolded the relevant section above.

• Causing the problems they complain about

  2024-01-12 16:30:00 +0700 by Mark Smith

  I’m having to write this post without internet connection because the world has decided that everything is my fault again. Let me explain.

  Yesterday a confusion happened at a food place I went to. Seemingly, I got blamed for, at best the disorganisation of the staff. Though it looked very much like an entirely manufactured situation. They apparently couldn’t coordinate between preparation staff and payment staff and somehow it all became my fault even though I was very polite and calm, saying thank you, giving them plenty of money. Almost exactly the same situation happened previously at the same place a month or so ago. I wrote about it then too. That time they tried to short change me. It was the same staff.

  This morning another confusion at another food place. I asked the women there not to cut the pork into strips, others there had it that way so it shouldn’t be an issue. Another staff member, a man, at the same exact moment walked past and said ‘good’. When she gave me the food, she shouted ‘Lie!’ very loudly in a very aggressive voice. I had literally no idea what she was on about, which I calmly said to her. I walked off confused by there entire situation. It’s good, yet it’s also all my fault. Clear as mud.

  Anyway later today, I had to buy some new cheap headphones. At the small kiosk shop I went to, the woman there, who was covered in tatoos, tried to take my device off me when I asked if I could test the headphones. I wasn’t comfortable with that so I clumsily put the device in my pocket. A short while later after I had tested them, because she eventually handed them over so I could while purposely coughing all over them which unfortuntely is standard malicious behaviour here, I figured I’d haggle a bit. I opened my device, brought up the Notes app to type the amount I was suggesting. At some point I must have fat fingered the screen because the word ‘No’ was on the empty screen. I deleted it and typed the amount I wanted to pay, asked if she would sell the headphones for that amount. She said yes, I said thanks. And then as I prepared to pay she said I was lying and told me to ‘learn’. She still sold me the headphones.

  Now several hours later my internet connection stopped working, and a woman on a motorbike just drove past and said ‘No’. To me. Everything in the universe is my fault again! Never mind that I probably wouldn’t have fat fingered the screen had the other lady not been so insistent on taking my device off me.

  And of course as is customary here, everything will escalate until total destruction of the multiverse. At which point everyone will go into an unhappy funk and guess what, I’ll get blamed for that too, and eventually I will get mutilated.

  Literally every single interaction is like this. It’s the same at other food places, at the bakery, at the internet places, the shops, the cafes. Every darn place I go to. Whatever happens in one place is followed up with punishments in the next places. And honestly imho 9/10 they cause the issues in the first place.

• Writing annoyances

  2024-01-12 10:14:00 +0700 by Mark Smith

  As I attempt to write more blog posts I'm aware of various annoyances that get in the way. Here they are listed, hopefully I'll find a way to streamline.

  Moving between folders - There's currently a seperate folder for each post type: blog, linkblog, podcast, newsletter and notes. That's partly historical. In the early days there was only the linkblog folder, then I added the blog, then podcast etc. It is a bit annoying having to jump between folders, especially on a small screen. I might make more sense in an interleaved world to have all the posts under the same calendar structure. The downside is it would likely becomes more difficult to find some posts. Currently blog posts are in year based calendar structure whereas links are in s day based calendar structure. In a combined world all the blog posts would be in a day calendar, so viewing all blog posts from 2024 for example would no longer be possible via the filesystem. That's actually very annoying, especially if you can't render a local copy of the website.

  Linkblog links for blog posts - Each time I create a blog post I currently have to create both a blog item markdown file and an accompanying linkblog link markdown file. That's mostly because the latest page is set out in sections. If I don't add a linkblog link then folks are unlikely to see there was a blog post, since they are all the way at the bottom of the latest page. Creating the linkblog file is tedious because of the previously mentioned jumping between folders problem. A combined folder structure would make this easier, but also linkblog links might not be necessary in an interleaved everything main page paradigm.

  Creating files with frontmatter - Creating the markdown files is really tedious at the minute. Both the filename which had quite a long format containing the date and time. Blog posts also have the title converted to snake case and used in the filename. What makes it annoying is you often don't know the title when you start writting the post, so you have to go back and update the file name afterwards. Also creating the frontmatter is annoying because you have to ensure you have the right date and time which also has to be in the filename. I would be great to have a way to make this easier. Long term it should probably be a custom UI, but it would be useful to have the bare bones method of creating files by hand streamlined as much as possible. You always want to be able to fall back easily to just creating a file in a text editor.

  These are the main annoyances at the minute. I'll add more below if I think of / discover any more.

• Viewing the everything feed

  2024-01-12 09:26:00 +0700 by Mark Smith

  The website's new everything RSS has been live for a couple of days now. I've noticed that it doesn't display all that well in some feed readers. The reason is that many readers don't currently do a great job at displaying posts without titles. Since the everything feed includes everything, that means it has both posts where titles make sense like the blog, podcast and newsletter, but it also includes posts that have no titles, like the links and notes posts.

  In the earlier days of the web, when most feed readers were created, pretty much all posts had titles. But then Twitter showed up and proved without a shaddow of a doubt that not only did posts not need titles, but people absolutely loved writting posts without titles. The RSS specification doesn't actually require a title. As long as you specify either a title or a description or both, it's technically a valid feed. Feed readers haven't adapted all that well to the new paradigm. It's somewhat understandable because to be fair it's a bit of a design challenge.

  With that said here's how I handled it in Feedly. The feed was initially showing up with a load of posts that said 'No title'. I checked out the display settings and updated the 'density' settings from 'compact' to 'comfortable'. With that setting change, the title still displays as 'No title' but an abbreviated version of the description shows up below, so you can at least see what the post contains.

  Other feed readers will likely have similar display settings. It's worth trying some of them to see if there's a view that works better with a mixed post type feed like the everything feed.

• NYT vs OpenAI oddity

  2024-01-11 16:55:00 +0700 by Mark Smith

  I've been listening to various podcasts recently that cover the goings on in the publishing industry vis-a-vis how they are handling the new dynamics that have been introduced by large language models (LLMs). Publishers aren't too happy that their content is being used to train AI models, and the models are then steeling their customers.

  The New York Times is sueing OpenAI for copyright infringement. It's an interesting case. Here's OpenAI's public response to the accusations. I've been reading it every few days, there was something about it that seemed very familiar. It's been really bugging me. Well it took listening to somebody else reading it out to figure out what my brain was matching.

  It's weird but the way they describe the tactics used by the NYT to get the model to do unnexpected things, well that's what it feels like living in Vietnam as a foreigner. People here are constantly trying to hack you, constantly. I can't emphasize this enough, it's literally every interaction, everywhere I go, all the time. The same groups of people show up at the same places each time, and they seem to try out new tricks / hacks they have created or learnt. It gets very involved, happens across multiple locations. It's mind bending, enough to send you barmy.

  You do sort of get used to it after a while but it pretty much destroys your mood, makes it super hard to concentrate, causes confusion, often ends up escallating. Anyway, just wanted to put that out there, the NYT's described behaviour really reminds me of many of the people I meet or pass by here in Vietnam.

  Is this what the whole world is turning into? I hope not, it's rarely very fun, and often unpleasant. It feels a lot like bullying, and unfortuntely relatively often results in violence or injury or both.

• Everything feed operational

  2024-01-11 08:51:00 +0700 by Mark Smith

  In the past few days I've completed a refactoring of all the website's RSS feeds, and I've also added an everything feed. It's part of the bigger redesign of the main page, which currently shows the latest posts from the linkblog, blog, podcast, newsletter and notes. The idea behind the redesign is to move away from each post type in it's own section because it feels disjointed, given that each post type has a different frequency. The redesigned main page will have all the posts interleaved as they occur.

  I've been mostly writing notes and links lately and blog posts seem to have fallen by the wayside. Having everything interleaved will hopefully restore some balance. This post is the start of that. Many of the recent notes posts could really have been blog posts. They are quite long and almost all of them have titles. The reason they are ending up as notes is that creating notes is slightly easier, but I think it might also be because the latest page doesn't give you a very good at-a-glance idea of what's being posted. And that is I suppose a bit ironic given that it's called the latest page.

  First was to fix how the posts are displayed. Hopefully if the posts are interleaved then the balance of post types is on display so I'll be more conscious of what I'm publishing, and naturally tend to publish a more balanced collection of posts. I started with the RSS feeds, but I'm also working on an interleaved website main page.

  Second is to make it easier to post blog posts. The biggest hurdle was that previously the RSS feeds were not full text. Instead they only included the description. That was because I didn't think code snippets would be formatted correctly, so I figured it was best to require readers to click through to the website. Turns out that code snippet formatting is fine, though syntax hightlighting is lost, which is a bit annoying. So the blog post descriptions are no longer necessary. That's already a bit easier.

  Ideally I'd like to find a way to create new blog post files from a template. I haven't found a way to do that yet. I looked into some iOS apps that give you ways to create text blocks from snippet templates but the ones I found either didn't do it in a way that I could integrate into my workflow, or they required too many security grants, akin to installing a key logger, which I'm not comfortable with. I need to be able to create a markdown file with frontmatter, both the file contents and filename should be templateable. If you have any good suggestions, please get in touch.

  Anyhow that about wraps it up for this blog post. Hopefully there will be more of these going forward.

2023 #

• All-in Dirty Sanchez

  2023-11-18 10:19:00 +0700 by Mark Smith

  The latest All-in Podcast episode where they have a discussion with presidential candidate Dean Phillips, is another great episode. They've really perfected the 4 successfull entrepreuneurs joking around but also having serious discussions format. It got me thinking, why don't we have a comparable podcast coming out of the UK?

  I guess it's got something to do with the UK not being anywhere as big and important as the US. We are after all a nation of part-time gardeners. Or are we? It brought back memories of Dirty Sanchez, which was Britain's, or more specifically, Wale's answer to US created Jackass. If you are at all squeamish I would avoid searching for episodes on YouTube. Dirty Sanchez was the smaller but much more hardcore version Jackass, which was a tremendous shocker when you discovered it, because surely nothing could be more hardcore than Jackass. And so you watched Dirty Sanchez and you just thought, holy fucking shit with your mouth wide open. It's 10 years on and I still find it difficult to believe.

  And UK politics is so much less large than the US, how could the UK ever ceate an All-in Podcast competitor? Yeah that's true. And it's a lot duller. But maybe that is bizarely it's greatest strength. I think the thing that made Dirty Sanchez so good, wasn't just the shocking things they were doing, it was that it was a complete break from how media until then had been produced. It didn't feel like it was made by any of the terrestrial TV stations.

  And also remember the stage is potentially a lot bigger than just the UK. A dive and a quick swim and you're in the European Union, a rich fertile ground of dysfunction, just waiting to be analised by techno entrepreneurs trying to set themselves up for time in government while creating fancy self promotional vodka. Although what they get up to might be very different, especially if they are gardeners.

  Just an idea, it might seem impossible for the UK to create something like the All-in Podcast, but I'm here to tell you that not only have we done it before, but there is ample scope to create something really really great, probably without ripping your nipples off. There are many less painfull dimensions that remain unnexplored.

  I wish I could articulate all this with better prose, but I can't right now. I don't even have an AI assistant, times are tough. The point is that we could create something quintessentially british or dare I say it european. The time is now. There is no better time to be the media than when the world is going to hell in a handbasket. Feel free to turn this half baked idea into something amazing.

• Getting the blogging balance right

  2023-10-19 14:19:00 +0700 by Mark Smith

  I introduced a notes feature to the website several weeks ago. The big success has been that it has definitely made it easier to post, and I have been posting much more often than previously. The shorter and less formal structure of notes means that I don't agonize about getting everything perfect. I have a thought and if I deem it relevant, I can have it in a note pretty quickly. That's a big win, and worth remembering.

  Having said that though it's becoming clear to me that I don't have the balance right. My links are too long, or to be precise, the text accompanying the links is often too long. I'm constantly thinking that the link I've created really should have been a note. Ironically notes are also often also too long. It happens so often after writing a note, that I realise it should really have been a blog post. I don’t write enough blog posts. The way I have things setup just isn't working.

  There are several post types:

  • Short posts (notes)
  • Long posts (blogs)
  • Linkblog posts (links)
  • Podcast posts (podcasts)
  • Newsletter posts (newsletters)

  It's made somewhat more complicated by the fact that links can take different forms:

  • Full title with comment
  • Comment only
  • Long post with extracted quotes

  At the minute I seem to be favoring the title with comment form for links. For example here are today's links. That form is popular in various programming newsletters. As a reader, I like being able to see the original full title, but with a comment from the currator. That way you get a good sense for what's at the other end of the hyperlink before you click.

  The downside is that I'm not convinced it's so well suited for my needs, because very often I find that the comment contains a fair bit of analysis, and could have probably been a note.

  The other side of it is more practical. I like to have each post in it's own seperate markdown file. The problem with that is there is quite a lot of overhead. The time when ideas that form notes occurs at the beginning of the day, often as I'm gathering links, or listening to podcasts. They don't occur in order, fully formed. They tend to occur all at once, then they get fleshed out as the day progresses. The fact that I need to create a seperate file each time really gets in the way.

  I'm finding that I've started keeping a daily scratchpad list, that I then later turn into notes and links. The scratchpad file then gets thrown away. It's kind of a waste. I want the scratchpad list to be the notes right from the outset. But like I said, I like them all to be in seperate files. Quite the cunundurum.

  I'm starting to think that I should create a specialised markdown file format for daily notes. It would be some sort of list. Each item in the list would be a note, but would also include a timestamp. That way each day I would only create 1 file, and all notes would be written to that file. I would then have a custom renderer that would split the notes out into seperate files when the SSG builds the site.

  I would also be able to easily include, links and blog posts into the daily notes file. I think such a setup might make it much easier to write unincumbered. It feels like I'm quite close, most of the pieces are there, they just aren't organised in a way that's very productive. I'm constantly triping over myself, and so I end up just not posting.

  Having a good writting setup is incredibly nuanced and difficult to get right.

• Where is all the missing money?

  2023-09-25 16:57:00 +0700 by Mark Smith

  Peter MacCormack asked on a recent podcast, and unfortunately I can't find exactly where as I write this, he asked where is all the money that regular people have lost on the stock markets in recent years? Good question. Today's Eurodollar University episode, perhaps not uncoincidentally covers FX, Jeff Snider highlights what this shadowy part of the money system does. It's a vital part of the system, even if it is very opaque. McCormack did an interview with Nik Bhatia covering FX just this week.

  Up to 4 trillion dollars traded per day. Around 80-100 trillion dollars in contracts outstanding. That's a lot! For perspective total physical cash worldwide is 9 trillion dollars. Total debt in the entire world is somewhere in the region of 30-40 trillion dollars last I heard, though a quick search puts this number into question. This trade happens between money market dealers. Essentially it’s currency swaps but can be better thought of as collateralised loans. They function as a way to match people that need dollars with those that have dollars. They are redistributing currency and money and resources throughout the world system.

  They keep everything functional, making sure there is the minimum money circulation necessary for the system to function. But because they are highly constrained, they do this off balance sheet, because they don’t have the capacity. The fact that it is off balance sheet means that it is very difficult to see what is actually going on. Interesting stuff.

  Jeff points out that FX tells us something about the system from the perspective of the system, in a way that can't really be faked. In a normal world I'd be nodding my head, but in the current world I find myself in, that's the sort of statement that either is hidding fakery or will lead to fakery. It's almost a challenge. Anyway just wanted to note that.

  I find it kind of mindboggling that the system is so inneficient that it basically relies on these dealers just to ensure a basic flow througout the system. Isn't it a little like saying the global food system relies on drug dealers to ensure people get fed regularly? I'm still trying to wrap my head around the whole concept. I also get the sense that this might not be the most unbelievable thing about all this.

• Where might the Vision Pro make a big impact?

  2023-09-13 17:16:00 +0700 by Mark Smith

  I previously wrote about the real opportunity for the Vision Pro. From that piece:

  Perhaps the opportunity with the Vision Pro isn't with consumer apps, but with industrial apps. Sectors like construction, interior design, synthetic biology, materials science, big pharma, supply chain management and many others, have money to spend and a need to process large amounts of data in new ways. I see the Vision Pro less as a way to watch movies and more like a welding mask, a speciality device that you use in specific parts of your workflow to accomplish previously impossible data analysis and manipulation tasks.

  I had been thinking something along these lines from the moment I read the descriptions of the device. I spent my undergrad amidst science and engineering departments. I myself studied Materials Science but when you are at a science and engineering university you are immersed in the culture of all the various disciplines, from biology to aerospace, from civil to mechnical engineering to chemistry and all the rest. So I have a pretty intuitive understanding how such a device could make a big impact in these areas.

  The Bankless Nation podcast had a really fascinating episode talking to among others Drew Endy, the father of modern synthetic biology. I've transcribed some of what he was saying here, as it really crystalises why the Vision Pro could make a huge difference in science. He is asked What is synthetic biology? to which he replies:

  I think of it as wetware. There’s hardware, there’s software, there’s wetware. A cell is a cell, let’s just let a cell be a cell (no metaphors required).

  Let’s say I took a bacterium that’s microscopic, 1 millionth of a meter long, a micron. And I’m just going to have a magic wand and make it 100 million times bigger. So now it’s 100 meters long, and I’m not adding more atoms, I’m just magically making everything bigger. So we can see it, it’s like the size of a building.

  Now we can say, what does it look like? Let’s say I have a protein, a green fluorescent protein that makes a green light. That protein is the size of a basketball. And the ribosome, which is a collection of molecules that makes proteins, is 2 meters tall. And then the genome, the DNA, it’s going back and forth in this building 1600 times.

  It’s a really thin thread and so it’s cross sectional area is like 4 square meters. And there are many other molecules in this building, all about basketball size, and they fill 30% of the space of the building. We call that volume fraction, it’s 30% packed with molecules.

  Now we are just building this instantaneous mental image of what a cell is, and of course it’s alive. And what that means is that it’s self mixing. Brownian energy is causing all the molecules to jostle around through collisions with water, so that soccer ball, that green fluorescent protein, is moving…this is where it gets crazy…

  It’s moving with an average velocity of 500 meters per second. It’s just instantaneous velocity is faster probably. What that means is that the molecular - molecular collision time is a nanosecond.

  So I’ve got this self mixing milieu that’s 30% solid, 70% water, with a gigahertz collision rate, and what it’s doing is being encoded by the genome, instantiated as a physical mixture, it’s receiving energy from the environment.

  Within the period of 10-20 minutes it could make a physical copy of itself. That’s wetware.

  Doesn't that sound awesome? Doesn't it sound like the sort of thing that could be revolutionised by the Vision Pro?

  And guess what, the whole of science and engineering disciplines are full of these types of visualisation problems.

• The Mental Health Deception

  2023-09-13 11:36:00 +0700 by Mark Smith

  When I was young we simply never spoke about mental health. There were normal people, and there were a very small amount of people that had mental disorders, and those people usually got locked away in mental asylums. That’s just how it was pretty much. It’s horrible to think that it was like that, but it was. In some places it might still be like this.

  We’ve progressed a long way from those days. People realise that mental health is something that can affect everyone. Folks that used to bully others, calling them crazies and mental, are now understanding that the way they treated people before wasn’t very nice. They have seen friends or family members be affected by these issues, or indeed have experienced something similar themselves.

  In part this is because the modern world is ever more complicated and this is reflected in the stress experienced by the average person. In the modern world people have more mental health issues.

  We are due another step forward in how we view such conditions, and it’s around the realisation that the notion of mental health issues is still very antiquated way of looking at it. The problem is that it is implicit in the term that the fault is the person with the mental health issues. That somehow they have brought this condition on themselves by the way they lived their life. That’s the unspoken piece, the unspoken belief that’s never said. It’s something ‘they’ have.

  If you spend a little bit of time thinking about it, this notion is absurd. There are a very very small amount of people that perhaps have a physical abnormality in the brain, but the vaste majority do not, yet there are many many people with mental health issues.

  The fact is that by definition these people that have no physical disability and have mental health issues, have these issues because of the people they interact with. It must be the case that these interactions are causing the so called mental health issues. In short we must realise that mental health issues are really issues of the group bullying individuals. The environment creates the mental health issues not the individual.

  Even if they have substance abuse problems, why do they have these problems in the first place?

  Mental health issues is really a failure of the group.

  It’s likely that it’s a system level problem where to the total information processing capacity of the system is being reached, and for whatever reason these folks always find themselves as the pressure cooker release valve. It’s not a coincidence. One day we will look back at our primitive understanding of group dynamics and realise that groups can turn into a mob and they might not even realise they are doing it, though they also might. Ultimately it doesn’t matter though. What matters is noticing it is happening and course correcting collectively in a way that de-escalates and moves the unhealthy dynamics to a safer configuration.

  In the present day, "mental health issues" is a euphemism for someone that the group systematically and relentlessly bullies. We need to evolve this understanding into a more truthful and effective model.

  This post was originally published as a note, but I decided to put it out as a blog post. Notes tend to be more off the cuff, perhaps less well thought out.

  Personal note: I wasn't particularly happy with the title. Specifically I feel like the word 'deception' might not convey the right message. I'm not saying there's some big conspiracy, it's more like a sort of emergent group deception. Anyway rather than get too hung up on writting the perfect piece, I just want to get it out there, it's the general idea of the piece that's important.

• Last Man Standing Tech

  2023-09-13 11:21:00 +0700 by Mark Smith

  Eventually there will be AI that will be so powerful they will be able to decide who will be the winner in a death match between 2 humans. They will be able to do it with accuracy and precision comparable to chess players playing chess.

  Except they will be able to basically see infiniti number of moves into the future, in infinite dimensions. And people simply won’t interact with those that the AI tells them that they can’t beat. It could very well be something that we don't consciously choose.

  Last man standing tech will be a very big sector indeed.

  I make no judgement here as to whether I believe this to be a good or bad thing, only that it's a possible scenario we need to consider.

  This post was originally published as a note, but I decided to put it out as a blog post. Notes tend to be off the cuff, less well thought out.

• I’ve setup a nip5 on my Nostr account

  2023-09-04 11:45:00 +0700 by Mark Smith

  Nip5 is the way you verify your Nostr account. I think the idea is that once configured, you can give people your nip5 instead of your public key, which is ridiculously long, so they can find you. If you self host your nip5 configuration it also proves who you are i.e. that your account was created by the person that controls your domain name. That's the theory. In practice I've found it to be quite confusing to setup.

  My nostr nip5 should be: [email protected]

  Confusingly it looks like an email address, but it's not an email address.

  Once you get past the ambiguities in the instructions, it basically boils down to putting a file containing some json formatted text on a server accessible at a known path and add it to a domain name that you control. The difficult part is that it's not very obvious what text exactly you have to put in the file. You have to add your 'name', but your Nostr profile has 3 fields that look like names.

  Here is my nip5 config. As far as I can tell it's in the right place, has my domain name configured, is accessible to all clients, and has the correct mime type.

  Also once you have it all configured, it's not obvious how to check that you have it correctly configured. I'm using a Nostr client called Damus and when I open it, my username has a tick next to it, but I seem to remember that tick was always there. I don't see my domain name anywhere, so it's not clear whether it's setup correctly.

  According to nostr.band my nip5 isn't showing up in searches. I guess that means it's not configured correctly.

  So much faffing about. I've literally spent around 8 hours at this point. That's ridiculous. How do they ever expect regular non techie people to get onto Nostr?

  Update: Discovered that I hadn't merged the feature branch into main so the config was there but only on the staging server. It should be in production shortly.

• Very long timescales

  2023-08-29 06:05:00 +0700 by Mark Smith

  We are enterring a new era where the timescales we have to consider for practical purposes when we build our software systems are much much longer than anything previously. And this is going to be very strange. Lots of narly edge cases and counter intuitive situations. The new Wordpress 100-year plan is an example of this. Long term nuclear waste storage is also an example that is related.

  From the linkblog 2023-08-27:

  Wordpress: The 100-Year Plan - "The 100-Year Plan ensures that your stories, achievements, and memories are preserved for generations to come. One payment. One hundred years of legacy. US$38,000." - It just so happens to be $1 per day, which means even if you only had 1 customer you might just be able to pull it off. What a curious price, don't you think?

  With that in mind the lastest Duncan Trussell Podcast with Mitch Horowitz is very interesting. Of course it's quite far out there in a lot of places, but like it or not, this stuff is the prior art and we should consider it in some way, because these questions that we need solves for are very challenging indeed. Much more challenging than we realise at this point. Being practical at timescales that approach a human lifespan is very difficult indeed.

  You have to start thinking generationally, and we haven't really even started building tools and processes for that. This is especially the case for free and open source software, which doesn't really have many parallels in our past. You might think that it's not that relevant, that these are outliers, but when you look at the technologies we are developing in the biology and life sciences sector for example, IMO these types of dilemas are going to start cropping up much more frequently.

  Update: I just listenned to the latest Bankess podcast. Not much else to add at this point, but it's clearly very relevant. This stuff is very complicated even before considering the very long timescale implications.

  • The Superchain Explained with Jesse Pollock & Ben Jones Ep#185 (Bankless Podcast)

  It's like we are all climbing a cliff together. It's potentially very dangerous but with the right process and equipment many of the dangers can be mitigated.

• Double meanings across cultures and languages

  2023-08-28 08:39:00 +0700 by Mark Smith

  This is my first attempt at writing about this so it might not make a lot of sense. Mixing cultures is complicated, especially when there is a language barrier. One thing that makes it very confusing for everybody is double meanings. Double meanings are just part of a language, sort of unavoidable, but as well as causing confusion they can create amusing situations, but also arguments. The point is it’s important to be aware of them, which isn’t easy, but over the course of a lifetime your brain gets better at it.

  The other thing to remember is that in any interaction where people are at different levels of understanding of a language, the focus of attention and comprehension of those involved will often be at different levels of abstraction. When traveling in Asia, I notice again and again that people here understand many english words but they often concentrate on syllables and sounds of the words. This makes sense because they are not writing the words so the word boundaries aren’t so obvious.

  Anyway what happens is confusion much of the time but actually you also end up learning double meanings in your own language(s) that you had never noticed before. It's unbelievable how many of these you miss when growing up, how often this happens. Much of the time this is in fact interesting, though it does sometimes get rather annoying being stuck right at the lowest level of abstraction. I'm constantly amazed at how often in english the meaning at high levels ofabstraction is so very different from the meaning of the individual words. It's often the exact opposite!

  No doubt there is some technical reason why this happens, due to how languages evolve, cultures shift, meanings change, even get reversed. If you know two or more languages, especially of neighboring countries that have historically been at war a lot, you'll notice lots of oddities that can't possibly be entirely by chance.

  Something else to remember is that the way some other languages are structured will affect how people interact with the language. The sounds you make have totally different meanings in other languages, and it’s difficult to not see those other meanings, which can lead to people being offended. Also you have troublemakers (on all sides!) that think it’s absolutely hilarious to play tricks with these language double meanings. Others that will use it as an infinite source of metaphorical things to hit you with.

  In short it’s quite a minefield but can be a source of joy and fun and learning, especially when there is some trust and a little understanding that these dynamics exist.

  Update: As I was converting the note that started this thought into this blog post, an elderly, somewhat beligerant, vietnamese chap in his mid seventies sat next to me. I mived my bags so tgere was room for him. He was going for his morning walk, exercising what looked to be a leg injury, but perhaps brought about by a stroke.

  We had a nice chat in english, even if it was short sentences, with some ideas that went nowhere, disapearing into the void. Some very long pauses, which isn't so bad, sometimes it's actually a good thing, gives you time to sense each other's presence. His son is a doctor, working in a hospital, which I think is nearby. Also vietnam is #1. He was pretty clear about that :)

  Eventually a couple of elderly ladies passed by and picked him up, helping him on his way to his next destination. Quite fitting such an encounter should happen as I'm blogging about cultures and language.

• UK politics cognitive disonance

  2023-08-26 18:53:00 +0700 by Mark Smith

  I was listening to the US politics discussion on the latest All-in Podcast earlier, it's very good you should give it a listen, and I was struck with how much more grandiose US politics is than UK politics. It makes sense because they are the world super power so everything they do is in that context.

  The slightly odd thing about UK politics is that because we used to be the world super power, all the places where politics happen still have the grandiose vibe, yet the politics isn't in comparison. It's sort of commical in a way. All these politicians acting out their careers on this incredible stage, yet the plays they perform are children's plays. I wonder how that affects their ability to govern. Are they out of touch with the people?

  And yet perhaps it is still important to be on a big stage. When we were participating in the European Parliament, would we have been able to have a big impact had the politicians not been trained on the world class british political stage? At the EU level, politics is a much bigger game so to speak. I'm not sure how to square the circle, or indeed what I'm trying to say. I worry british politics is the wrong shape, bigger on the inside than on the outside.

• The real Vision Pro opportunity

  2023-08-22 10:43:00 +0700 by Mark Smith

  I've been meaning to write a blog post about the Vision Pro for several weeks now. It keeps getting relegated on my various todo lists in favour of other more pressing items. I was reminded of this by a Dave Rupert piece that explores why his initial Vision Pro entusiasm is now wanning.

  I'm still kind of excited about the device, but I was always somewhat puzzled by the expectation that this is going to be a mainstream device. A device that folks not on a plane are going to be using to to watch movies and have video conferences. This just seems plain ridiculus to me, especially given the price.

  I think we are on the cusp of many industries being revolutionized by AI, LLMs, cloud computing, cheap sensors and there is going to be a need for powerful visualisation tools. For these types of industries the price isn't an issue. Traditionally these areas of the econony have not been 'cool'.

  Perhaps the opportunity with the Vision Pro isn't with consumer apps, but with industrial apps. Sectors like construction, interior design, synthetic biology, materials science, big pharma, supply chain management and many others, have money to spend and a need to process large amounts of data in new ways. I see the Vision Pro less as a way to watch movies and more like a welding mask, a speciality device that you use in specific parts of your workflow to accomplish previously impossible data analysis and manipulation tasks.

  Remember there's this other product called the Mac Pro that was introduced quite recently. Top of the line, very powerful. Also very expensive. I bet it would look quite nice as a package with the Vision Pro. They are going to have to figure out some way to make it more attractive given that even high end visual effects artists are happy just using their Mac Books.

• I ran out of build minutes

  2023-08-22 08:55:00 +0700 by Mark Smith

  I ran out of build minutes, so that's the end of new content on the website until next month. It's the same thing that happened last month. Last month I ended up going on a bit of a coding frenzy. Of course I couldn't test any of the code so I ended up with around 25 feature branches that I then had to merge in once the new billing cycle started and I had a fresh set of build minutes. That merge extravaganza was in a way what led to me running out of build minutes this month.

  The silver lining in all this is that I have the new notes feature, which I'm finding really useful. It's also raising lots of questions in my head about frictionless publishing workflows. This month I'm going to try not to write any code while build minuteless. That's harder than you might imagine. The impulse to code is quite strong.

  I'm going to keep publishing links, notes and blog posts though. They will be quite easy to merge back in because you don't tend to get bugs or merge conflicts with prose. There will probably be an increase in spelling mistakes though, as I won't have the opportunity to read back what I've written after publishing.

  These two incidents have highlighted a downside to the current website architecture. Namely that if the build system is blocked for whatever reason, then that not only affects coding new features, but it directly affects adding new content. I spent a while thinking of ways to mitigate that but I haven't managed to come up with a full solution.

  The limited build minutes was the reason why I batch all content updates to once per day. That way the number of build minutes needed is predictable. I'm also looking to optimize the deploy workflow. The npm cache I added hasn't worked for some reason, but I'm confident I'll find a way to speed things up.

  Having said that, the build system could go down for other reasons too. Ideally there would still be a way to publish even if the build system was down, even if it was a slower method. I'm not sure how I could achieve that short of rendering the markdown on the client, and I'd rather not do that. Something to think about I suppose.

• Text template rendering should be standard in all operating systems

  2023-08-21 08:11:00 +0700 by Mark Smith

  It should be possible no matter where you are located or what you are doing to render out a template and get back the resulting string of text in a way that you can use it to accomplish the task you are doing.

  Seems like an easy ask, but it's actually quite tricky because the UI is different depending on where you are and what you are doing. What that means is that it's not obvious how you get or provide the required inputs to the template rendering, but also how you use the resulting string.

  Are you on the command line in a shell? Or on the Desktop? Or maybe you are in an app?

  Where are the inputs you need to insert into the template? How are you going to get those? Manually type them? Auto generate them by running a script?

  What are you going to do with the rendered string? Copy and paste it? Stick it in a file? Perhaps insert it into a form?

  Once you stop to think about it for a few minutes, it's not at all obvious how to implement a system wide feature like template rendering.

  The most obvious way might be to make it an enhancement to the standard copy / paste feature of the operating system. Have a way to run a script to fetch data, pass it into a template you select, and get back the rendered output on the clipboard ready to do with it as you please.

  That would be really useful to me right now. I already have a title for this post. It's right there in the frontmatter. What I want to be able to do is select the title using the standard operating system copy feature, then run it through a template that would transform it as follows:

  Text template rendering should be standard in all operating systems
  

  would get transformed to:

  text-template-rendering-should-be-standard-in-all-operating-systems
  

  That would give me the text slug which I could use in the filename of this blog post's markdown file.

  2023-08-21-081100-text-template-rendering-should-be-standard-in-all-operating-systems.md
  

  You would probably need to have a library of utility functions you could use in your templates. In this case we would need some sort of slugify utility function as well as datetime manipulation functions.

  I could continue using whatever editor I was using as normal, using it's 'Save As' feature, but no need to mess about getting the filename correct. In a couple of clicks it would be correct every time. It's small features like this that have the potential to remove huge bariers.

• Testing auto posts to social media

  2023-07-18 17:36:00 +0700 by Mark Smith

  I realised this morning that I'd forgotten to update the service I use to auto post the daily links to Twitter when I moved the website from Heroku. For many months now the links haven't been getting syndicated.

  I've updated the configuration with the new RSS feed url, so hopefully they will start being posted again. Previously I was just posting a daily linkblog link, but since I've been blogging more recently I decided to configure for new blog posts to also get posted. And then I figured, what the hell might as well post them all to LinkedIn as well as Twitter.

  So anyway this post is just a test of that. Once published it should appear both in Twitter and LinkedIn. No support for Threads yet, though I've emailed dlvr support to ask them if they'll be adding that as an option.

• Nicer looking linkblog

  2023-07-11 18:47:00 +0700 by Mark Smith

  When I migrated the website from Heroku there were so many things that needed to be done. It was pretty much a complete site rebuild. At the time my driving thought was to get the basics right, or at least mostly right. With a good foundation I could make things better over time. Some of the big things were:

  • jamstack with github workflows
  • single domain rather than multiple subdomains
  • all content in markdown files rather than json files
  • server side components

  There were loads of others. When I was laying out the HTML, I wanted the site to use latest HTML5 tags and best practices, and I also wanted the whole site to at least look pretty good with a small amount of inline CSS. I spent ages trying to figure a way to organise <main>, <article>, and <section> tags that made sense. It's not that obvious when you get into the details.

  A blog post is an article for sure, but what about a day on the linkblog? What about a single linkbkog link? What about the blog/podcast/newsletter listing pages, is each entry an article? And what about the latest page, should I even bother with article tags there, maybe just use regular unnordered lists? Should I nest articles?

  So many small details that can actually make a lot of difference. Initially I went a bit overboard on unnordered lists. I had <ul>’s basically everywhere. Lists of blog posts, lists of links, lists of everything. The nice thing about lists is you get a small amount of visual structure and spacing for free, especially if you sprinkle some <p> tags in the right places. I figured I could style the lists later with CSS to make everything look better.

  I don't know that I've gotten it right or even mostly right yet, though it feels like I'm making progress.

  Anyway one place that's been bugging me was the main linkblog page. I initially went with a list of lists approach. A list of days, with each day containing a list of links. It worked reasonably well, though the bullets change on each indentation level. Level 1 has normal looking bullets, circular whole black dots. But level 2 they are hollow, with the center of the black dots removed. They look wierd. The indentation also looks kind of bad on mobile devices, because you waste a lot of the valuable screen real estate on the left side of the screen.

  Here's what it looked like:

  

  I've been trying to style the lists for a while now. I never got it looking good. The styles weren't applying the way I wanted. I would style the outer list, and the inner list would also get styled, or the opposite way around, even though I was targetting different class names. Lots of finicketty stuff like that. I gave up eventually, figured it was good enough, I had more important things to worry about and get right.

  Anyhow it occurred to me yesterday to just ditch the outer list. Instead just have a load of <articles> items on the page, each one still containing a list of links. I wonderred what the default spacing would look like. Well it was a quick change to remove the outer list, tried it last night, and it worked wonderfully.

  Here's the updated linkblog page:

  

  Ok so yeah it looks like a very minimal change, but I think that looks way better. I still get the structure, spacing and indentation of the bulletted lists for the links, but less wasted space and only one type of bullet point. I know the whole site is still too minimalist, but broadly speaking it's way easier to read, with the day dates snug on the left. It definitely is way easier to read on mobile, and these days that's how most people read the web.

• The problem with some finance writing

  2023-07-11 18:38:00 +0700 by Mark Smith

  This going to end up sounding kind of complainy, sorry.

  I was listening to the latest Eurodollar University Podcast episode Where did it all go? It was all a lie. and it starts off with a bit describing quantitative tightening using quotes from a finance article. It perfectly illustrates the problem with many finance articles. This isn’t a hit on Eurodollar University per-se, I know he’s just quoting other people’s words, but I think this type of dynamic in financial writing is a bit endemic in the industry.

  It’s worth listening to the segment first, it starts at ~00:50. It’s about the main stream view of quantitative tightening. I’ve pulled out the main steps to this logic chain:

  1. Quantitative easing is a way to stoke animal spirits.
  2. It’s the exact opposite of quantitative tightening.
  3. They are depressing animal spirits by pulling money out of the economy.
  4. The FED only directly controls short term rates.
  5. By selling bonds they are pushing down the price of long term securities.

  I listened to this and was like, yeah sort of makes sense, but either I’ve missed something, or there is some missing information. Remember it’s delivered via audio rather than text. I guess in an ideal world I would be consuming more text, but my internet access is such that I have to do a lot offline, hence lots of podcasts.

  Let’s go back over it in more detail, this illustrates vaguely my thought process as I listened:

  1. Quantitative easing is a way to stoke animal spirits.
     • Yup high level statement, things get easier, people feel more confident, exuberant sentiment, they do more stuff, take more risk, trade more etc.
     • Note they didn’t say how that was implemented.
     • That would have been helpful, even if it’s obvious to many, it’s not obvious to all, especially newbies.
     • So already at step 1, the very foundation of the logic chain we are missing a vital brick.
     • The finance monster just took a bite out of your working memory.
     • You are forced to do mental gymnastics along the lines of "ok let’s keep going, that will probably become obvious".
  2. It’s the exact opposite with quantitative tightening.
     • Yup quantitative easing made sense, so it makes logical sense that tightening would be the exact opposite. Remember we are still doing mental gymnastics, holding that missing brick for the finance monster. We are ok but a little unstable.
  3. They are depressing animal spirits by pulling money out of the economy.
     • That sounds good, finally we get implementation detail. But it’s not for our missing brick, it’s the implementation of the opposite of the missing brick. Ok hold on while I create even more space in memory for that. It’s f-ing with the base brick, so worst case is that you have to double the amount of memory, because you now have a logic chain with a brick and an entire separate logic chain without a brick. But the missing brick now has implementation detail, so in a way you have freed up a bit of memory but, you have to use up memory to compute that, so it’s like a little bit worse but then better as long as you can still remember how we got here. Anyway, moving on…
  4. The fed only directly controls short term rates.
     • This appears to be completely separate from the previous logic chain, so we have to park the unstable two headed logic chain from before, heave ho.
     • That was quite heavy, but it’s parked now.
     • Uh oh we are getting behind now, what was this FED stuff again? Yes the FED changes rates.
     • Yes indeed the FED does change interest rates, makes sense, but wait what kind of rates did they say?
     • Oh no too late, no time, we’ve missed that, we’ll have to park that now too. Heave ho.
  5. By selling bonds they are pushing down the price of long term securities.
     • Ok so we start here with a parked unstable double headed logic chain that sort of makes sense, and another parked statement about some kind of rates that also made sense but there’s a brick missing, i.e. it’s also unstable.
     • Selling bonds? Hold on that’s new, yet he’s saying it as if he’s been banging on about bonds for ages. But he hasn’t. Oh ok, let’s go with it anyway assume the best.
     • Why are they selling bonds? It’s totally not obvious, even though he’s pretending it is.
     • And remember we are already using a shitload of memory to balance all the previous memory debt incurred.
     • You need to compute it yourself.
     • The writer is offloading both memory and compute onto you dear reader.
     • So let’s unpark our unstable double headed logic chain from before. Heave ho.
     • Rewind, remember that we are talking about quantitative tightening, which was pulling money out of the economy. Ok, right.
     • So selling bonds is tightening. Ok but why?The answer I think is something like, they sell bonds, which people buy, i.e they pay for them with money they have, thus money is pulled out of the economy.
     • But wait there’s more, what’s this ‘pushing down long term prices’ bit?
     • Why do prices get affected?
     • Oh look, again we are having to compute for the writer.
     • Selling bonds probably means there are more bonds in the market. They create and sell bonds so effectively they are diluting the existing bond market. I might have this wrong, I’m not an expert.
     • The overall market is bigger, i.e. it’s size is inflated, so the bonds that were originally there are worth less.
     • Why don’t they just say it’s bond inflation? Same as money inflation. Why are we constantly describing similar things using different mental models? Anyway…
     • Oh look and just to confuse us even more, the writer is now calling them both bonds and long term securities. They are the same thing. You have to compute that too.

  A finance explanation that uses up so much tech debt doing the explanation that it’s basically impossible to quickly understand the issue because the writer has used up all your memory and compute.

  Oh the irony, an explanation of how the economy avoids bankruptcy, is itself doing a very effective job of bankrupting the minds of the people it’s teaching.

  It’s basically a fractal, at which point I’m out. Fuck this this shit, it’s probably best to just walk away and listen to something else.

  I say all this knowing that this article that I’m writing is kind of slap dash. I don’t have the time or resources to spend editing it until it’s perfect in every dimension. I’m sorry about that. I’m learning too. I guess my gripe is that a lot of finance articles have similar issues. It’s weird to me that no one else is complaining.

  I’m left wondering how different the world would be if the explanations were better. At a macro level don’t these types of explanation just restrict the total number of people that can learn about how finance works? The total compute capacity of humans is likely finite without adding more neurons. Are people doing this on purpose or are we just terrible at explanations?

  Is it any wonder the world has issues understanding money, if we can’t even balance the books explaining the thing, how are we ever going to balance the books doing the actual thing?

• Final destination mutilation

  2023-07-09 06:45:00 +0700 by Mark Smith

  

  Just another final destination mutilation. Hate to say it but, I told you it would happen again literally yesterday. It always does. This time it involved a ring, questionable soup and a piece of floor bread, but had been building over several days.

  Thanks world, you win again, most cowardly punch in the universe.

  FYI, I literally asked for world peace.

• Bulk importing the HN bloggers OPML

  2023-07-07 18:00:00 +0700 by Mark Smith

  Following on from importing the HN bloggers OPML file, figured I'd quickly document the process.

  Here are some screenshots of what that looked like:

  

  

  A little bit tricky to do from a mobile device. The import OPML button was only just visible on the screen, and didn't appear at all in the iOS app. Importing took a really long time, around 30 mins. I had to restart the import several times, which was a bit annoying, but at least it was clever enough to not add duplicates. Lots and lots of errors, I guess many of feeds didn't validate, or got incorrectly added to the list.

  Once all 692 feeds were imported, here's what the combined feed looked like:

  

  

  

  

  The bulk import via OPML is nice but the ability to subscribe to an OPML file would be better. That way when the OPML file gets updated, whichbit likely will, the combined feed in Feedly would stay up to date. Why isn't that a standard feature of RSS readers at this stage?

  I haven't had much of a chance to read through many blogs yet. I've been busy getting multi-paragraph linkless linkblog posts working for my static site generator. Oh and installing Meta's new Threads app.

  Anyhow it's cool that we have RSS. It's cool that it's permissionless, that you don't need an invite, but I'm OK with having other social media apps too. It's good to have variety, each one has a distinct vibe.

• Hello HN bloggers

  2023-07-07 18:00:00 +0700 by Mark Smith

  Just a quick post to say hi to any folks that might have made their way here from hacker news. Howdy :)

  From the linkblog:

  Hacker News Blogroll - Someone on HN decided to compile a big list of people's blogs, and they turned it into a website. Someone else created a giant OPML file with all the blog feeds, and I imported the whole thing into my Feedly. It's a bizare sort of mayhem now in my feedly, I've seen english, russian, german and vietnamese posts pass by so far. It's nice to not require an invite, even if it's quite confusing :)

  I'm Mark, web developer, blogger, sometimes traveller. More about me on the about page.

  I haven't had a lot of time yet to read through the 692 (!) blogs I added to my RSS reader. Hopefully I'll have time over the weekend.

  Welcome to the blog, I don't have comments but I read emails, so feel free to mail in comments.

• Building digital asset economies

  2023-06-30 08:00:00 +0700 by Mark Smith

  Dominic Frisby had an interesting episode on his Flying Frisby Podcast this week. It was an interview with Microstrategy CEO Michael Saylor. Saylor has made a name for himself initially by successfully running a publically traded software company for over 20 years, but more recently for his rather adventurous corporate investments in Bitcoin, something made possible by focussing on fair value rather than GAP accouting.

  Microstrategy holds some of the largest amounts of Bitcoin worldwide, and has quadrupled in value by taking Bitcoin onto it's balance sheet despite the volatility. Most organisations haven't been willing to do that due to gap accounting practices which don't differentiate between gains due to operation and those due to balance sheet changes, for instance appreciation in commodity assets. Gap accounting companies run the risk of big volatility swings in Bitcoin's value eclipsing otherwise normal operations, making a company's performance look much worse than it appears.

  Saylor is a Bitcoin promoter, given his holdings that much is clear, but the narratives he pushes are very well thought out, always attempting to be rooted in fundamentals. One of the narratives he has been putting forward recently, and which seems to be catching on, is that of building digital asset economies. He believes this is something that countries around the world are going to be getting into in the near term. And with the recent announcement of many large TradFi companies (Blackrock, Fidelity, Citadel, Schwab and Deutchebank) creating various flavours of Bitcoin ETFs, it does indeed seem to be trending in that direction. Multinational corporations need to park their money somewhere, and given the prospects of the US bond market, that place might very well be Bitcoin.

  In the interview he notes how it’s important to distinguish between the different digital asset classes. Regulation takes time to evolve and people are only starting to understand the difference between these. Namely the difference between:

  • Digital commodities
  • Digital securities
  • Digital currencies
  • Digital tokens

  I've transposed his definitions here, as I think they are useful, even if they haven't been fully endorsed by regulators yet, because they paint a very clear picture of the future financial environment that crypto and bitcoin proponents envisage.

  Digital commodities

  • Assets without an issuer
  • Like Bitcoin

  Digital security

  • Assets with an issuer, a controlling body creates the currency and they control it
  • Like Solana
  • Like Algorand

  Digital currency

  • Stable coins, they proport to be the US dollar as a bearer instrument and a digital asset, a blockchain representation of dollars
  • Like Circle
  • Like Tether

  Digital tokens

  • Like an NFT, piece of art 1 of 1
  • Like a Tom Brady coin, 10000 super fans coins that he issues, holders get to for example come to his house for a BBQ every month

  Saylor then outlines the growing importance of distinguishing these:

  A digital exchange is something that can trade all these things. To create a digital asset industry in your country's economy you have to have a universally accepted taxonomy of digital assets, a digital asset framework and political consensus that it’s ok to issue, trade and custody those assets.

  We don’t currently have the political consensus for many things. There is a big divide. Saylor gives several very specific examples that help to illustrate the sea change that decentralised finance is bringing to the table.

  • Moving 1 million dollars from A to B without KYC would be considered money laundering in TradFi
  • Trading a security 24/7
  • Self custody a million dollars of digital assets in stable coin from an android phone
  • Self custody a million dollars of Apple stock on an android phone in India
  • Trade with a counter party that you don’t know via smart contract
  • Buy a perpetual swap derivative on a Saturday night that represents a registered public company
  • A person living in India that has 10 million dollars of Apple stock, borrowing against that from any counter party in the world on a Saturday afternoon, getting the best bid

  All these things and much more are totally possible with DeFi. It's going to be interesting to watch how this evolves.

• Ideal offline reading app for web development content

  2023-06-29 17:33:00 +0700 by Mark Smith

  I've spent the early part of the week exploring various Read Later apps for reading web development content offline. First I compared Safari Reading Lists, Pocket and then I added a comparison of Instapaper. These are the most popular. What I found was that none of them were adequate. It's important because it's one of those things you don't realise you might need until you do, at which point if it doesn't work, well it's too late.

  The main reason the current apps all fail is their handling of images and code snippets. A huge part of web development content is diagrams, screenshots and of course code and configuration examples. Without these it's very difficult to understand what the author is attempting to communicate. Most of the current apps remove page styling in order to save on space and apparently to create a consistent reading experience.

  That's all well and good but by removing the styling they also remove the code syntax highlighting and occasionally the formatting, including newlines. Without newlines the entire code snippet is squashed onto 1 really long line. That makes it really really difficult, nay impossible, to comprehend web development articles with any kind of complexity.

  The code snippets problem is bad enough to make things really terrible, but on top of that, very often images from the saved articles don't get saved. Instead either empty rectangles where the images used to be, or indeed, nothing at all. You might not even know that something important is missing. Both these bugs together make reading web development content offline effectively impossible.

  I put together an opinionated list of features I think is necessary for a Read Later app in order for it to be useful for reading web development content.

  Basics

  • Setup should be easy, install the app and you are ready to start saving articles
  • It should be totally obvious how to save a page from Safari, no wondering if you have to use a bookmarklet or the "share sheet", whatever that is
  • No need to configure anything in iOS settings
  • Saving always at least adds a new item, or displays an error somewhere if there was an issue, a viewable activity stream for all modifying operations performed would be useful
  • All images in a page get saved no matter what
  • Code snippets and config file examples remain untouched, with syntax highlighting, newlines etc
  • App displays when it is downloading / synching items
  • App downloads and syncs in the background so you never have to check that it synched before you go offline, no surprises when you get offline, the articles you saved should be there ready for you to read
  • Clear docs with guidelines / best practices for website devs so they know how to structure their pages in order to have their pages save correctly
  • App doesn’t randomly refresh pages, once the page is downloaded / synched, you can switch between pages quickly and easily without the page going blank or loosing content
  • App doesn’t insist on restarting every time you bring it to the foreground
  • Title, description and website address are always saved and displayed correctly on saved articles listing screen
  • All text of an article can be copy pasted, including author, date/time, title, images and code snippets
  • Images can be zoomed
  • Listing page has a thumbnail next to item

  Nice to haves

  • The page looks identical to the page you saved, don’t remove styling
  • In the background save copies of pages linked to by the saved page, and be able to read these pages offline when you click on associated link in the saved page
  • Videos get saved for offline viewing
  • Widget to easily see the list of items being downloaded / synching
  • Automatically create an archive of the page using an archive service like archive.md or archive.org way back machine

  I think that a Read Later app with these features would be good enough for reading web development articles offline. Reading articles offline is not only extremely convenient in a busy world but it could even prove vital should you experience internet connectivity issues. Hopefully that's something that many of us have learnt, having now got through the many pandemic lock downs of the past couple of years. App builders please consider making your apps ready for these types of scenarios.

• Revisiting Instapaper for offline reading

  2023-06-29 07:12:00 +0700 by Mark Smith

  Following on from Monday's piece about the issues reading web development content offline with Safari Reading Lists and Pocket, I wondered again about Instapaper. I had previously tried it but gave up after getting frustrated with the experience. There was a lot going on last time I tried it. I was having to jump around between apps, and was also in a bit of a rush. It's easy to get lost in a sea of features sometimes. Was the overall app experience really as bad as I remembered?

  I re-opened the app to have another look. The default screen on opening the app is the saved article list. The first thing I noticed was that it just sort of looks a bit classier than Pocket's listing page. The title font and the layout of each item results in a vaguely newspaperish vibe. The content just seems to fit the available space a bit better. There are both titles and descriptions and less white space around the thumbnails. The article titles all seemed to be correct too. When an item does get saved, it looks kind of nice. I decided to give Instapaper another try.

  Let’s look at the same screenshots:

  

  

  

  As with the Pocket, the article saved in Instapaper has most of the page styling removed. It looks much the same as Pocket’s version. The site’s style is lost but the page is still readable. I guess some folks like the consistency. Personally I prefer being able to see the original website styling.

  

  

  

  As with Pocket, the article saved in Instapaper is missing images. In fact it’s a bit worse because literally all the images are missing. It’s already game over as far as learning CSS. There’s not much point continuing to read the article. But let’s continue for the sake of comparison.

  

  

  

  Code snippets are looking better than in Pocket. At least the newlines haven’t been removed, but syntax highlighting has, and that has exposed the style hint meant only to inform the syntax highlighter what language is being used (CSS in this case). Readable but looks kind of ugly. An entire article full of these would be a real slog to get through. Syntax highlighting really makes a big difference when trying to comprehend code.

  

  

  

  The listing page for Instapaper looks pretty good, though notice it’s still “downloading”. That never completed by the way, nice though it is to have a progress indicator. I don’t have a screenshot of my Bitcoin article, but the title did get correctly saved. So it seems Instapaper has the better webpage parser, or at least it works better with my website.

  

  

  

  The article page for Instapaper is very similar to that of Pocket’s, though I think I prefer Pocket’s version, less wasted space. But as previously described, Pocket gets the title wrong, at least for my website.

  The overall Instapaper experience is better than I remember but the following issues are still there:

  • Images still missing much of the time
  • Code snippets are there and readable, but look weird, no syntax highlighting
  • You can’t copy and paste article titles on an article page, that’s really annoying
  • When images do get saved, there is no way to zoom, so anything with detail is unreadable

  This time I figured out how to ensure the Instapaper icon always appeared in the Share Sheet. That’s the box with the arrow emerging from it that you tap to share things throughout iOS. I think of it as the share button, no idea why they call it Share Sheet. Previously it was there sometimes, other times it wasn’t. I installed the bookmarklet too, very difficult installation procedure, and I found it didn’t always work. There was a 3rd way to post via email, which might be useful but it wanted access to my contacts and I’ve had bad experiences with that.

  The thing that struck me was how confusing setup was. That’s what got me last time. It never felt like I got to a safe place during setup. Even after completing setup it still felt like there were two many ways to post and when it didn’t work I wasn’t sure if it was me, or the setup, or iOS config. Eventually I gave up.

  I’ve been using Instapaper for a few days now, and for non-web development content it’s actually quite decent. The item download indicator is the biggest improvement over Pocket. The classier look is a big draw too. It’s nice when apps have the right vibe.

  Overall though for reading web development content while offline, none of these apps are adequate. If you are relying on them to learn web development you will be at a profound disadvantage compared to being online. That’s currently a certainty.

• The state of offline reading for web development content

  2023-06-26 07:00:00 +0700 by Mark Smith

  I've been reading quite a few CSS related articles recently, several really interesting ones appeared in both my RSS reader and some of the newsletters that I subscribe to. I tend to skim through these when I'm online saving them to read later when I have more time available. I currently use a combination of Safari Reading Lists and the Pocket iOS app, but I have also tried Instapaper. Overall the experience has been really lousy :(

  When I find time to read the articles later I find 80% of the time one of the following has happened:

  • The article just didn’t get saved
  • The saved article is missing some or all the images
  • The saved article has squashed all the code snippets down to a single line
  • The saved article lists with an irrelevant title

  It’s slightly different in each app. For instance Reading Lists is pretty good with the titles, Pocket often gets it wrong. All the apps have the missing images issue, same with the squashed code problem. I usually also leave the page open in Safari but 9 times out of 10 when I bring the page back into focus Safari insists on trying to reload the page, which of course fails because I’m offline.

  I was trying to read an article by Chris Coyier earlier about Modern CSS which looks to have quite a few things that might be very useful for making my website look a bit more styled. But once again when I sat down later to read the saved article, I got blocked.

  I’ve taken some screenshots to illustrate the problem:

  

  

  It starts out pretty good with both Reading Lists and Pocket displaying something that is faithful to the original page. Though notice that Pocket has decided to save it in Article mode, which is essentially a stripped down version of the page, with most styling removed. That apparently saves storage space, but it also makes all pages you save look the same.

  That’s not a show stopper much of the time, but kind of ruins the vibe. It’s way nicer to see the actual page you were looking at. And in fact sometimes it is a show stopper, especially for web development articles, because authors tend to often reference their own site when describing various techniques they have used - “Check out the header for an example of that”.

  

  

  Things deteriorate quickly though. We get to one of the first major sections, one about Logical Properties & Layout in Reading Lists, and all the images have been disappeared. That's a massive show stopper when you are reading an article about CSS. It’s basically impossible to imagine what a web page looks like just from the textual description. You might get a rough idea, but basically it’s already game over at this point in Reading Lists.

  Switch over to Pocket and it’s a little better, but not by much. The section image has appeared in the right place, but the image of the website that Chris is commenting on has gone AWOL. So it’s basically game over in Pocket at this stage too. There’s some difference between the two apps, so maybe we can just switch between them? I doubt it, but I really would like to learn some CSS. Let’s continue.

  Update: As I write this article, looking back at the Reading Lists version, it appears as though some of the images have loaded, for example the website Chris was commenting on, but many are still missing in the rest of the page. It’s a perfect illustration of the frustrating offline experience, it somehow takes longer to load the images offline than online!

  

  

  Code snippets are looking good in Reading Lists (this time - it’s not always consistent). However in Pocket they’ve been squished down to 1 really long horizontally scrollable line with no syntax highlighting. That’s basically impossible to read and comprehend for anything with moderate complexity.

  

  

  How about the listing pages in these apps? Reading Lists displays Chris’ article well, but notice my recent bitcoin blog post just below it. It says “Not available for offline reading”. That happens on a huge number of sites, like above 50% of the sites I try to save. Again Chris’ site looks good in Pocket but my bitcoin blog post just has my website header as the title. That’s the same for any page on my site, which is obviously a show stopper.

  

  

  As for article pages, you can see above the 'Top of page' screenshots for Chris’ site. Both look good. But my bitcoin article on Reading Lists results in possibly the most annoying popup in offline reading. That's because it happens so darn often, and always results in a sinking feeling of disappointment. Oh no, not again :(

  I could continue with more examples, it’s essentially the same throughout the article. Between the apps there is no way to see screenshots or code snippets. You might get lucky switching between the apps, but probably not, and even if you miraculously did, it would be very difficult to get into flow of the article. It’s much the same for other web development content, such as javascript articles. Reading web development content offline currently is always frustrating and disappointing, if not impossible.

  I've been in touch with the Pocket support team. I brought up the following:

  • Since Webview mode fixes the code snippet squashing I asked if it was possible to set it as default. That's not possible and in any case it turns out the Webview mode requires an internet connection
  • If they had a guide on how to structure a webpage HTML for the content and title to be correctly captured by Pocket. No such guide exists, they said that Article view, which is the mode that removes styling, requires websites to be compatible with their parser. They shared a link to a doc that describes Article mode, but it does't have the necessary information

  I suspect that I need to modify my HTML in some way, perhaps adding meta tags or just better use of HTML5. I'm left having to figure it out myself using trial and error, not a great result.

  Offline needs to be one of the first things we as developers think about. It’s basically progressive enhancement. Not considering it is a recipe for shooting yourself in the foot some time in the future. I’m still looking for a way to have my entire website rendered offline on my iOS device. It happens so often that I need to check something while offline. I read recently about support for webapps on iOS being added, so perhaps I'll look into that at some stage.

• Unfortunately bitcoin alone doesn‘t fix the fundamental problem

  2023-06-24 12:00:00 +0700 by Mark Smith

  Bitcoin and crypto in general are truely marvelous technologies. I've spent the past year or so really going deep to try to understand the fundamental dynamics. Along the way I've learnt a lot about the current financial system, as well as the history of money and trade, and of course thought about the possibilities that these new monies could open up in our future.

  The conclusion that I think I'm reluctantly coming towards is that bitcoin and crypto alone don't solve the fundamental problem, and in fact I think it's very likely they could make it a lot worse. As with a lot of things that are present in so many of the things we do as humans, it's a mixed bag. It could undoubtably make things better in many respects, but at the same time it could make things worse for a lot of people.

  The fundamental problem is that people exploit other people

  I think the fundamental problem, and I say this knowing it's rather vague, but you have to start somewhere. If I were to put it into words it would be something like - people exploit other people. Of course it's much more nuanced and it's a spectrum with well paid, fullfilling and consentual jobs and businesses on one end and really unpleasant things like slavery, entrapment, starvation, thirst and other types of physical and psychological mutilations on the other.

  How people treat each other varies enormously. I've travelled to many places around the world, seen and experienced many cultures, and it's abundantly clear to me that the notion that most people are good, is totally incorrect. In some situations, it's the complete opposite. Now whether it's malice or some sort of emergent group behaviour, or a combination, is perhaps something different entirely, however the fact remains that some people are treated horrendously by others.

  One way to think about it, and admitedly it's a somewhat crude way of approaching a very complex situation, but I think it's useful nonetheless because you can get a sort of macro sense of what might be happening, is to imagine that people either fix or break other people's lives. Again it's a spectrum, from small breakages to things that are more permanent or even terminal. People can be both fixers and breakers, at the same time even. Life is full of contradiction.

  We have to fix much faster than we break

  In this hypothetical world of fixers and breakers it's pretty clear that overall the fixing has to happen faster than the breaking, or else eventually everyone and everything will end up broken. You can look at the world at many scales from countries to individual people's lives, but it's sort of the same result. The reality is that we all live on the same planet shaped boat.

  Current fixing rates are woefully innadequate

  You just have to spend a little while speaking with homeless people, or indeed have a brush with homelessness yourself to realise that the fixing rates are woefully inadequate in most places. Supply outstrips demand 99 to 1, yet there's not enough. Yes but all homeless people are drug addicts you might say. Well there might be some truth to that statement, but the question is what circumstances led to that. There are very bad people out there who benefit enormously from thoroughly breaking other people's lives. We aren't talking about lone individuals either, we are talking groups that cement their places in society by ruthlessly oppressing others.

  There are also many people in need of help that are sober, and even for them it's an almost impossible existence. Guess what looks like addiction: hunger and thirst. In fact it's basically the same thing, humans are in a sense food and water addicts. Thinking all homless people are drug addicts is not only incorrect but it's basically a death sentence to sober homeless people. And one day you too could be a sober homeless person fighting for your right to survive because others broke your life.

  Crypto ups the game by an order of magnitude

  The thing with crypto is that it opens up enormous possibilities for everyone. That's great in an evenly ballanced world, but the reality is that people that break other people's lives whether through malice or misguidely thinking they are doing the opposite, are much more active. Much like the dynamics in open source software, there are just more people trying to find exploits (notice the similarity in language) than there are people writting and fixing the software.

  That's the current state of affairs, but we are entering a period of rapid change. Crypto technologies could enable great things to happen, lets say it's an order of magnitude change, well if the fixing rates remain the same, then things will necessarily get a lot worse for many. And we aren't just talking about homeless people, or people in developing countries. Think about how many people are stuck in jobs or relationships they want out of. This issue cuts across all parts of the world and through all parts of society. I don't see any evidence that fixing rates got any better in the previous big monetary system change. Moving from gold to fiat money made a lot of things possible, but I would argue that fixing rates didn't increase proportionately.

  AI will up the game even more

  Remember also that this is happening at the same time as the mass introduction of powerful AI technologies that can analize our collective behaviours in mindboggling detail. That's at least an additional order of magnitude in the technological power available. Oh and guess what, after that it's quantum computing. Yet another order of magnitude.

  AI financing is the subject of this week's newsletter, the part about cost curves and training is particulary relevant. Out a bit later today @ 12:45 +07:00:

  The Economics of Catching the AI Wave (Issue #120)

  In a different world

  In a different world, bitcoin and crypto could be truely amazing. Yesterday I had the realisation that if a small amount of people are controlling the supply of fiat money, the equivalent to controlling water spigots, literally starving and thirsting vaste swathes of people, then why not create new monies, new water supplies? Surely it's the moral thing to do.

  I imagined a world where there were a multitude of crypto currencies, for different loosely affiliated groups. Coins for scientists, coins for musicians, for people into a multitude of fringe and mainstream hobbies. Initially it sounds like it would be too complicated, but with a bit of imagination it needn't be. You would be able to effortlessly spend all of these from your crypto wallet because all shops would be crypto exchanges, and your wallet would be smart enough, because of AI, to always have enough for what you needed or wanted to do. With integrations with other AI software that would help you manage other aspects of your life, it's not unfathomable to think that it could 'just work'.

  And by the way, it really is a small amount of people controlling the spigots, cash money is being created at 7 times the rate than actual humans are being created. That's an indication to the current ridiculous innequality throughout the world. It's so mindboggling that it's difficult to fully comprehend. It doesn't have to be that way though. Things could be a lot better for everyone.

  And that's why I'm so conflicted on crypto. I return to the main point of the article. These amazing bitcoin and crypto technologies simply are not enough. IMO the cultural make up of our societies is in a state that could make crypto adoption very dangerous long term, where a class of folks, that remain somewhat in the shadows, have enormous power by ensuring breakages in peoples lives happen faster than they get fixed.

  That's the core dynamic that needs to be fixed. Maybe Fiat is the worste money, but until the breakages problem is handled, perhaps it's only the worste money except for all the others.

• The return of number go up

  2023-06-22 06:55:00 +0700 by Mark Smith

  From the linkblog 2023/06/22:

  Bitcoin Breaks $28K for the First Time Since May - A flurry of activity in the crypto space has raised the price above $28k, up from 22k in just a few days. It appears to be in response to the world's largest asset management firm Blackrock's announcement of the first Bitcoin ETF, which would open up the possibility of investing to many that previously would not have been able to. Previous to this, it had been a slow and steady decline in price, which appeared to be following the general sentiment that the economy is heading into a deflationary period. This comes at the same time as US regulators have been charging big crypto exchanges for handling unregistered securities, and prominent TradFi figure John Stark has been straightup advising folks to 'get out of crypto' because of the storm brewing and major podcasters in the space are predicting big regulatory headwinds. Nevertheless at least for now, it's the return of number go up.

  I thought the graphs were pretty interesting given all the background. Here they are:

  

  

  

  

  

• Fragments of Capital FX

  2023-06-16 14:00:00 +0700 by Mark Smith

  If you have been reading the blog or newsletter for a while, you'll know that I'm a big fan of podcasting. I've been listening to podcasts since the medium started in the early 2000s. I'm always reading about the developments in the industry with a curious eye. A recent pivot by Spotify that I read about really reminded me of my time working in the VFX industry, there are a lot of similarities in how that unfolded.

  With a hint of nostalgia in my sails, I went looking for Capital FX, my old company, online and was suprised to find there weren't that many traces remaining. This post is a compilation of some of the fragments I found during my search.

  It's not that suprising that there wasn't much remaining. It was a different time back then. The internet wasn't so ubiquitus. Sure we had the web, but mostly that just meant being able to send email, looking up company addresses and perhaps reading the news on the BBC website every now and then. Social media wasn't a thing yet. Facebook was only just starting to get popular at about that time. The idea of having an internet presence was totally foreign to basically everyone. Why would you want to tell everyone about the cash cow you had built? To most normal people that actually seemed like a rather stupid idea. You would occasionally put out press releases, but many of the publications that published those have since gone offline.

  First a bit of background info. Capital FX was a feature film post production house specialising in special effects, digital intermediate, and later, digital cinema. I worked there from around 2005, building out the infrastructure as the company grew rapidly in size, taking on increasingly large high profile Hollywood blockbuster projects. The guys that started the company were, in a lot of ways, film production pioneers. They provided various services to film makers, most important and lucrative of which was foreign language versioning. They started small but quickly became central in London's cottage feature film industry.

  Capital FX was acquired by Deluxe in 2006, and I continued to work there until 2008. It was a very interesting time to be there, because the entire industry was transforming to fully digital pipelines, from capture to theatre, but we were also dealing with movies shot on old school film. We worked on hundreds of movies, some of which are listed in IMDB. The aquisition was particulary interesting because we suddenly had access to other Deluxe properties including world famous digital intermediate EFilm. We worked closely together to build precission transatlantic production workflows that were used on films such as Children of Men, and The Golden Compass. Big name directors liked working with us because we had the most advanced equipment in the industry, and we knew how to get large complex jobs done on time at very high quality.

  So what remains of Capital FX? Well the website is long gone but I was able to recover the company logo from the internet archive.

  

  This is perhaps my favorite fragment of all. Look at that beautiful cubist and somewhat brutalist combination of small and big letters. And only 1 color! Imagine the confidence that this little photoshop fill tool special gem inspired as it zipped by on reels being watched on a Steenbeck, or indeed on expensive 4K digital projectors in blacked out colour grading theatres. All the goodie bags we gave to clients had this printed on the side of them. Wonderful. Such memories.

  Before moving on to the rest of the fragments in our VFX archeological digg, let's take a short detour to learn about How London became the VFX capital of the world. I think it's important to set the scene a little. Ok now that you are feeling the vibe, lets look at the rest of the pieces that I was able to recover. I have also included links to internet archived versions of these pages, denoted by [ia].

  Company info listings

  DSG DIGITAL LONDON LIMITED (Companies House Listing - The Company) [ia] - You can see it was previously called Capital FX before the acquisition.

  DSG DIGITAL LONDON LIMITED (Companies House Listing - People) [ia] - Some of these folks were the big cheeses so to speak, also features names of folks I used to hear about all the time but didn't actually meet because they weren't actively envolved anymore.

  With Capital FX [ia] (IMDb - Sorted by Popularity Ascending) - This one gives you a sense for all the awesome movies we worked on. There were a lot more that aren't listed here. Turns out it's quite difficult to get in the credits, especially when the company that acquires you, in our case that was Deluxe, literally has it's logo at the bottom of about 1/2 of all films. The other 1/2 by the way have rival company Technicolor's logo at the end of the credits.

  Capital FX, London (IMDbPro - Client & Contact Info) [ia]

  Capital FX (London) (BFI - British Film Institute) [ia]

  Capital FX Ltd - CFX - London (4rfv) [ia]

  Staff press releases

  Shelton, Clarke join London's Capital FX (Screen News) [ia] - We had many people join us as the company grew, it was a very cosmopolitan environment, folks from all over the world.

  Tech press releases

  Capital FX expands Digital Intermediate Services (TV Technology) [ia] - We built a for hire digital intermediate facility that directors would hire out during their production. We developed workflows where footage was being shot in LA and graded the next day in London, all sent digitally over the internet. The color science to get it all looking the same in all locations was quite complicated.

  SAN Speeds Capital FX’s DI process (Infostor) [ia] - So much storage, everytime we setup new petabtyes of storage it would get gobbled up almost immediately.

  Capital FX Speeds 2K and 4K Workflow on Major Hollywood Movies Using SGI InfiniteStorage Technology (Markets Insider) [ia] - We had a lot of SGI equipment, very fast very large storage for editing at high resolution. This article has probably the best description of the digital pipeline we built. Such a lot of equipment all pieced together in a few short years all while using the infrastructure at near capacity. Not only did we upgrade the plane's engine while flying the plane, so to speak, but no one had ever really built such an engine before. Using this kind of technology at such scale in the creative industries just hadn't been done before.

  Lustre Digital Color Grading System Shines Worldwide (Animation World Network) [ia] - All the color grading software and hardware cost enormous amounts of money.

  UK's CAPITAL FX BUYS THIRD FURY FILM RECORDER (Post Magazine) [ia] - We also had many Arrilaser recorders, and of course an Arriscanner which could scan between 4-8 frames per second.

  Doremi DCI mastering and playback purchased by Deluxe Digital Cinema (UK Broadcast News - 16/05/2006) [ia] - All the d-cinema stuff was super high security, everything encrypted, very expensive pieces of kit, often required custom workflow integrations.

  Case studies

  Case study: Archiving the blockbusters at Deluxe [ia] - The tape machines were awesome, basically a giant robot jukebox. So many gigabytes of storage.

  That's all the fragments I found, there's likely more out there. I remember looking about 4-5 years ago and there was a lot more. So many things on the internet eventually disapear, that's why I wanted to capture a bit of a snapshot now of Capital FX, the pioneers of digital intermediate, visual effects and digital cinema. It was tremendously fun working there!

• Organising my RSS feeds

  2023-06-02 14:00:00 +0700 by Mark Smith

  One of the items that's been on my todo list for a couple of weeks now is to organise my RSS feeds. I've been tidying them up best I can in short bursts here and there, but I'm struggling, feels like I'm walking up an escalator that's going the other way. It never feels like I'm making much progress.

  Feedly has been my app of choice for many years now. The interface is pretty good, there are some nice features. I have the iOS app, and it works quite well, but the one thing it doesn't do well, or at all, is offline. When I'm offline, the app opens but it's just an empty screen, no way to do anything. I find the best time to catchup on RSS feeds is when offline so that's not very practical.

  I had a bit of a look for some feed readers in the App Store and found one called Newsify. It's quite good, even offers to login using my feedly account so I was up an running quite quickly, no need to export and re-import feeds. I configured it so the articles appear in a compact way, and the reading experience is relatively good.

  All the articles get re-formatted so they all look the same no matter which website you are reading. That's sort of annoying. I know it probably saves a bunch of bandwidth, but I really like to read from the website in question. Striping out all the styling kind of ruins the vibe.

  Couldn't there be an option to fetch the actual published web page and store it similar to how Safari Reading Lists work? Surely in this day and age that would be possible. Looks like I'm able to re-organise my feeds in Newsify, though I haven't tried that yet.

  I think though what I'd like to be able todo is to manage my feeds in the same way I manage my blog, linkblog, podcast and newsletter data. I just want to add a feeds folder in my data repo and have a markdown file for each feed. Then I could build one or several OPML subscription list files which I would host on my website. I'd like to be able to import that into whichever feed reader I was using. That way I can easily try out new readers. Maybe even try using the feeds in some app I could build myself sometime in the future.

  The biggest show stoppers:

  1. Feed readers only seem to support OPML files for bulk import, not for dynamically getting the list of feeds
  2. The OPML file would need to be public, or if basic auth could be supported by the reader app, how do I add authentication to the static files I host on Netlify?

  Maybe these are solvable issues, but figured I'd write a blog post about it. If you know any solutions that could help, please email me, I'll write up the setup if I manage to get it working :)

  Btw, here's an example OPML file that lists the feeds on this website.

• When will AI start to affect macro?

  2023-06-01 08:30:00 +0700 by Mark Smith

  The world of AI is moving at such a huge pace at the minute and it seems to be accelerating. ChatGPT started it all off with it's chat based interface, and we were all blown away with what it was able to do.

  Shortly after that people started to create multi-step versions of these types of applications that can use external tools to retrieve and manipulate data. Some of the first use cases for these AutoGPTs are code assistants but they can be used for any task. These AutoGPTs are incredibly powerful, orders of magnitude more powerful than their predecessors.

  There are already comprehensive new javascript libraries with which you can build much more than just chat apps. Entire advertising campaigns that previously would have taken months to construct can now be created in minutes. And there's a lot lot more.

  And the money is flowing in that direction too. There have been some reports that important venture funds are repositioning away from crypto to so called frontier tech. In fact blockchains and AI are a very good match, with the ability to autonomously spin up wallets and perform very complex chains of activity.

  There's also the recent annoucement that Elon Musk's company Neuralink is going to start clinical trials of human computer interactions via brain implants. It might be a good time to seriously consider our right to mental privacy in the age of brain sensing tech, because when you combine some of these technologies, say with something like edge computing, the possibilities for creating unpleasant adversarial environments are very real. Nice income streams you have there knowledge worker, it would be ashame if you were suddenly unable to function. The future is here, and it's going to be crazy weird.

  With all that in mind, it's also important to step back and ask how this will affect the wider world. Something that worries me is escalation. We know things in the economy already escalate. To a large extent it's always been this way, you literally can see it in the data. But with AI the occurrences of dynamics of escalation are going to increase significantly. These aren't always bad per say, but I believe they can be, and it could get a lot worse so to speak.

  We should start collecting key metrics so we know how things used to be before AI, what the world looked like, what it felt like to live in. We need to be able to see if and how things are being affected at a macro level. Wouldn't that be prudent?

  We've also got quantum computing on the horizon, IBM plans to have a 100000 qubit machine operational with 10 years. What will happen when we add that to the mix? Surely it's going to be another step shift.

  Big leaps are also being made in nanotechnology, where tiny robots can be sent into the body to deliver medicine in an incredibly targetted way. All very amazing, but the fact is they could also very well be used to deliver harm to people too. We will need to be able to detect things at that level too. Vulnerable populations like the homeless are already targetted with foreign material placed in their food. It could get a lot worse, and when it does, it could happen at scale and very quickly.

  Let’s boldly go into the future and explore the possibilities of these new technologies but let’s avoid a multi story car park collapse of humanities’ cultures. Let’s make sure some of the worste outcomes are avoided with a bit of structure in important areas. Remember ultimately we are all in the same earth shaped boat.

  What type of data would we need to collect to get some visibility without destroying the very culture and creativity we are trying to protect in the first place?

• What makes culture so fascinating?

  2023-05-28 08:30:00 +0700 by Mark Smith

  I’ve always been fascinated by culture. I’m not sure what I find so appealing about the whole idea. Personally I've always been more attracted to fringe and subcultures, things that are a bit odd, outside of the mainstream. I wonder what it is exactly that I find so fascinating.

  There is something magical and mysterious about culture, but also re-assuringly imprecise, like drifting clouds or bubbles in an enormous lava lamp. You can interact with it. It’s a highway for your imagination, but it’s massively multiplayer.

  You don’t even need an internet connection, it’s an infinite World Wide Web with no boundaries, hosted entirely in our brains, distributed peer-to-peer, no wires or transmitters required.

  It’s like invisible super high level software, it doesn’t exist, but it does.

• Digital media companies are going out of business

  2023-05-19 05:53:00 +07:00 by Mark Smith

  There's been a bit of a collective retrospective happening the past few weeks, brought about in part by the publishing of a new book that details the goings on during what is being called "the golden era of digital media". I feel like it's a bit hype, but since that period was quite formative in my introduction to the online world, I've been somewhat swept away with the whole story. It's been fun reading and learning more about what was happening, from folks that were closer to the eye of the storm.

  Brian McCullough has an interesting recent podcast episode interviewing the author Ben Smith.

  • Golden age of digital media companies (Techmeme Ride Home Podcast)

  This was the period, starting around 2003, where people were using the web to redefine journalism and reporting. News sites such as Gawker, GigaOm, ReadWrite Web, Huffington Post, Politico, Digg, Buzzfeed, The Verge, Fark and Drudge appeared and brought with them a refreshing approach to news making that was more direct and unnencumbured by main stream media conventions, but also that specifically aimed to have online as the target medium.

  It was more interactive than mainstream media, with the audience often influencing the coverage as it unfolded via web forums and comment sections. It also incorporated meme sites such as cheezeburger and an endless supply of gifs. It was the era that social media sites such as Facebook, Twitter, Stumble Upon, Pinterest, Snap, and Tumblr were gaining promenance. It was the era of APIs and web2.0. New tech, new people, and fresh ideas, all being experimented with to create new culture.

  For me that whole period was characterised by the rise of videoblogging, with shows such as Rocketboom, Ze Frank and The Idea Channel. I was also getting into tech, programming and open source with podcasts like the Gilmore Gang, IT Conversations and of course the Daily Source Code. It felt like we were finally breaking free from mainstream media, like we were finding a new, more authentic way, building the path as we went along it. For the first time it felt like a "We", rather than just passively consuming what we were being fed by the media makers. It felt like anyone could participate.

  The retrospective has also been brought to the fore by the bankruptcies, closing downs and re-structurings of several digital media companies that were once hailed as the next big thing in online news, politics and culture.

  Again some good coverage on the Techmeme Ride Home:

  Motherboard, the tech vertical from Vice has declared bankruptcy, and will sell assets for $225 million. At it’s peak Vice was valued at $5.6 billion, the money faucets were wide open.

  Buzzfeed News was shutdown, with the reporters there publishing the Definitive oral history of BuzzFeed News 2012–2023 as they walked out the doors, which makes for interesting reading.

  One of my favorite quotes is from Tom Warren, investigative reporter:

  Despite the warnings of more experienced journalists that I would be mad to work at BuzzFeed, I applied as soon as they opened an investigations unit. All we wanted to do was blow the fucking doors off…and guess what happens when you give a group of twentysomethings unlimited resources to chase stories? They go batshit crazy. Absolutely fucking bananas.

  We flew round the world chasing tips, conducted surveillance ops on money launderers, took sources out for mindblowingly expensive meals in the hope of getting leaked documents, rode through rainforests on the back of motorbikes, and engaged in a host of other wild adventures, not all of which are fit to print. There were wild parties, lots of them. One week you would be at a warehouse rave in Brooklyn, and two weeks later you would be flying into Istanbul for security training and heading for a night out with whichever members of the press corps were in town. We had the wind at our backs and had a beast of a fucking time. It was majestic.

  Which paints a very fear and loathing in las vegas, Anthony Bourdain-esque picture. I didn't realise things were so crazy there.

  These were part of a small set of digital media companies that were going to be the next big thing as they catered for millennial audiences. Now that attempt at building the future of media seems to be wrecked and washing up onto the beach in small pieces. Honestly I feel a bit sad about the whole thing.

  I'm left wondering what's going to happen next? Is it the end of an era, or the beginning of something even better?

• Money liquidity and hunger

  2023-05-12 11:54:00 +07:00 by Mark Smith

  Here’s a calculation / research I’d like to see done. What follows is very macro and hand wavy but hopefully you’ll see what I’m trying to understand. I want to get a better sense for how close to capacity the money system is running in relation to the most essential thing for humans: food.

  We have the technology to make food on an enormous scale, there’s more than enough for everyone, but money isn’t making it to lots and lots of people, and so neither is food. Many people are starving and thirsty, and that’s unacceptable.

  Economic conditions are causing people and institutions to hoard making the situation worse. Forget for a second the exact mechanism that money gets into the hands of people, imagine that’s solvable. Forget all the difficult social issues, just focus on the numbers. The question that needs answering broadly speaking is how much money liquidity is needed in the system to feed every single person per day/month/year?

  Base it on current prices in your location in regular shops people use. Don’t get overly complicated, it’s just an estimate, it doesn’t need to be super optimised. Just price out 3 healthy microwave meals per person per day. You just want to have the minimum worst case. Is that amount, larger or smaller than current total money iquidity? Is it even possible? Remember to multiply by the total population of earth, I guess that's currently around 7 billion.

  It should then be obvious how much people and institutions can hoard before starvation shortages occur. The only way to make things sustainable is to manage this pool of money. In many ways money is like water, it’s not a coincidence it’s called liquidity. It’s a resource we all share.

  I suspect that the total money is way way way more than the required liquidity to feed everyone. How we have things organised currently clearly isn’t working very well. There are a few people standing around the money spigots and they aren’t letting the water get to people that desperately need it.

  When you know the answers to these questions, then it should be easier to figure out how to get that money into the hands of people, whether it’s jobs, UBI or something else, first figure out what’s possible, set targets and make a solution that fits.

  When hunger is solved, many of the world’s problems start to disappear as they are often self inflicted, because hunger is used as a weapon of control, but really it’s a massive escalating humanity foot gun that will at some point affect us all.

  Bonus question: how do you ensure evil people don't take such an estimate and re-engineer the money system so there is always not quite enough to feed everyone?

• Reading through the Bluesky protocols

  2023-05-06 08:47:00 +07:00 by Mark Smith

  There’s been a huge amount of activity in the alternative social medias space the past few weeks. Whereas the focus in social media for the past 10 years has been on centralised systems such as Twitter, Instagram, Facebook etc, the current focus is towards decentralised systems. Many have been expecting and waiting for this change for a while now, and they have been building open protocols that would enable such a transition to become a reality.

  I've been trying out some of these new decentralised Twitter alternatives and related open protocols. First I tried out Nostr, with mixed results, though I remain intrigued by it’s possibilities, and I very much like it’s simplicity. Then I took Scuttlebutt for a spin, and that was ultimately very disappointing for my situation, but I could see how it could be quite cool if you already had a group of people. I also did somewhat of a deep dive into ActivityPub, a social media protocol started a number of years ago, that has gained a lot of traction recently. In some ways it feels a bit old fashioned in its design, but you can actually build some very interesting things on top of it.

  The latest hot decentralised social network to blow up is Bkuesky. It was started what feels like a couple of years ago and got a lot of attention since it was being led by the Twitter founder Jack Dorsey. At the time his vision was for Twitter to be powered in some way by the new protocol. The project sort of faded into the background but resurfaced in the past month, and it appears they have been very busy designing and building.

  It’s still invite only, and sadly I don’t have an invite yet, though I have added my name to the waiting list. The hype cycle is in full force, with loads of articles being written about people’s initial experiences, and it sounds very much like how I remember the first few years of Twitter to be like. Other people agree with the sentiment. Oh goodness maybe the good old days are back, this time with hellthreads and people being very silly. Hellthreads, fun in social media, not so fun in real life. Let's please remember the difference.

  With no access to the network I decided to have a read through the protocol docs, to get a sense for what the foundations look like. A good place to start is this blog post because it highlights some of the key high level design goals:

  • A Self-Authenticating Social Protocol

  I note that they started out doing an ecosystem review, looking at existing solutions including ActivityPub and SSB for social, Matrix and IRC for chat, and RSS for blogging. That’s essentially what I’ve been doing on a smaller scale! Great minds… :)

  The high level mission:

  Bluesky’s mission is to drive the evolution from platforms to protocols

  Why would I be interested in all this? What are they trying to build?

  An open and durable decentralized protocol for public conversations can allow users a choice in their experience, creators control over their relationships with their audience, and developers freedom to innovate without permission from a platform.

  Some the features and goals that stood out to me:

  We want users to have an easy path to switching servers, even without the server’s help.

  We want to give users the option to participate in global conversations the way they do on large social networking platforms.

  Make global conversations possible while preserving the freedoms users gain from interacting through an open protocol.

  Build around trust from the start by exposing what’s going on under the hood and allowing users to adjust their experience.

  Work towards a transparent and verifiable system from the bottom up by building a network that is open by default. We’ll do this by giving users ways to audit the performance of services and the ability to switch if they are dissatisfied.

  All very great. We are talking about systems that will mediate our day to day interactions, so it’s important to get the high level stuff right. This definitely feels like it’s in the right direction to me.

  On a more technical level, the whole system is built around the idea of self-authentication, the key pillars of which are:

  The three components that enable self-authentication are cryptographic identifiers, content-addressed data, and verifiable computation.

  With the pre-amble out of the way, I moved on to the protocol, or to be more precise, the collection of protocols. There’s an overview that lists the various relevant documents and various guides that look at concepts in more depth. I spent what was probably about an hour reading through everything.

  My first impressions:

  • It’s quite short
  • Very easy to read, it’s not a bullet point heavy W3C hellspec
  • Quite a lot of new concepts and entities but they are very clear and differentiated
  • Initially I was a bit put off by how many documents there were, but actually the way things are divided up makes sense
  • Feels like it’s been well thought through
  • Entities are loosely coupled to avoid lock-in
  • Lots of different types of ids, but users mostly probably won’t need to interact with these
  • Compared to the others I’ve read, it feels the most professional and well put together
  • I like that you can easily drop down into the technical detail and see how things are strung together

  As I read through these docs, I found myself nodding my head quite a bit, and was pleasantly surprised by some of the elegant design. Whereas some of the other protocols in this space feel a bit disorganised, convoluted and like they were mostly designed on paper napkins, Bluesky feels like a design that is more towards the Cisco side of the technical design spectrum. I am overstating things here somewhat. The other protocols aren’t paper napkin protocols and the Bluesky AT protocol isn’t a Cisco concoction. Neither of those characterisations would be desirable for social media. But broadly that’s the sense I get.

  I like it. I like it quite a lot. I can envisage it being very cool. I can imagine many interesting tools and services being built on it. It could be amazing. Having said that, I can also see possible dangers. There are a lot of new things introduced, and it might take a while to really iron out the wrinkles.

  We will have to remain vigilant to be sure we haven’t done something dumb. How these protocols get used in practice might be unexpected. I think it will be important to have a diversity of protocols going forward. Keep using and improving the simpler protocols ‘just in case’. Remembering that it took close to 10 years to figure out that entirely centralised systems weren’t desirable. It could take the same amount of time or longer to really get to grips with modern decentralised social media.

  I’m looking forward to being able to build and participate.

• Scuttlebutt and Manyverse first impressions

  2023-05-04 14:31:00 +07:00 by Mark Smith

  I've been exploring Twitter alternatives the past few weeks. Ideally I'd like to setup server instances for each of these, and write some rudimentary clients to post some messages into these networks. However I'm severaly resource and time constrained at the minute, so that's just currentky not possible. :(

  I figured I could at least try what's out there and write about it. A few weeks back I wrote about my experience setting up Nostr and Damus. That ended up in me having to put a pin in the whole affair, but I have been thinking and reading more about Nostr since. I haven't openned the app again though.

  So I also took a look at Scuttlebutt and it's accompanying app Manyverse. The first thing that jumps out at you about this project is the amazingly friendly looking website, cool back story and cool tech.

  Scuttlebutt can be transformative for society, decentralizing and enabling local community development free of big corp. It is a fast growing decentralized social network. As an alternative to the large corporate social networks it enables autonomy for the users and a free zone from big data harvesting

  There's a bit of a hippie vibe, which is kind of cool. They have some videos but my internet connection wasn't good enough to watch them.

  Based on the description on the website, it's great for situations where you are offline a lot, you don't even need a network to sync up with others, it can be direct device to device, or through online room servers or relays. The founder apparently lives on a boat.

  The getting started guide looks quite straight forward, though there is a 'get an invite code' step. First warning sign. Anyhow, forget about that for now. Choose a client and get that setup. I chose the iOS Manyverse app.

  The Manyverse iOS app installed without any issues. It's a great looking app, appears to have all the functionality you would expect. It's clear and obvious where everything is, feels very neat and organised. Only problem is, it's very empty of people, and by empty of people, I mean like not even a single other person. It was literally just me.

  I setup my profile as best I could and wrote a few short posts. The cool thing is that you can do this totally offline, and sync later. At least I think that's the idea. The main issue is the 'invite code' step. How do I do that?

  So I read the docs some more and click around. I do some google searches. I also notice a place in the app to connect to servers. Eventually I find a list of servers. These all seem very small scale. After successfully connecting to a few different servers, it appears to be a total ghost town. Digital tumbleweed is passing me by. Is this project still alive?

  My patience has wained. My back hurts from standing, an unfortunate feature of my current internet connection. After around 30 mins of fussing around, I still can't figure out how to connect to anyone.

  Several weeks previous to this, I had spent a while reading through the developer docs, and I was very impressed. It looks like it could be a very robust protocol. I imagine that if you are already a group and need to keep in touch in a decentralised async way, this would be awesome. As a new solo user though, I feel like there was such a lot of promise but ultimately it was very disappointing because I couldn't connect with anyone at all.

• Oauth shows open protocols often get messy

  2023-04-27 18:11:00 +07:00 by Mark Smith

  Yesterday I wrote a commentary and writeup piece about ActivityPub, one of the contender protocols that has emerged in the federated social media space. A hot topic at the minute as many creators are looking for Twitter alternatives. What I discovered is that the visions of these protocols often sound wonderful, but the realities when you actually try to build something with them aren't always smooth sailing.

  That's not to say that it isn't worth the effort, open solutions do offer fantastic long term benefits, but the road is invariably long and bumpy. With that in mind, it's useful to look at other open protocols just to get a sense of the types of difficulties involved in creating such a thing. One such open protocol is OAuth which has been used for authentication these past 10 years on pretty much all major APIs.

  Robin Guideber writes:

  The real-world OAuth experience is comparable to JavaScript browser APIs in 2008. There’s a general consensus on how things should be done, but in reality every API has its own interpretation of the standard, implementation quirks, and nonstandard behaviors and extensions. The result: footguns behind every corner.

  Guideber would know, he's apparently been involved in building the authentication for many large APIs including amoung others Google (Gmail, Calendar, Sheets etc.), HubSpot, Shopify, Salesforce, Stripe, Jira, Slack, Microsoft (Azure, Outlook, OneDrive), LinkedIn, Facebook.

  His article is ultimately a sales pitch for a product he's working on, however the piece is interesting nonetheless because it highlights very detailed real world examples of how these protocols often get stretched and used in different ways.

  • Why is OAuth still hard in 2023?

  ActivityPub, which everyone is talking about at the minute, feels somewhat complicated to me. I'm interested to see whether Nostr gets much traction mainly because it's much simpler, but it too has issues. I wrote about some of these last week in a writeup piece about my experience setting up the Damus iOS app, which is a Nostr client.

  Hopefully this gives a bit of perspective on open protocols. They come in all shapes and sizes, and adoption is hard to predict. It's clearly worth it in some cases. The world would be very different without, to name just a few, pop3 and iMap for email and TCP/IP, HTTP and FTP for the web.

• What’s ActivityPub like really?

  2023-04-26 14:14:00 +07:00 by Mark Smith

  David Pierce from the Verge has an article out about ActivityPub and the future of social media protocols. It’s a great piece, well researched and really sells the decentralised social networks vision.

  • ActivityPub is the next big thing in social networks

  The crux of his pitch:

  If our current social system was decentralized, you’d be able to post a picture on Instagram and I could see it and comment on it in the Twitter app. Your friends could read your tweets in their TikTok app. I could exclusively use Tumblr, and you could read all my posts in Telegram. Different apps would have different strengths and weaknesses, different moderation policies and creator tools, but you’d have the same set of followers and follow the same accounts no matter which platform you use

  Sounds pretty cool.

  So what is ActivityPub?

  It’s a technology through which social networks can be made interoperable, connecting everything to a single social graph and content-sharing system. It’s an old standard based on even older ideas about a fundamentally different structure for social networking, one that’s much more like email or old-school web chat than any of the platforms we use now. It’s governed by open protocols, not closed platforms. It aims to give control back to users and to make sure that the social web is bigger than any single company

  This new world would be very different, better in many ways:

  A couple of dominant platforms would be replaced by millions of interoperable smaller ones

  and

  No one will be tied to a platform and any platform that tries to tie down users will lose them instead

  Great I love the vision. Sounds amazing. It’s worth reading the whole article to get the full picture.

  After reading that I wondered how difficult would it be to post a link from my linkblog into the ActivityPub. At which point I realised I’d brushed up against the first hurdle, because one does not simply “post to the ActivityPub”. It’s not a website, it’s a protocol. First you have to find a website that supports ActivityPub and then you post to there using the ActivityPub protocol. Fine it’s not as simple as just using the Twitter API, but how complicated can it be?

  The first thing I did was check npm for ActivityPub nodejs modules. Lots of search results, but they are almost all for creating your own ActivityPub server. The activitypub-express module indeed looks very comprehensive. It appears to be quite straight forward to add ActivityPub routes and functionality to an express server. That’s great but way more than what I want to do, probably would need to hook up a database. I just want to post a link mate.

  Next step was to do a Google search to find the specification, and some tutorials. The spec shows up immediately, but one glance at that will have most developers running for the exits. No one reads the W3C specifications. Fuck that shit.

  But look there’s an ActivityPub Rocks! website. Hey that reminds me of when HTML5 got released. They had cool HTML5 Rocks websites for that too. This must be what I’m looking for. Click…page loads…uh…there must be some kind of mistake. This website most definitely does not rock. There’s nothing on there about posting a link. It’s all about implementing servers.

  I then decide to look at existing implementations. I find that Mastondon has a tutorial about posting a message, which looks somewhat doable but it’s got Ruby code in it. Yeah I know, I should be able to replace it with nodejs but I already don’t understand a lot of the terminology, and my brain is already going yuk, especially after skimming through the horror that was the W3C spec earlier. And why do the JSON examples have property names that have differing font sizes. That’s not normal. What is this JSONish text blob? I’ll come back to this.

  A bit more googling and I find this article which is a bit rambley and very hodge podge, but is reassuringly clean and nicely formatted black text on white background. It basically points out which are the important parts of the spec, and which parts can be safely skipped.

  • A highly opinionated guide to learning about ActivityPub

  That was actually quite useful with a few good examples.

  Ok so ActivityPub uses a special type of JSON called JSON-LD that includes URLs where you fetch sub parts of an object from different locations on the web. Sounds interesting but potentially complicated.

  Anyway after reading the full article I have a better idea of the vocabulary being used, the various entities etc. Still feels very complicated. Plus it appears to take the form very similar to email where users have inboxes and outboxes. Which is weird because haven’t we spent the last 10 years trying to specifically get away from email?

  I go back and re-read the mastodon tutorial, which makes a lot more sense now. Basically just some HTTP post requests with JSON payloads, and some sort of complicated crypto signatures. Feels quite doable but also I’m feeling like the reality of ActivityPub is very scattered and messy. I’m certainly not super eager to dive in. It has all the signs of an ecosystem littered with cracks and crevices, though there are a few large implementations so it likely does work.

  The vision sounds amazing, but existing developer resources are a bit underwhelming and disorganised. I want to like ActivityPub and suspect that I probably would end up liking it, but I don’t have the time or resources to get involved, I’ll have to put a pin in this for now.

  Update: this just dropped in my podcatcher as I was publishibg this blog post. I havent had a chance to download and listen to it yet but their shows are usually very good quality and sometimes include suprisingly good opto-electronic based AI generated rap:

  • ActivityPub is the next big thing in social (Vergecast Podcast)

• Materials Science is pretty cool actually

  2023-04-24 10:26:00 +07:00 by Mark Smith

  Tyler Cowan published a great interview with Jessica Wade, who is a researcher at Imperial College, London. Jessica is doing some really fascinating research using incredible tools that enable you to peer into the nano world at a molecular level. Debugging the very fabric of reality.

  • Jessica Wade on Chiral Materials, Open Knowledge and representation in STEM (Conversations with Tyler Podcast)

  I did my Materials Science & Engineering Masters at Imperial. It’s a long time ago now, in the mid 90s. I wonder how things have changed. Back then the Materials department was part of the Royal School of Mines, along with other disciplines like Geology. It was a bit of an odd mixture of the really old school with the very futuristic. I did some Raman spectroscopy myself as well as electron microscopy and IR spectroscopy. Very large and expensive equipment. Many fond memories, though writing dissertations was a bit of a slog.

  I chose Materials because it had an interesting balance of maths, physics and chemistry. It had a bit of a reputation for being somewhat less hardcore than say mechanical or electrical engineering, but there was something kind of cool about it. That’s what I thought anyway. Turns out that much of the interesting parts of things in the real world actually centre around materials science. It was becoming true back then, but it’s even more true as we transition to a digital world.

  The difficulty for me back then was that it was very hard to see how anything I was learning would translate into the real world. After I graduated, I interviewed for some materials engineering jobs that sounded great on paper, but in practical terms, it would have been 8 hours a day in sterile labs looking through microscopes. I had absolutely nothing in common with the people I met at these companies, nice though they were.

  Everything these companies did felt so far removed from how their products were used in the real world. After a tour of the labs, some of the lab coated employees there asked me if I had any questions, and I simply had nothing to ask them. Total blank. I was completely disinterested in what they were doing. It just wasn’t cool in any way. Even though it was futuristic everything somehow felt really archaic. There was no energy, enthusiasm or purpose. It felt empty.

  The jobs were also always in small towns, far from the capital, and I just couldn’t envisage not being in London. These days the idea of living in a small town seems kind of nice to me, but in my 20s, it felt like some form of suicide. What would I do at the weekends for heavens sake?

  After a few years working in gastropubs, an interesting cultural detour in itself, I ended up going back to university to do a Masters in Computer Science at University College London. That was awesome. I read Levi’s Hackers book, and started to get into the whole history of Silicon Valley, entrepreneurs, open source & the free software movement. Now this felt like something was going on. These were the companies building on top of the materials science tech. It felt like this was were the future was at.

  Looking back it’s not so surprising to me that I went in this direction. I’d spent years following bands, going to live gigs, being immersed in mythologies, stories and fashions. In cultural scenes. That’s where all the action was. I was looking for something similar but in technology. As it happened, technology had its own version of this, but much of that had been happening in software rather than hardware. And at the turn of the millennium things were starting to get very interesting because the world wide web was starting to really catch on, and companies like MySpace and later Facebook and Twitter were taking off. Tech was beginning to enter the mainstream cultural space.

  These days I wonder what the path is like for Material Science graduates. Educational wise it feels to me that there are so many more resources that connect the science to the real world. Independent media being produced by scientists and shared in places like YouTube, Wikipedia and podcasts. I love consuming all this content. I’m fascinated by it. Plus software has become pervasive basically everywhere both in industry and academia. Open source development is the norm now. Everyone codes.

  Would I have stayed in science if that would have been available to me as I left college? Hard to say. Quite possibly. I ended up working in film visual effects and then tech startups.

  The state of the world is very different today than it was back in the 90s. The tech landscape is massively different. We’ve been through a couple of boom bust cycles, we’ve had web2.0, crypto has started making serious progress, most people use computers, user generated content is normal, there’s an exciting Cambrian explosion in generative AI. We have moved forward technologically but also culturally, and we are starting to wrestle with the impact of all this tech. Two areas that are making the science and engineering disciples cool again are Space Travel and Quantum Computing.

  Elon Musk and SpaceX is building rocket ships that can carry 50x the load of previous rockets into orbit. These are using new materials designed from first principles to push the boundaries of what is possible. They plan to turn humanity into a multi-planetary species by colonising Mars. There are also plans to send missions to the moons of Jupiter and mine asteroids. That’s pretty darn cool.

  The latest All-in podcast has a great interview with one of the SpaceX engineers involved in the latest launch, which ended in disassembly but was nonetheless a big success. Really worth listening to:

  • SpaceX launch, Fox News settlement, “Zombie-corn” exodus to AI (All-in Podcast)

  Quantum Computing is very much at the early stages of development however real world production hardware is beginning to be commercialised. The UK government has bought some of these recently. All this tech is brand new, which means new materials need to be developed. New companies will be build. New software will be written. New applications will be found. New tools will be forged. There’s so much fertile ground in this space both technologically and culturally. Are we going to see films written about quantum computer hackers? Surely. Technology and culture is at a point where materials science and engineering disciplines are very very cool.

• Nostr and Damus first impressions

  2023-04-20 14:00:00 +07:00 by Mark Smith

  I’ve been listening and reading to much bitcoin, crypto and blockchain content these past few months. I decided it would be interesting to try out Nostr, which is a blockchain simple client & relay based social messaging protocol and network. Loads of the bitcoin people are talking about it, especially since Elon bought Twitter. It’s sort of like Twitter except it’s decentralised, so there’s no central company that runs it. Rather than use a username and password, you use public/private keys to create your identity.

  Nostr is a simple, open protocol that enables global, decentralized, and censorship-resistant social media.

  Though it’s similar to Twitter in many ways, you post messages to a timeline and can see other people’s messages, which you can reply to, and you can post pictures and videos, but in other ways it’s different and a bit confusing. You have to connect to relays which forward your messages throughout the network for instance.

  I installed Damus, which is the iOS app. It installed without any issues and opening it up for the first time was mostly quite straight forward. The app is minimal, with all of the functionality you would expect for a messaging app. It’s a nice looking app.

  They automatically set you up with some relay servers and one or two users so that you can see some messages in the public timeline. That makes big difference, even if it’s a bit odd to only have messages from one user viewable. In my case it was the account of one of the project devs.

  I suppose it’s the same for everyone. He was going on about quite technical dev stuff which didn’t make much sense to me. Not a huge deal, but I can imagine that would scare off most normal non dev folks. I clicked around and was able to view a few threads where others had commented on some of his messages. It all felt a bit disjointed, but made a bit of sense.

  After looking through the settings and filling out some basic details I was ready. Then the big unanswerable question: where is everyone? How the heck do you find people?

  I had seen that some folks have been sharing their Nostr npub on their Twitter profile. Npub’s as far as I can tell are the user’s public shareable key. I found one of these and stuck it into the search box, which tried to autocomplete it into some sort of hash tag. The search results that came back were weird long text strings. Didn’t look like it had found anyone.

  Then I searched for a few usernames of bitcoin podcasters. That did return what looked to be the right accounts, but this is where I started wondering how safe this all was. Was it safe to click on long strings of text? There were loads of messages with folks sharing text strings, but what happens if I click on one that then somehow hacks my phone? Nothing against bitcoiners and crypto people but they are all using nyms, voice modifiers and balaclavas and you read about hacks like every few days in the news. I decided to hold off clicking on anything and following anyone.

  I found this Github repo from cryptoquick that has some general Nostr info but also suggests usernames of other devs to follow. They come with long text strings that don’t start with npub. Are these npubs or something different? It’s not clear.

  There is also a way to attach a domain name to your username by going through a mostly automated verification process. Looks like it’s relatively easy to do, you have to put a special text file on the domain you use to prove it’s your domain. I guess I could do that once I’ve figured out how to add some users.

  By this stage I’d used up all my available energy and good will, and the thought of having to determine who to follow was too much. It just felt like there was a quite high non zero chance I could screw up my phone if I made s mistake. So that’s where I left it. I can see all the messages from the initial dev bloke and whoever replies to him, but that’s it. I’ll have to revisit this when I have more time. How do you safely find people on this thing?

• I miss watching films

  2023-04-20 10:06:00 +07:00 by Mark Smith

  I read an interview yesterday with film director Nastia Korkia, where she answers questions about her new film Almost Spring (Pochti Vesna). It’s about the normalisation of corruption in everyday russian life. After reading the article I remembered how much I used to enjoy watching films, especially weird and unusual foreign films. Some of you reading might be aware I used to work in the film visual effects industry.

  The way the director describes the film making process and how she tried to re-create the ambience of the time and place where the story unfolds really speaks to how magical film is as an art form, a window into different cultures and places.

  It’s been a few years since I have watched a movie. Life circumstances have for the moment at least removed that from my day to day activities. At first I was just too busy, but since the pandemic, without going into any details, it’s basically become no longer possible for me. Even as I write these words the world is strongly pushing back against me.

  Anyway I just wanted to say I miss watching films a lot.

• My favourite javascript proposals (2023)

  2023-04-08 07:29:00 +07:00 by Mark Smith

  The 95th meeting of the TC39 just finished, meeting notes have been published, and blog posts are being written. Figured I might as well pick out my favorites, based on what I can see being most useful.

  I'm writting this without an internet connection however I have the article listing the proposals linked above saved. Based on descriptions alone these to me look like the most interesting / least boring:

  • Import Attribute - mentions importing JSON, which I was trying to do recently and was suprised to discover that it doesn't work with impprt, I do this with require all the time, so this might be cool, if it's what I'm thinking it is
  • Async Context - Looks interesting, but also looks like it could turn out to be super boring, it will depend on what you can actually do with the feature
  • Await Dictionary - still not totally sure what this is, though any additional async/await control structures would likely be awesome given how popular async/await has become, I use it all the time now
  • Time Zone Canonicalization - again not totally sure what this is but I read through temporal's github project a while ago and thought it looked kind of cool, dates and especially timezone stuff in javascript has always been very annoying, so improvrments there would no doubt be very welcome

  Right those are my initial picks. Time to find an internet connection.

  Incidently, wouldn't it be incredible if Safari could save a page but also automatucally save all the main article's linked pages? IMO, that would be the best feature literally ever.

  After checking online, it's mostly as I thought, and the asyc context proposal does imo look like it could be useful for a few advanced use cases that I can think of.

• Normal process for banks in distress

  2023-03-25 11:30:00 +07:00 by Mark Smith

  I've extracted the following from a recent episode of the Eurodollar University podcast with Jeff Snider and Steve Van Metre. They discuss various topics but figured it was worth re-sharing the step by step process for better understanding of the dynamics at play.

  In normal times banks going through distress for whatever reason would do the following:

  1. Go into illiquid loan book
  2. Take say 100 million dollars of illiquid loans out of the book
  3. Liquify that by pledging those 100 million dollars to a money dealer like Goldman Sachs or JP Morgan
  4. Borrow US treasuries, not at same value, there’s probably going to be a significant financial penalty, but at least they will lend the bank US treasuries
  5. Then go into the repo market with those treasuries, in order to secure some short term financing, so they can at least keep the doors open
  6. That will keep them from having to fire-sale assets

  That process hasn’t been happening, at least not in enough banks. We know that because the amount of banks going directly to the FED asking for loans has sky rocketed. Total amounts via discount window are much bigger than usual.

  Also 4 week treasury bills mid week (15 March which is typically a seasonally troublesome time) suddenly had a massive drop in treasury bill rates (~60 basis points), much lower than normal. The only reason people keep buying these treasuries at such low rates is likely (the argument goes) to be because they are being used for something other than investment. They are being used as collateral for loans.

  The fact the FED can charge such low rates implies there is a massive need, so points to the fact that many many players (likely the smaller banks) are trying to get collateral in order to get loans.

  Two systemic problems are occurring:

  1. Big banks don’t want to lend treasuries against illiquid assets
  2. Money dealers don’t want to lend cash against the treasuries

  What do they know about the long term illiquid assets that the smaller banks don’t know?

• Review: Robolledo RA#876

  2023-03-21 20:11:00 +07:00 by Mark Smith

  Resident Advisor Mix: RA.876 Robolledo

  "Mauricio Rebolledo is one of those veteran artists who has stood the test of time. He can play Burning Man, go back-to-back with DJ Tennis and mentor young producers with unwavering finesse, staying true to his signature sound of sparse, psychedelic chuggers. Where many of his peers have embraced a more commercial sound, the Mexican artist likes to keep it weird, opting for touches of Krautrock and hazy atmospherics in both his sets and on his albums."

  I read this in the mix description and I instantly feel at home, this is the sort of realm I feel comfortable in.

  I start to listen, I stare off into the distance, I am removed from the general flow of things, and I am transported to a place where I am watching myself being defragged, all the chaos of the past few weeks is being neatly, efficiently, effortlessly re-arranged, it feels good, it feels clear, high fidelity, it feels stereo.

  Groups of red green and blue blocks of varying sizes move between shelves in an infinite library of 12 inch records, and music videos, and memories of clubs and gigs I’ve been to, from small 30 person rooms to stadiums and festivals, the euphoria of all these things permeates my self.

  The ropes that tie me down are anialated and I levitate above all struggles, the impossible becomes possible, I see, I sense, I feel paths through the chaos.

  But wait hold on, about 1/2 way through, the world stops allowing you, things become strained and difficult.

  You are tied once again to the Earth, you fall, you struggle, and eventually you accept you are no longer hovering, no longer floating, your head drops, you exhale.

  And then a chorus of angels swoops in like a wind from on high, the beat picks you up, and you’re off again, the bondage of earthly things is once again dissolved and you are free.

  You continue, things are passing, and you realise quite unexpectedly you have reached orbit, a wonderfully retro electronic landscape stretches as far as you can see.

  The trip continues for some time until it gently fades and you glide in and softly land.

• Monetary system link dump

  2023-03-21 15:30:00 +07:00 by Mark Smith

  Quick link dump of podcasts covering the very hectic goings on in the finance industry.

  I've been finding that some of the best coverage has been from Bitcoin podcasts. This is because they are likely currently some of the most knowledgeable about the fundamental dynamics at play because they have had to go through so many ups and downs in such a short period of time.

  Analysing the monetary system is very similar to debugging a gigantic bit of software. The data is spread out all over the place, but it's very much like hunting down a tricky bug.

  • A global buying panic in bonds triggered by more than a little panic-panic in the monetary system (Eurodollar University)
  • Bank Runs and Macro Update with Dylan LeClaire (Citadel Dispatch Podcast)
  • What markets are telling us about potential for more bank runs and failures (Eurodollar University Podcast)
  • Beginners Guide To Bitcoin (1/3) with American HODL (What Bitcoin Did Podcast)
  • Bank Failures & Bitcoin w/ Steven McClurg Ep#121 (We Study Billionaires Podcast - Bitcoin)
  • Bank Crisis & Bitcoin with Jesse Myers #Ep94 (Citadel Dispatch Podcast)
  • Bank Runs, Bailouts and Bitcoin with Caitlin Long (What Bitcoin Did Podcast)

  If you only have time for one of these, I think probably the Caitlin Long episode was the most insightful, but they are all phenominal.

• Gang stalker premeditated goading escalating to violence

  2023-02-20 08:07:00 +07:00 by Mark Smith

  Early this morning I was involved in a gang stalking situation that was purposely escalated to violence. Following some typical drive-by goading, something I have previously written about in my newsletter.

  I retaliated with a small amount of water from a water bottle onto the T-shirt of one of the purpotraiters to signal "back off buddy”. After which the attacker punched me in the face several times, and many more times on the shoulder and side, he was apparently trained in martial arts, adopting a sort of karate/kung-fu stance.

  He was wearing a motorbike helmet and was apparently a motorbike taxi driver. I pushed him back forcefully and perhaps hit him on the shoulder, I can’t quite remember, it happened very quickly.

  Some people from the street, apparent bystanders, ran towards us but instead of separating us they only pushed me back. The attacker then basically had a team of collaborators that pushed me back any time I tried to fight back while the attacker could land more punches on me at will.

  I did the only thing left to free myself from the situation, pushed his motorbike over, after which he came at me with a running jump onto my back from afar. I was able to wrestle him to the ground into a sort of fight 69 position, where I was able to kick him in the jaw and head, which I did only quite lightly all things considered, because I probably could have caused him some serious damage had I wanted to. My aim was just to get him to leave.

  Following a bit more of pushing around from his gang members, that had previously just been standing around as if they were randomly there unrelatedly, they all left. None of them stayed to make sure I was okay. It's very clear to me that they were all operating as some sort of gang.

  I've ended up with quite a bit of bruising to much of one side of my body, a lot of scrapes and grazed on one knee, and a bloody nose. The nose might be broken, I'm not enturely sure about that, it hurts quite a bit to the touch.

  So far today I’ve been approached several times by other motorbike taxis that seem to be aware of the incident, finding it very amusing. Twice while writting this post.

  This is a clear escalation of the gang stalker harassment that I’ve been experiencing daily for many months now, and I just wanted to document it publicly here so I could remember exactly what happened, and so others could know what these types of organised violence incidents look like.

  Incidentally the security guards from nearby shops and cafes did absolutely nothing to intervene, it was more like entertainment for them. A bit later after the incident, I overdressed two of them having an unnaturally loud conversation where the words “wow” was used in apparent amazement. It wouldn’t surprise me one bit to discover that they (or at least some of them) were in on the whole situation.

  Note a similar situation happened to me about a year ago in a park with an apparent gardener who sprayed water on me on purpose using a water sprinkler then proceeded to violently hit me with it when I kicked it to the ground to protect my electronics equipment. I wrote about it in the newsletter at the time.

  At that time an older gardner stepped in to separate us, but was also ultimately much more concerned about pushing me back than the attacker, which I hadn’t thought about until today. That situation felt pre-meditated too, with several goading incidents moments before the attack.

  Update: FYI - I'm in HCMC Vietnam, I have been here for approxinateky 3 years now.

• Your life is a sort of personal internet

  2023-02-04 21:30:00 +07:00 by Mark Smith

  Earlier today I was thinking about some event that happened to me quite a while ago, and I was remembering various people that were around during that period, as you do. The thought unfolded in my head something along the lines of, such and such was there, and also this other person, oh yeah, they came online about then via this other person. And I monentarily paused and smiled because I had clearly just used the internet as an analogue for my life.

  This will perhaps make less sense to folks that have always known the internet. Most of those from my generation can remember the moment the internet appeared in their lives. It's fascinating to me that the internet has become so real to me that I now completely naturally use internet language to think about people arriving into my life.

  Life is a bit like a personal internet, we all have one, and though they overlap, they are distinct. People come online into your life, and sometimes they also leave. Since our personal internets overlap at times, we share a lot of things, but our overall experiences of life, our personal internet, are very different. Something that makes sense in one person's life, might make absolutely no sense whatsoever in another person's life.

  A lot of times though stuff does make sense across personal internets. Finding those common areas and learning about other less familiar things is what makes meeting people fun, though it does sometimes get a little bit weird.

• The importance of good RSS feed GUIDs

  2023-01-25 16:30:00 +07:00 by Mark Smith

  The past few days I've been trying to migrate my existing RSS feeds to the new site. The blog, linkblog and podcast all have distinct feeds, and they are currently hosted on seperate subdomains, which each resolve to a seperate server. The linkblog and podcast feeds are both created by the initial version of my static site generator, whereas the blog's feed is built using a static site generator called Jekyll.

  When I wrote the initial implementation of the ssg tool, I just went with what seemed like the default settings of the feed generation library I was using. That library expects each feed item to have a post url, which makes sense because clearly consumers of the feed are highly likely to want to know where the post is located on the web. The library docs also state that you can provide a GUID for each feed item, but that it's optional, and if you don't provide one, then it will just use the post url as the GUID. That seems to make sense because urls are essentially unique, which is good because GUIDs need to be unique. So what's the problem?

  Well the problem only really happens if you decide to move your website. Of course each post on your new website will at the very least have a new domain name, but if you change website software, then the structure of the urls could change too, depending on how the new software organises the site pages. With new feed item GUIDs, there's a chance readers of your feed will suddenly see duplicate items appear in their feed reader.

  The reason for this is because many feed clients use the GUIDs as a way to keep track of items in the feed. If you change the GUIDs, then existing posts could very well end up looking like totally new feed items, and be displayed as such, right next to the ones with the original GUIDs. Readers will unnexpectedly be faced with doppleganger feed items! It's freaky enough in real life, but in an RSS reader? It's almost too scary to contemplate.

  I wish I would have been aware of this when I initially set the feeds up. All the feed items have a uuid which I could easily have used. Now I'm faced trying to figure out a way to keep old GUIDs the same. There's a relatively easy fix for the podcast. Since there aren't that many posts I could easily go through and set the uiid to the post url. That way they will remain unchanged in the new location. The podcasts data will look a bit strange, but at least things will look good in Apple Podcasts. An other way to approach the problem is to add some special cases to the feed generation code, which I could do, but I really don't want this sort of special case brittleness in the codebase.

  Changing the data for the linkblog isn't really an option because there is in the order of 170000 posts. I'm not sure what I'll do there. I haven't checked what Jekyll uses for GUIDs. I'm guessing it's post urls.

  Does it even matter? I doubt there are many folks actually subscribed. And it might very well be that the duplicates issue only really arises for the latest items. After all the RSS feeds only actually contains the last 20 posts for the blog abd 50 posts for the linkblog.

  That's pretty much what you need to know about RSS feed GUIDs. I'm still not sure what I'll do. I think I'll likely try to make it look good in Apple Podcasts, and just go ahead and change to using uuids for the blog and linkblog. If you know of any reason I should do it differently please email me.

• Safely Moving Your Podcast Feed

  2023-01-24 20:00:00 +07:00 by Mark Smith

  A quick note on how to safely move your podcast RSS feed in case I need to refer to it later for some reason.

  I recently migrated my main website off of Heroku and I’m consolidating most of my sites into the one site to make things easier to manage. I’ve managed to migrate most things successfully, but one remaining item is the RSS feeds. This article covers the podcast feed migration.

  Apple has published a couple of useful articles:

  • Change the RSS feed url
  • How to change hosting providers

  The main idea is to setup a redirect on the existing feed location which returns an HTTP 301 and the location of the new feed. You also need to add a special custom <itunes:new-feed-url> tag to the new feed. With these items in place clients that use the feed will get redirected to the new feed location, and they should also be able to see the new feed url from the custom tag, and update their settings with the new location.

  Finally you have to ensure that the existing post GUIDs in your RSS feed don’t change. Without doing this, existing podcast clients could end up with duplicate episodes being listed because they will appear as new episodes.

  Once setup you need to keep the redirect and custom tag in place for 4 weeks. During that period Apple Podcasts will update the feed in your account. The reason for doing it this way is to ensure that existing subscribers to your feed continue to receive posts without needing to update their podcast client.

  Here are the steps from the docs:

  1. Set the web server where your feed is hosted to return an HTTP 301 redirect response when it receives a request for the old feed.

  2. Use the itunes:new-feed-url tag in your new feed to point to the new URL. This response updates Apple Podcasts with the new feed URL and your subscribers continue to receive new episodes automatically.

  I have my existing podcasts site hosted at Netlify. While setting up the redirect I ran into some issues so I opened a ticket in the community forum:

  • Issue setting up redirect

  If all else fails then you can also directly change the feed in Apple Podcasts account. However doing it this way will cause existing subscribers to stop receiving new posts.

  As I write this I’m still waiting to get an answer about the redirect. Hopefully the migration will happen without any major issues.

• Perpetual website DAOs

  2023-01-22 12:00:00 +07:00 by Mark Smith

  One of the things website owners don't spend much time thinking about is what will happen to their website after they die. Once they aren't there to maintain it, and perhaps more importantly, to pay the hosting and donain name costs, then the site will most likely ultinately go offline.

  Some folks make arrangements with long running organisations / foundations to host their site, but that night not akways be possible if you don't have content that these institutions are interested in preserving, or just don't have the contacts to make that happen. There's also the internet archive which could keep a backup, but that won't help the live site.

  In web3 there exists somethibg called a decentralised autonomous organization, or DAO for short. They are essentially code hosted on a blockchain with an associated digital wallet address. People are using them for all sorts of things, often to help with project governance and organisational tasks.

  What if you created a DAO to pay the costs to keep your site online? Theoretically that way you could keep your site online after you pass. I'm not an expert in this area, just thinking out load.

  Your DAO would need a way to make money, it could be related to the content, but maybe it could be a simple AI that could access the stock markets. Make little bets everyday, make a few pennies here, a few pennies there. It could be as simple as donations from site visitors.

  It's an interesting thought experiment, because one could imagine a whole ecosystem that would be built to accomodate such activity. Assuming your site is fully static files, your website DAO might want to be able to perform tasks such as migrations between hosting providers, offsite backups etc. Who knows maybe even a bit of web design to keep it looking fresh?

  I wonder if anyone else is thinking about self sustaining DAOs?

• The list of titles that didn't make it

  2023-01-07 18:00:00 +07:00 by Mark Smith

  Several weeks ago now I started a new section in my newsletter which I've ended up calling the "list of titles that didn't make it". It has it's origin in a list I had been keeping every week of possible titles for the newsletter. At the end of the week I would read through the list and pick one, or sometimes disguard the entire list and go with something totally different. At least with a list I can remember a lot of the interesting topics that I spent time reading, listening and thinking about.

  One week I was struggling to find a title, and I think I was rushed for time, and the week had been going so terribly that in a sort of burst of exasperation I just thought, well fuck it post the whole darn list, which I think, in it's own way, was quite illustrative of the moment.

  So I did, and I immediately felt a lot better. A weight off my mind.

  The next week, I did the same thing again, and kept doing it. The list rapidly turned into a place where I was venting frustrations and worries, but also a place where I was exploring thoughts and ideas. I don't think it has completeky settled into something concrete yet. It varies week to week. I often look back at them and wonder what the f I was thinking or doing at the time, but I just post it anyway, warts and all.

  Sometimes the entries are real title candidates, but other times it's more like a sentence or a short paragraph that I don't have any intension of turning into a title, but they usually express something I think is in some way important. I don't edit them, and only occasionally remove some. They are often cringey, controversial and risqué. They don't necessarily convey my point of view, often just taking the form that they popped into my head. They could be from any point of view, and often are meant to get the reader, which includes future me, to think about certain aspects of an idea that are new or unnexplored. Sometimes they are humorous, other times very serious, or a mixture.

  They have so far been overwhelmingly dark. There's a lot of turmoil both in the world and in my life at the minute and it's definitely showing in the lists. I'm going to keep doing them, because it's helping me deal with the crazy world, but it's also a bit of a barometer of the present moment that perhaps some find interesting. If you don't like them, feel free to skip over them.

  Now you know what those wierd lists are.

2022 #

• Code syntax highlighting for static websites

  2022-12-17 18:00:00 +07:00 by Mark Smith

  It’s the first blog post on the new site!

  The past few weeks I’ve been finishing off the migration to the new website which now houses the blog, linkblog and podcast, plus a bunch of independent one off pages. It’s been slowly taking shape but yesterday I got syntax highlighting working, and given that I mostly write about tech, javascript and the web, it started to feel like a proper website.

  The site is still very minimal, which I like because I want the focus to be on reading, but I do have aspirations to improve on the design in the coming months / years. This site for instance has a nice simple elegant and readable style. That’s vaguely the sort of thing I’m aiming towards.

  In any case most of the essentials are there and so what better way to celebrate all this by a blog post describing how I did it? Maybe it will help others out there on their programming journey.

  A quick note about my site. It’s built using a static generator that I wrote, which generates the static HTML files that make up the site. If you have a Wordpress site or use one of the popular static site generators then there will likely be some sort of syntax highlighting plugin available for you to use. On the other hand if you have something custom or you are curious how syntax highlighting works in general, then you might enjoy this post.

  The way syntax highlighting of code works:

  1. Add your code to a block surrounded by <pre> and <code> tags in your static HTML file
  2. Add special syntax highlighting CSS classes throughout the block of code
  3. Add the syntax highlighting javascript library to your page
  4. Add CSS rules to your page that use the classes peppered through the HTML to colorise the code on page load

  Once you have it working with the default setup, it’s really straight foward to swap out those CSS rules for a different theme. There are loads of themes available which work for most popular programming languages.

  I used two libraries. The first is marked, which renders the HTML from the source markdown files. The second is highlight.js which is the syntax highlighting library. There are other options, which might be more appropriate depending on what features you are after, but I’m going to focus on these since that’s what I got working on my site. I think they all work in a somewhat similar way. The code that glues all this together is written in javascript and runs in a standard nodejs runtime.

  As I was saying, marked reads your markdown files and renders them, outputting HTML files. It also has hooks for syntax highlighting. What this means is that with a bit of additional configuration, marked will add in the syntax highlighting classes to your HTML (step 2 listed above) when it renders the markdown files.

  I put together a minimal example that I’ll run through here, but it’s also available in this github repo. It is meant to run on Unix/Linux type systems. You will likely need to make some modifications to have it run on Windows. I got some great help on the marked repo issues page after posting a question.

  The place to start is the markdown file that will be transformed into HTML.

  index.md

  # minimal-syntax-highlighting
  
  A code block:
  
  ```js
  function hw() {
    console.log('Hello World'); 
  }
  hw();
  ```
  
  That's all folks! 
  

  There’s a code block in there delimited by the triple backticks. Some people refer to them as code fences. When the file is rendered, the renderer wraps their contents in the <pre><code></code></pre> tags, which help to keep newlines, spaces and indentation in your code intact. The other way you sometimes see to add code blocks is to indent all your code by four spaces, however that doesn’t work with the syntax highlighting.

  Using backticks enables you to specify the programming language, which can be used by the highlighting library. In this example I have specified javascript by adding js after the opening set of backticks. The list of available languages is on the website.

  You will need to add a package.json which will be used by the node runtime to install the libraries:

  package.json

  {
    "name": "minimal-syntax-highlighting",
    "version": "0.0.1",
    "description": "Minimal example demonstrsting syntax highlighting",
    "main": "index.js",
    "scripts": {
      "build": "mkdir -p dist && node app.js"
    },
    "author": "Mark Smith",
    "dependencies": {
      "fs-extra": "^9.1.0",
      "marked": "^2.0.3",
      "highlight.js": "^11.7.0"
    }
  }
  

  The most important part is the npm build script which creates the target directory then runs the main app file. Also note the listed libraries which will be installed when you use npm to install the project libraries.

  Then the main app file which ties everything together:

  app.js

  #!/usr/bin/env node
  
  const fse = require('fs-extra');
  const { readFile } = require('node:fs/promises');
  const path = require('path');
  const marked = require('marked');
  const hljs = require('highlight.js');
  
  console.log('starting...');
  
  const templatePath = path.join(
    process.cwd(), 
    'index.md'
  );
  
  const outputPath = path.join(
    process.cwd(),
    'dist', 
    'index.html'
  );
  
  console.log(`templatePath: [${templatePath}]`);
  console.log(`outputPath: [${outputPath}]`);
  
  marked.setOptions({
    renderer: new marked.Renderer(),
    highlight: function(code, lang) {
      const language = hljs.getLanguage(lang) ? lang : 'plaintext';
      return hljs.highlight(code, { language }).value;
    },
    langPrefix: 'hljs language-', // highlight.js css expects a top-level 'hljs' class.
    pedantic: false,
    gfm: true,
    breaks: false,
    sanitize: false,
    smartypants: false,
    xhtml: false
  });
  
  (async () => {
    try {
      const contents = await readFile(templatePath, { encoding: 'utf8' });
      const renderedContent = marked.parse(contents);
      const html = `
        <!DOCTYPE html>
        <html lang="en">
        <head>
          <title>minimal-syntax-highlighting</title>
          <meta charset="UTF-8">
          <meta name="viewport" content="width=device-width">
          <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/styles/default.min.css">
        </head>
        <body>
          ${renderedContent}
          <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/highlight.min.js"></script>
        </body>
        </html>
      `;
      await fse.outputFile(outputPath, html);
      console.log('done');
    }
    catch(e) {
      console.error(e.message);
      console.error(e.stack);
      process.exit(1);
    }
  })();
  

  The first line tells the shell which runtime to use to execute the program. In this case it’s node.

  The marked configuration is done using the setOptions function that expects an object with configuration options. The important keys are ‘highlight’, which is the function that will be used to add required CSS classes. It detects which language is being used, the calls the library’s highlight function. Different highlighting libraries have slightly different highlight functions.

  The other option that’s important is the langPrefix option which is a string that will be added to the CSS classes throughout the code. Again this is going to be different for different highlighting libraries.

  Most of the important code is in an async IIFE so we can use await for asynchronous calls. We read the template into a variable, render the template using marked, insert the rendered HTML into the layout using a template literal, and finally write the file to disk.

  Notice that the layout has both a @ <link> in the head which specifies the libraries CSS rules, and a <script> at the very bottom of the body, which will be used when the page loads to add the colorisation to the code.

  You can run this in a terminal by executing:

  npm install
  npm run build
  

  That will output a file in the following location:

  dist/index.html
  

  If everything worked as expected, when you open that file in a web browser, it will contain a code block which will have all the syntax highlighting, making it much nicer to read. You will need to be connected to the internet since the js and css files are hosted on a public CDN.

  Here’s an example deployment on Netlify.

  There’s also a Github action configuration in the repo which will be triggered for code pushed to branches with names matching the regex. The workflow clones the repo, configured the nodejs runtime, installs project dependencies, builds the HTML, then deploys the file to Netlify. You will need to add the capitalised environment variables in the deploy step to your Github repo’s secrets. You should be able to get the required info from your Netlify account settings.

  .github/workflows/buildAndDeploy.js

  name: Build And Deploy
  on:
    push:
      branches: 
        - main
        - fix-*
        - feature-*
    workflow_dispatch:
      inputs:
        git_ref:
          description: Git Ref (Optional)    
          required: false
        deploy_enabled:
          description: Deploy after build? (yes|no)
          default: 'yes'
          required: true
  
  env:
    deploy_enabled: ${{ github.event.inputs.deploy_enabled || 'yes' }}
    git_ref: ${{ github.event.inputs.git_ref }}
   
  jobs:
    build_and_deploy:
      name: Build And Deploy
      runs-on: ubuntu-latest
      env:
        node-version: 16.14.2
        
      steps:
        - name: Clone Deploy Repository (Latest)
          uses: actions/checkout@v3
          if: ${{ env.git_ref == '' }}
   
        - name: Clone Deploy Repository (Custom Ref)
          uses: actions/checkout@v3
          if: ${{ env.git_ref != '' }}
          with:
            ref: ${{ env.git_ref }}
   
        - name: Setup NodeJS
          uses: actions/setup-node@v3
          with:
            node-version: ${{ env.nodejs_version }}
   
        - name: Install modules
          run: |
            npm install
   
        - name: Build Site
          run: |
            npm run build
          
        - name: Run tree
          run: |
            tree -ap dist
            
        - name: Print output to console
          run: |
            ls -la dist/index.html
            cat dist/index.html
   
        - name: Deploy using Netlify CLI
          if: ${{ env.deploy_enabled == 'yes' }}
          uses: netlify/actions/cli@master
          env:
            NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
            NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
          with:
            args: deploy --dir=./dist --prod
  

  A couple of things to watch out for when integrating this into bigger projects, both of which happened to me simultaneously, causing many days of tedious and difficult debugging.

  My static file generator was running all output through an HTML prettified library, which basically makes the output look nice by standardising the indentation and using consistent line breaks. The prettifier removed all the newlines in the code blocks, so in the final output, each code block was a single really long line. I added a way to specify not to use the prettifier to fix that issue.

  The static site generator also runs all the renders through an EJS render so that I can have EJS syntax in the markdown files. I was using a render option called rmWhitespace, which removes excess whitespaces that sometimes get added by the EJS render process. The problem though is that it was removing the indentation in the code blocks. So it was all rendering fine except every line was squashed up against the left margin. I set the default for markdown files to be false to fix the issue.

  That’s pretty much it for this tutorial. If you’ve made it through to here then you should now be able to implement syntax highlighting on your website, and you’ll be aware of some things that can trip you up when integrating it into a bigger project. Best of luck with your projects.

• Alternative Nations

  2022-06-24 12:30:00 +07:00 by Mark Smith

  I got inspired while listening to this excellent mix - Maybe you can guess where exactly, listen as you read

  
  

  Where do you come from?

  Hmmm, well I come from maaaany places…

  
  

  I come from jazz

  I come from rock

  I come from folk

  I come from pop

  
  

  I come from heavy metal

  I come from grunge

  I come from punk rock

  I come from ska

  I come from britpop

  
  

  I come from hip hop

  I come from gangsta rap

  
  

  I come from breakbeat

  I come from drum and base

  I come from house

  I come from ambient

  I come from disco

  I come rave

  I come from acid

  I come from reggae

  I come from dub

  I come from dancehall

  I come from bangra

  
  

  I come from live music

  I come from radio

  I come from music festivals

  I come from carnivals

  I come from parties

  I come from after parties

  I come from after after after parties

  I come from Ibiza

  
  

  I come from noise

  I come from experimental

  
  

  I come from podcasting

  
  

  I come from pubs

  I come from riverboats

  I come from front of house

  I come from kitchens

  I come from chefs

  I come from chalkboards

  I come from daily menus

  I come from split shifts

  
  

  I come from sledging

  I come from skiing

  I come from snowboarding

  I come from tennis

  I come from camping

  I come from basketball

  I come from baseball

  
  

  I come from javascript

  I come from css

  I come from html

  
  

  I come from sql

  I come from bash

  
  

  I come from perl

  I come from python

  I come from java

  I come from c++

  
  

  I come from linux

  I come from unix

  I come from irix

  I come from windows

  
  

  I come from engineering

  I come from science

  I come from feature film visual effects

  I come from software startups

  I come from software development

  
  

  I come from web2.0

  I come from APIs

  I come from user generated content

  I come from bit torrent

  I come from copy left licensing

  I come from gnu

  I come from free software

  I come from open source

  I come from blogging

  I come from photo journaling

  I come from videoblogging

  I come from hacking

  I come from making

  
  

  I come traveling

  I come from road trips

  I come from digital nomadism

  I come from Europe

  I come from the Middle East

  I come from the USA

  I come from Canada

  I come from South East Asia

  
  

  I come from bad relationships

  
  

  I come from sobriety

  
  

  I come from COVID

  
  

  I come from loving life

  
  

  I come from racism

  I come from abuse

  I come from exclusion

  I come from blocking

  I come from surveillance

  I come from police (fake?) brutality

  I come from security guard / tuk tuk / taxi / delivery bike mafia intimidation

  I come from stalkers

  I come from capture gangs

  I come from exploitation

  I come from conspiracies of silence

  I come from an infinite list of contradictions

  I come from years of continuous and co-ordinated daily harassment

  
  

  I come from starvation

  I come from thirst

  I come from torture

  I come from crucifixion

  
  

  I come from homelessness

• The occasional listening issues of my head

  2022-06-16 09:54:00 +07:00 by Mark Smith

  I previously wrote about [reading and memory issues I some times have](/blog/2022/ 03/23/reading-and-memory-issues-i-sometimes-have). Around about the same time I wrote that, I also wrote another piece about audio listening issues. I never published it though. This week I’m looking for blogging topics, and maybe it helps someone out there, so here it is.

  Similar to my occasional reading issues, I’ve noticed that I also at times have issues listening to audio. It takes a very similar shape to the reading issues, in that it feels like some sort of buffering is happening.

  I can hear all the words and I’ll be listening attentively. Then I notice that I’ve sort of blanked out for some amount of time, and I can’t remember what I’ve just been listening to. I then have to rewind to a place I recognise, sometimes it’s quite a long ways back, occasionally a minute or two, but usually several seconds.

  It also happens when my mind is being triggered by the audio I’m listening to. I will start to formulate lots and lots of questions about what I’m listening to, and in doing so I loose my place in the ongoing audio. Mood also has an affect.

  At times it’s annoyingly small side thoughts that totally derail my listening, I’m left trying to remember what I was listening to, and I simply can’t remember, conscious that the not remembering is causing me to miss yet more stuff because the audio I can hear is basically going straight to the void. And so it gets even worse.

  At times it gets so bad that the only way I can absorb anything at all is to write copious amounts of notes. I have to pause and rewind, continuously and write notes, so I understand what’s going on, but I loose the flow of the overall audio, and I suspect that it sort of spoils my long term memory of the piece, because a lot of the subtleties and nuance get clobbered.

  Taking a break helps a bit, and having food + something to drink. In case it wasn't already abundantly obvious starvation and thirst does not help one bit, something which I have been unfortunate enough to verify.

• The HTML5 Phone

  2022-06-10 20:01:00 +07:00 by Mark Smith

  I’ve been having this re-occurring thought. I don’t know how feasible it is, but I wanted to write about it just in case. Wouldn’t it be awesome if there was an HTML5 phone?

  The HTML5 phone would have all it’s UI built from HTML, CSS & Javascript, would likely run on a minimal Linux distro. Native apps would run in a custom NodeJS runtime, perhaps using something like Electron. It would be great for web developers, with the possibility to develop for the phone, on the phone. It could also leverage the web platform via PWAs. There’s a lot of variation possible, but the focus would be on HTML5 everywhere.

  I think there are a lot of current trends that could make this crazy pipe-dream a reality that could gain traction:

  • Recent big push towards open web standards
  • Move to privacy, phone OSs like Purism, the e/OS phone
  • More alternative custom javascript runtimes
  • Phone hardware is fast
  • There are a lot of web developers out in the world at this stage

  Wouldn’t a progressively enhanced OS be cool? Always know, that the basic apps you need will always work, no matter what phone you have.

  It might not even be that difficult to start. Initially just create a bare minimum suite of apps, progressive enhancement style, that just do the very basics of the following apps:

  • Browser
  • Email
  • Camera
  • Music
  • Video
  • Notes
  • File Explorer
  • Contacts
  • Messaging
  • Calling

  Build them as super simple web apps first. Once there’s something functional, enhance as needed, port to native apps etc.

  The most difficult app is probably the web browser. I had initially thought that you could just have a browser and do everything with PWAs, but I think it’s sensible to have native apps to make sure you’re not forced to wait for web platform features.

  Anyhow some interesting posts about creating browsers that I found, just to get a sense of what’s involved:

  • How to get started building a web browser?
  • How to build a web browser -- Part 1: Specifications
  • Building a Web Browser Using Electron

  Yeah I know it’s a sort of crazy half baked idea, but I just keep thinking how awesome it would be, even if it was quite rough around the edges initially.

  As a web developer, I’d like to have more control around my phone / mobile / tablet experience, so why not just do it all in HTML5?

• OS Progressive Enhancement

  2022-06-10 19:54:00 +07:00 by Mark Smith

  I’m trying to get back into the blogging flow after a period of not writing much, so the next few posts are likely going to be a bit light on well-thought-outness, which isn’t even a word, but whatever. The enemy for me right now is getting bogged down, just trying to keep moving…

  I really like the practice in web development to create websites using progressive enhancement. Essentially you first create a bare minimum app using mostly HTML, and platform features, make sure that works, even if it looks a bit weird. The key thing is that functionality-wise it’s operational.

  Then you enhance that with styling using CSS, and improve the functionality using javascript. This ensures that in pretty much any situation, even if for some reason the CSS and Javascript fail to load or get blocked, you still have an operational website.

  I’d like to have something similar with Operating Systems. I’d like to know that no matter which OS I have to use, be it Desktop, Laptop or Mobile, that the default apps provide a minimum of functionality so I can at least do basic things.

  Major apps like contacts, notes, browser etc could have minimum functionality specs, so you could quickly see where you could move to and still be functional.

  Slightly scattered thoughts, but the gist of it is progressive enhancement for operating systems.

• The Big Tech Sandwich

  2022-06-10 19:36:00 +07:00 by Mark Smith

  Sandwich

  • Tech
  • Culture
  • Religion & history

  Background

  I recently wrote a series of newsletters where I explored the idea of a Big Tech Sandwich. A mental model for how to think about tech in the most broad context, how to handle the mess that is society. Hopefully it’s helpful in moving things forward so that we can make the world a slightly better place for the next generation. I personally have found it useful.

  This blog post is a sort of summary, to have something I can point to in the future. I’m going through a moderately severe case of writers block the past few weeks, combined with very patchy internet access, which isn’t a great combination. Consequently this post is unlikely going to be a super smooth read, but I think the ideas are interesting. Consider this a v1, I’ll update later, better to have something published than nothing at all.

  These were the newsletters:

  • Mid-week special: Navigating the big tech mess (Wednesday 27th April, 2022)
  • The full full stack, and the weight of history (Saturday 30th April, 2022)
  • The Big Sandwich (Saturday 7th May, 2022)
  • Culture Smash (Saturday 14th May, 2022)
  • Demons be gone (Saturday 21st May, 2022)
  • Mid-week Special: Dilemma Addendum (Tuesday 24th May, 2022)

  A quick note about culture because it means different things to different people. When I’m speaking about culture here I’m mostly thinking about culture of society rather than something like company culture. They are similar and related, but I’m thinking more about music, art, movies, books in general, with large movements that develop mostly organically.

  Scale

  Where is tech in the bigger picture? We that spend most of our time in the tech industry have a tendency to over play how important we think tech is in society. My observation is that, yes things are complicated, and yes we’ve made a lot of progress, but things are a mess, much messier than is immediately obvious. When you start looking around it’s a bit unnerving to realise. Things are a mess.

  But wait, it gets worse, technology is really just a small part of our overall culture. We have some effect on it, but culture is huge, and guess what? It’s also a mess, a really big mess actually.

  As you zoom out some more, the timescales increase dramatically. You soon get to the scale of history and religion. You realise how small tech is in the overall picture. Oh yeah and by the way, you thought tech and culture were a mess, well let me tell you, things get mind bogglingly messy at this level. It’s literally unbelievable that we are all living side by side in this small rock we call Earth.

  Society, culture, history and even religion play a much much bigger role than we like to admit to ourselves. This sort of stuff is absorbed automatically as we grow up, we don’t even notice a lot of it.

  Culture is important, but very difficult to get right. It can help us build temporary scaffolding around difficult areas of our shared history. It gives us the ability to move forward in the present while being informed by the past, but without getting too bogged down. It’s by no means perfect, there are a lot of bumps in the road in some places.

  I wrote quite a bit about the example of how popular culture helped to integrate very different parts of UK society. That issue:

  • Culture Smash (Saturday 14th May, 2022)

  I also wrote a related but different piece a bit earlier which is relevant:

  • Punk Rock Internet (Saturday 19th March, 2022)

  The future

  Things are actually getting better. When you stand back and see the bigger picture over thousands of years, it’s apparent that things are getting better. The large empires of the past that ruled in very violent ways are becoming much more modern, slavery has been abolished, we collaborate across borders, in different languages, build things together, and we do it using tech.

  It’s difficult though because we have all these tough histories that we’ve all been through. There are some very scary things that happened, and for many it’s still horrendous. Some people have a lot, others don’t have much.

  So that’s my first attempt at a summary, it glosses over a lot. To get a better sense you really need to read some of the newsletters, and listen to some the podcasts linked in those issues. I encourage you to do that, it’s fascinating stuff. We can get culture moving and popping again, and in even more diverse ways. It might even be fun.

  It’s a mess, but slowly, one step at a time, we can together make it better for the next generation and ourselves.

• Budism vs Christianity - WTF World?

  2022-05-25 10:24:00 +07:00 by Mark Smith

  Fat Budha: Abundance!

  Jesus nailed to cross: Everything is my fault!

  WTF world? Seriously.

• The good homeless person’s (and society’s) dilemma - Addendum

  2022-05-24 12:30:00 +07:00 by Mark Smith

  An additional thought to the post from a few days ago.

  The non-homeless man is possibly part of a Capture Gang - See section about audience capture for details.

• The good homeless person’s (and society’s) dilemma

  2022-05-21 10:00:00 +07:00 by Mark Smith

  A homeless person has a series of daily run-ins with a non-homeless person, where the non-homeless person is unknowingly doing something that puts the homeless person in some sort of danger. The danger could lead to a bad thing happening that has the potential to eventually cause the homeless person’s death.

  The homeless person tries each day to tell the non-homeless person that he’s being dangerous but the non-homeless person simply does not see the danger.

  After a few days the non-homeless person realises what it was that he was doing that was dangerous and stops doing the dangerous thing.

  Meanwhile everybody else has stopped giving the ‘crazy’ homeless person money, because they don’t want to incentivise his ‘crazy’ behaviour, so he’s been starving and thirsty for several days.

  The homeless person essentially had to pay in advance to fix somebody else’s problem.

  Eventually, if this sort of thing keeps happening, the homeless person doesn’t have enough flesh on his body and so he dies.

  Also, the non-homeless person might be lying.

• Npm installing multiple private Github repos using ssh aliases

  2022-04-21 17:10:00 +07:00 by Mark Smith

  When you are developing in NodeJS it’s often useful to be able to install private modules.

  The main ways to do that are:

  1. Github Packages using Personal Access Token (PAT)
  2. Private Github repo using Personal Access Token (PAT)
  3. Private Github repo using Deploy Keys

  Apparently (1) is the most popular among developers, however it requires setup and management of additional workflows to create the Github Package.

  (2) Only works for a maximum of 1 private repo.

  (3) Works for 1 private repo, and can be extended using ssh aliases to work with multiple private repos.

  Deploy keys are also more granular in terms of security than PAT.

  This article focuses on method (3).

  Github has an example repo that demonstrates how to install a private repo using deploy keys. It’s worth testing that out.

  If you try the same methodology for installing more than 1 private repo in the same workflow, you’ll run into an issue. The reason being that Github doesn’t allow you to re-use the same deploy key across repos, and by default npm will use the default ssh private key (~/.ssh/id_rsa), of which there is only one.

  However if you configure ssh to use aliases, you can specify a different private key for each alias, and then use those aliases in your package.json to specify the private module dependency.

  Read more details about how to do that in the Github docs.

  For example if you had 3 private repos:

  • username/repo1
  • username/repo2
  • username/repo3

  Where repo1 installs both repo2 and repo3, then add an ssh config ($HOME/.ssh/config) as follows:

  Host github.com-repo2
    Hostname github.com
    IdentityFile=/home/user/.ssh/private_key_repo2
  
  Host github.com-repo3
    Hostname github.com
    IdentityFile=/home/user/.ssh/private_key_repo3
  

  Then specify the modules like so in repo1’s package.json:

  "dependencies": {
    …
    "repo2": "git+ssh://[email protected]:username/repo2",
    "repo3": "git+ssh://[email protected]:username/repo3",
    …
  }
  

  It’s important to specify the protocol correctly. Without ‘git+ssh://‘ the install completes but none of the files actually get installed. Instead some type of symlink gets created. At least that’s what happened in my tests.

  Store your private keys in Github secrets, and before your install step, create the private keys on the filesystem from those secrets:

  /home/user/.ssh/private_key_repo2
  

  and

  /home/user/.ssh/private_key_repo3
  

  Now in your repo1 workflow you should be able to install all repo1 dependencies using:

  npm install
  

  It’s worth having a verify step after install that lists the installed files using ls -l and/or tree.

• Infected leg wounds

  2022-04-15 10:30:00 +07:00 by Mark Smith

  I’m still battling with infected leg wounds, but things are a bit better than earlier in the week. I can at least sort of walk now.

  What happened? I don’t fully know.

  A month ago, a stranger nonchalantly walked up to me in a park, touched my right knee, smiled and walked off. Very weird.

  A couple of days later that exact spot started feeling sore. I didn’t think much of it, but I ran out of disinfectant, and it rapidly turned into an abscess.

  I got bandages and disinfectant, but then ran out again. I had to use moving tape to secure my last 2 bandages. I was able to get to a pharmacy but the moving tape created more sores, and those turned into more abscesses.

  Somehow the abscesses jumped across to my other knee. The whole knee ballooned with one absolutely giant abscess. Over the past few days that has turned into a strip of abscesses that looks oddly like an upside down exclamation point. What are the chances?

  Earlier in the week I was worried it might permanently affect my mobility, I could barely walk. I’m now worried about scaring. An improvement but not much better.

  Dressing and cleaning these sores has been close to some of the most intense pain I’ve ever experienced. Having to spend literally hours at a time squeezing out puss, with the same intensity and force as an arm wrestle.

  Something similar happened to me in India a few years ago, I’ve got a small scar from that experience. Looks like I’ll have a bigger scar this time.

  There’s been a lot of unpleasant reactions from people, from mild racism to people finding it hilarious seeing a foreigner in pain.

  I know these are a small group overall, but as with trolls in online comments, they can make it seem like the whole world is against you. I’m both mentally and physically bruised at the minute.

  You might be wondering: Are you ok?

  My response: No not really.

• My brain's favorite passtime

  2022-04-15 10:00:00 +07:00 by Mark Smith

  My brain: Do this now or I will delete this…

  Me: Yes that does sound interesting, I’ll just finish what I was doing first, it will only take a few seconds

  My brain: Too late! this has been deleted, haha, that’ll learn you

  Me: Shit, what was I doing again? :(

  I think perhaps it happens more often when "the external world" is constantly butting in trying to prompt me todo things that I am already about to do. Eventually it devolves into some sort of race condition, and the only thing you can do is walk and move really really slowly like you are a grandad.

  Makes you wonder about alternative scenarios for where grandad’s actually come from, which doesn’t help the situation much either.

• What I love about loud guitars

  2022-03-28 12:30:00 +07:00 by Mark Smith

  I’ve been listening to loud guitars for about 40 years. That’s quite a long time.

  I took a bit of a break from listening to music recently. It was for about two years. It wasn’t something that was planned, life circumstances arranged themselves in such a way that listening was more difficult than it had been previously, I was very busy with other things, and I just sort of stopped.

  I was still listening to podcasts, but not music. At some point I listened to a podcast that reminded me about my love for music. It was a strange feeling, like oh-yeah-I-used-to-really-love-music, in that moment I realised that I had basically forgotten. It was the strangest thing, similar to remembering a dream.

  I’ve since started listening to all sorts of music again, electronic and band / guitar based. Not loads, but some. I’m so glad that I remembered.

  I was listening to some great punk/hardcore/indie/metal recently and was moved to try to put into words the way it makes me feel. The following are extracts of what I wrote.

  That thing where you feel like you’ve just been plugged into the mains, and all you want to do is shout YES over and over and over again, really fucking loudly.

  It’s like you’ve just been injected with the adrenaline of 1000 horses.

  And then the bit finishes and you’re like…"God damn".

  It’s odd because the feeling doesn’t happen in your physical body, you can listen to a whole song and not have any physical change, but the feeling is like it’s in some sort of virtual body, that’s somewhere there in the background, that’s momentarily become very very alive.

  It also feels like you are inside the sound, which is odd because it’s the sound that’s inside you, but when you hear that guitar none of that matters anymore, whether it’s you or the sound or whatever, because you’ve just been plugged in, that part of you is alive again.

  It’s all the music you’ve ever listened to, all the music videos you’ve ever watched, all the live shows you’ve been to, all the late nights, and all the people you’ve shared those experiences with, in that moment of guitar electricity, they are all there alive again.

  It’s such a strong feeling that it’s really bizarre to me that some people can listen to the very same sounds and feel nothing. It just sounds like noise or something.

  Those were the bits I wrote. I also love listening to quiter music too, but in that moment, it was all about the loud guitars.

  I don’t drink alcohol anymore, it’s been about 3 or 4 years now. I’ve noticed that the music is even better, the drum sounds are crisper, the guitars are more alive, for electronic music, everything is much more hightenned. I notice so much more, it’s like the sound is now in HD when before it was SD. I didn’t expect that at all. It's awesome.

• What can web developers learn from the industrialisation of farming?

  2022-03-27 12:30:00 +07:00 by Mark Smith

  Stuff You Should Know published an interesting episode all about the chicken farming industry. It’s a really great piece, I encourage you to go and listen to it.

  The truth between Cage-free and Free-range (Stuff You Should Know Podcast)

  I was appalled to learn about how horrendous the conditions are for those animals. It’s shocking to learn about some of the practices that we have developed while industrialising farming, that are normal in our societies. It’s good that people are reconsidering some of these norms.

  It also got me thinking about our future with the forward march of technological innovation. We are building ever more complex technologies. Here’s the thought process I can’t seem to shake:

  1. Anyone can write software

  2. We can build virtual environments

  3. There exist people that have industrialised extremely cruel animal farming

  4. Those people will use similar techniques on other people, it’s going to happen, it’s just a matter of time

  It might not be as visually obvious as chicken cages, in fact it’s likely that it would be designed to fit in our societies and appear normal.

  How would we detect instances of people being raised using battery farming techniques?

  How would we get out of such a situation if we were already in such a situation?

  What about related concepts like bullying? How will that be in the future?

  Very relevant and really worth listening to, especially if you are into psychology:

  The Dangers of Concept Creep (The Art of Manliness Podcast)

  I think it's worth trying to imagine this sort of thing because ultimately the thing we are farming with the world wide web and the metaverse is ourselves.

  Let's imagine a better future for everyone.

• Reading and memory issues I sometimes have

  2022-03-23 20:25:00 +07:00 by Mark Smith

  Difficult to talk/write about this.

  Firstly because it’s not something I’m particularly happy to admit, but also because I’ve only recently become aware of it, or at least aware enough to be able to describe it. It’s something I’ve been in some way aware for years, and it might be getting worse, I can’t tell. So what is it?

  Sometimes when I read, I lose the ability to read consistently and with ease. It doesn’t always happen. It seems to happen more when I’ve had a turbulent day, something that’s been happening a lot lately. I get into a state where instead of words coming off the page, so to speak, going through my eyes and being perceived by my brain, in a mostly consistent flow, it feels like there is some sort of buffering going on.

  Instead of the words just streaming into my thoughts and brain, my brain seems to change into a mode that operates a bit like that quiz show where the contestants answer questions and have to bank the money at the right time or they loose it. I forget the name of that show, it’s with that dreadfully scary red head woman.

  Anyway what happens is this. I read several parts of the sentence and at some point a hole chunk makes it through to my consciousness, all in one go, and on to the next bit that gets slurped in the same way. I have to speed up and slow down in just the right way, or bits start to get missed.

  I become aware that it’s happening and I kind of feel when the next chunk is going to be ‘let in’, and that screws up the rythm, and so some chunks get skipped or only partially comprehended, and I find that I have to constantly go back and re-read passages to fill in the gaps.

  Perhaps it’s got something to do with being tired. But I’m thinking it might be something else. When it’s not happening, when I’m feeling uncluttered and fresh, the buffering thing doesn’t appear to happen, reading is as smooth as ridding a hover board. When it’s happening it feels like someone is turning a tap on and off.

  Something that I’ve noticed that is very different the past several months is that I see a lot of people swinging their arms. This sounds really weird, but at some stage a few months ago basically everyone everywhere started a new exercise routine, where they swing their arms backwards and forwards. Some people do it standing still, others do it while they walk. Sometimes they are doing it while walking backwards.

  It’s the new trend where I am. I see literally hundreds of people doing this every single day. That wasn’t always the case. I’m not sure exactly when it started happening. Anyway it’s something that people do, a lot, fair enough, but I wonder if it’s affecting my ability to read smoothly because I’m visually seeing this repetitive up down motion over and over again. It might be completely unrelated, but I think it’s worth mentioning.

  I wonder if it’s happening to others without them realising. It’s quite subtle, easy to brush off, especially if you operate day to day with things like caffeine and alcohol. I don’t consume alcohol (it’s been 3-4 years now), and caffeine relatively rarely.

  I’ve also noticed my memory at certain times isn’t as good as I expect. I’m trying to remember something, I almost remember it, I can just about feel the thought poping into my head, and then it’s gone, sometimes this happens several times before the memory finally makes it’s way through to my conscious mind.

  Occasionally I have to get on with something else, to distract myself from myself, and the memory comes back randomly like a callback function that just got delayed by a very slow internet connection, and a temperamental firewall.

  Depending on where you are, you might find people around you that know about this will try to ‘hack’ you. That’s something that definitely happens. It’s a good opportunity to practice keeping calm under pressure, or just to surrender and move on to another place.

  That’s my best description of the reading and memory issues I sometimes have. Just putting it out there in case it helps someone else that might be having similar problems.

  I haven’t found many ways to improve it apart from just not doing anything for a few hours. I guess meditation can be good too. I do that occasionally. These are not always possible options, sometimes you just have to soldier through to get what you are doing done.

• Typescript makes function declarations difficult to read

  2022-03-21 19:31:00 +07:00 by Mark Smith

  I’m not a massive user of Typescript, though I see it could be very useful in some situations.

  The main issue is that I think the additional cognitive load isn’t trivial, and so code that you could easily skim through to get a good idea of what’s going on becomes full of little hurdles. Perhaps it’s something you get used to, but I fear it would seriously impact my ability to read and understand code. It decreases visibility.

  Having said that, I really like that you can specify the definitions in JSDoc. That’s cool because the code remains regular, fast to read javascript code. If you are wondering about types, just look at the JSDoc, which is always right next to the function declarations anyway.

  Though you don’t get the full power of Typescript using it that way, only the type checking. I’d also like to be able to use interfaces and other object oriented features that come with Typescript.

  Which brings me to the point of this post. If you are using full fat Typescript, IMO the way to define the types in the function arguments sucks. Reason being that it uses the colon (:) character.

  function equals(x: number, y: number): boolean {
      return x === y;
  }
  

  I don’t like that because the colon character is for objects declarations. Following years of writing javascript objects, my brain’s muscle memory visually associates that character with objects, so when I’m scanning a page I can easily jump to places on the page without having to fully read the code. Using : in function arg type definitions breaks that for me. I find it’s much more difficult to quickly get my bearings in Typescript code.

  I’m also not a fan of long function argument lists and Typescript doubles or triples the length of function argument lists. I feel that function declarations need to be short and to the point. It makes reading code much easier. I don’t find that having the return type is very useful in the function definition. It just feels like clutter to me.

  Some people say they like to know the types before they get into the function. Personally I find that once you start reading the code inside a function, it’s usually very obvious to differentiate between objects, arrays, strings, numbers etc. If I want to know the type, it’s preferable to glance at the JSDoc for the function.

  There’s a proposal to bring some Typescript syntax to javascript. Personally I’d like it if the type definitions were kept separate, keep function declarations readable! But I would like javascript to have interfaces and additional OO features.

• Creative Economy vs the Cruelty Economy

  2022-03-21 08:55:00 +07:00 by Mark Smith

  Some thoughts on these two concepts, don’t take these as fact, just some observations, might lead to something interesting, or maybe not.

  In the cruelty economy you are rewarded for receiving and disposing of cruelty. People optimise for not being hurt, not getting angry, for catching the purpotraiters of cruelty. Things tend towards a imprisonment.

  In the creator economy you are rewarded for receiving creativity and releasing it. People optimise for maximising the creativity, even bad stuff can be good. Things tend towards anarchy.

  Things go bad when your capacity to process is exceeded. Creativity turns to cruelty, but perhaps also cruelty turns to creativity.

  Real true anger is a capacity problem, it’s the group mis-managing capacity.

  It’s a sort of symbiosis, perhaps similar in concept and behaviour to other natural phenomena like the El niño - La niña weather system. In the case of the creator-cruelty economy, ultimately it’s all based on chemical fluctuations in our brains.

  It’s a really tough problem to think about, because we are all in the map so to speak, whatever actions we take, including thinking about the dynamic, affects the dynamic, and the balance.

  A potential danger is that you get caught in one extreme or the other, without enough capacity to switch. And then you are probably in for a bad time. Smashed against the rocks, over and over again, in plain site.

• The cruelty economy

  2022-03-18 10:00:00 +07:00 by Mark Smith

  The Creator Economy has been pretty great.

  So many things to watch and listen to. But that’s nothing compared to what’s coming down the pipe: The Cruelty Economy!

  Creative Economy + Web3

  Crypto, blockchains, memecoins, DAOs, cosplay, algorithms, Anon, AI, brain-computer-interfaces, WOW.

  War and peace…AT THE SAME TIME!

  Some of you are gonna love it! At least some of the time.

  What could possibly go wrong?

  With our patented starvation&thirst algorithm, nothing!

• Cool things that were in web2.0

  2022-03-18 09:00:00 +07:00 by Mark Smith

  Last week I listened to a John Gruber interview with Tom Watson and Daniel Agee, the founders of the photo sharing app Glass. They talked a lot about what it was like during web2.0, including some of the cool things that got developed in that period.

  I enjoyed the brief trip down memory lane, an oasis of calm amongst the current madness all around. I spent a few minutes making a list of some of the things I remember, and enjoyed wondering how some of these types of technologies and trends could blend with things in web3.0.

  Here’s that list:

  • Rocketboom, Ze Frank, and videoblogging
  • Citizen journalism
  • Creative Commons licensing
  • Crowd funding
  • Community events and meet-ups
  • The Maker movement
  • 3D printing
  • Photo sharing
  • Video streaming
  • Online maps and navigation
  • Blogging and RSS
  • Live photo feeds from events with photos displaying on your desktop as they are taken
  • Journalism integrated with data analysis tools
  • Social media
  • Newsrooms integrating social media into their broadcasts
  • Wikipedia
  • APIs everywhere
  • Web services - AWS, Azure etc
  • Podcasting and podcast tools
  • Open source hardware
  • Open source software
  • Github repos and actions
  • Bit torrent for sharing assets with listeners to collaborate on
  • User Generated Content
  • Live video streaming mixed with forum/chat rooms with bots that trigger video and audio overlays when events like tipping occurs
  • Gifs
  • Memes
  • Newsletters
  • Weird one offs that never took off like Chat Roulette

  Some of these were around before web2.0, but got very popular during web2.0 and arguably some were developed more recently but IMO feel very web2ish. Nonetheless when I think about web2.0, these are some of the things that come to mind.

• My unsophisticated view on whether we have free will

  2022-03-18 07:00:00 +07:00 by Mark Smith

  People are always talking about this in podcasts and around the internet, and I suppose it’s all quite interesting, but I just wanted to have some clear thoughts about it myself. So I spent a few minutes thinking about it and this is what I came up with.

  It’s probably completely wrong. I wouldn’t take this as life advice.

  Do we have free will?

  Yes. Because…

  1. You have a sort of biological choice algorithm in your head, implemented from neurons organised in neural networks, and it develops in your brain over time, and at any one time that choice algorithm is used to make choices that act on the inputs your brain is receiving. The choice is independent from the inputs, rather than being somehow packaged up and smuggled into your brain along with the inputs

  2. We have the ability to introspect, so in principle we have the ability to observe and see instances of unfree coercion, and change our thought processes accordingly - YMMV, every person's situation is very different, some have a lot more external pressures than others, which isn’t always bad, but it definitely can be

  3. If you remove everything, so you are an astronaut floating in the emptiness of space, you can still make choices

  Therefore humans have free will. In this view, free will is very similar to independence of thought.

  Feels kind of good to have that written down, at least I’ve got something I can point to now.

• Offline

  2022-03-16 09:15:00 +07:00 by Mark Smith

  Yesterday I wrote about the idea of offline pull requests. It wasn’t a fully formed idea, more of an observation that the current git tooling could improve the offline experience. In the modern, always on, always connected world, the idea of offline might sound a bit strange or out-dated. It might also seem a bit weird for a web developer to be interested in offline because developing for the web is all about being connected.

  I like the idea of being able to move relatively seamlessly between online and offline. I’ve found though, that the offline experience is often lacking.

  For example, Safari has a feature called Reading Lists. It’s cool and I’ve been using is a lot recently. With a couple of clicks you can add a page to your reading list. It’s s bit like a bookmark, but in addition to saving the url, it has the ability to save the entire page, so you can read it later offline. When it works, it’s amazing.

  I’ve found that my workflow has changed significantly. Instead of endlessly infiniti scrolling, I can spend 10-15 minutes in the morning, skimming through stuff in my feeds, adding what looks interesting to the reading list, then disconnecting. At a later point in the day, or maybe the next day, when I have some time, I can read through the articles, make notes, have ideas, write blog posts etc. Awesome right, so what’s the problem?

  There are a few. First and foremost it doesn’t work on all websites. I have no idea why that is, I guess websites have to be written in a special way for it to work. I haven’t had time to research this yet, I’d like to know why it only works for some websites. Another issue is that there is no indication that the site isn’t “offlineable”. You click “Save Offline” and it looks like it saved. Later you try to open the site while offline and you just get an error message, I forget the actual wording, but it feels a lot like “You can’t read that because you are offline, idiot”. You used the feature as advertised, and you still got a slap in the face. You keep calm, and carry on with your life. It happens a lot.

  I found an option, buried deep in the app settings, that automatically saves each item you add for offline reading. Nice feature, definitely makes things better, but you still get slapped in the face witch the error messages a lot, because loads of sites just aren’t “offlinable”.

  I wonder if website owners just don’t like offline because it blocks the click info they get when you are online, maybe it somehow affects their add revenue. With better tooling that’s an issue that could be solved.

  It would be great if in addition to the saved page, each linked item in the saved page was also saved for offline reading. And wouldn’t it be cool if there was a way to click a button to get the current page, as well as a selection of other popular articles on that site.

  Matt Mulenweg when in Antarctica, prepares for his trip by downloading portions of Wikipedia. Cool idea. I wonder how he does that, I guess he’s got a special Wikipedia reader tool. I’d like to be able to do that sort of thing in more places. Like when browsing a Github repo, save the site, but also save the associated documentation site. I wonder if Progressive Web Apps (PWAs) might be in some way a solution here.

  Reading Lists are useful but they don’t work with video. I miss watching Youtube videos. With an offline workflow, there’s no way at the minute to watch video. In the early days of podcasting, people often put videos in their feeds, that was what videobloging was before Youtube. I’m not saying that’s better, I still like watching on Youtube, but I’d love it if there was a “Watching List” similar to how Reading Lists work.

  Online experiences are great, but I like offline too. Podcasting has shown us that offline is worth considering, that it’s beneficial to our lives. Let’s imagine a future where offline and online are complimentary.

• Offline Pull Requests

  2022-03-15 08:41:00 +07:00 by Mark Smith

  Aside from being a phenomenal version control tool, git’s ability to work offline is one of it’s best features. This is especially true if you move around a lot, but even if you don’t, sometimes you just need to disconnect from the network, avoid distractions from things like social media and email, and do some heads down focussed development.

  Once you’ve made some progress, it’s trivial to sync back up with the repository remotes. Pull in the changes since you were last online, merge with your code, resolve any conflicts, and push your changes back up to the remote. This is possible because when you clone a repository, you have an entire copy of the repository on your local machine.

  This way of working is standard with the command line git tool. It’s how it was designed to work. Each developer has a complete copy and can work entirely independently. This worked really well for many open source projects, but as git hosting platforms emerged, they added new features. One of the most praised has been the Pull Request (PR). It’s become so central to modern development, that most developer workflows in some way revolve around them.

  They are essentially a way to co-ordinate ongoing feature work. They take the form of a web page that has a discussion thread where contributors can talk about the changes they are making to the code. They make many code commits to a feature branch on their local machine, push those commits to the remote, and at some stage a PR is created. Discussions then happen, more code commits can be added, and when the feature is deemed to be complete, it can be merged into the main branch and the PR is closed. At that point all the commits that were pushed to the feature branch will be in the main branch.

  The PR has become more that just a discussion area. Many integrations with 3rd party tools enable running of test suites, with results displayed in the PR page along side discussions. There are other neat features like bots that can scan code, and post results to the discussion, code reviews, analysis of comments to see mentions of other PRs, and for example to trigger workflows. The automation features can help speed up development, and maintenance of the project. Overtime PRs become a key place where knowledge about how the code was developed is stored. It’s very usual to spend some time browsing through closed PRs to get a sense for how things are moving along, or how a particular bug was fixed. The PR has been very much a successful feature and has been adopted by most git hosting services.

  It’s not to say that PRs don’t have their downsides. Some of these were highlighted in a recent episode of the Changelog podcast(~20:00). They speak about many of the pros and cons. It’s an interesting discussion. There has also been a “PRs are bad” meme making the rounds the past few weeks (but I can’t find it right now), and much has been written about the pain of PRs. We love PRs, but some of us, at times, find them an impediment to progress.

  In my personal work I often want to be able to work entirely offline, but I miss the ability to write notes as I develop a feature. There isn’t much point in using PRs since they aren’t accessible offline. PRs are entirely provided by the platform, standard git has no such feature. Making PRs available offline would be an incredible feature, but it probably doesn’t make that much sense, because the threaded discussions would get all out of sync. I’d be happy to be proved wrong on this though.

  Offline PRs would make moving between providers feasible. If you have ever tried to do that you’ll know that it’s not at all straight forward. Moving the repo is easy, moving the PRs and all the accumulated knowledge within them, not at all easy.

  However with PRs acting more and more as a place to combine the results of many tools, I wonder if there couldn’t be some form of independent notes that couldn’t be written offline that could then be automatically attached to a PR along with the pushed code commits.

  If I were developing git based tools, enhancing the offline experience would be something I would spend some time on. Working async is becoming the norm for remote teams, and though it’s great to work together and essential at times, the ability to drop off and work offline is something that is very important in order to be able to keep a healthy work/life balance, but also so that when you do get together with the team, that time is even more beneficial because you’ve been able to make much progress offline, without all stepping on each other’s toes.

• Exploring iOS Creation Tools

  2022-03-15 08:12:00 +07:00 by Mark Smith

  I recently had to re-install most of the main Apple iOS apps as the previous versions were all crashing on startup. While I was doing this, I took some time to look at the feature sets of these apps, most of which I never use. I was pleasantly surprised, there’s actually quite a lot you can do with these default apps that look like it could be very useful. A lot if the apps are quite minimalist, and have enticing design.

  However functionality is not obvious straight away. I find that most of these apps don’t seem to follow typical conventions for where features are or how they are implemented. Each one appears to do things in a sort of unique way. The first 20-30 mins of playing with an app, I was constantly taping the wrong place, opening the wrong menu items, getting stuck and having to close the app and re-open it just to get back to a place I recognised. It’s way too easy to delete things in iOS apps, and there’s no undo. I’ve lost / nearly lost loads and loads of stuff accidentally deleting something when the touch UI started miss-interpreting my gestures, or accidentally making an unintended gesture. So it’s not obvious and learning is very frustrating.

  Having said that it looks like the following things might be possible with standard Apple apps:

  • Publishing ebooks (Pages)
  • Recording and editing audio (GarageBand, Voice Memos)
  • Recording and editing video (iMovie)
  • Some basic automation (Shortcuts)

  Being able to do all these things from a mobile device would be awesome.

  The design of these default apps have a very “Apple” look and feel to them which is great. However I’m a bit disappointed that documentation and marketing pages are a very scattered. The default selection of apps is actually quite good, but I don’t get the impression that Apple is taking them very seriously. Each one should have a canonical page on the website and there should be downloadable documentation. The whole offering feels more like a shabby patch work than a suite for creators. It’s like they did all the hard work of building the restaurant and then gave up right before creating the menus.

  Anyway, in my experiments with GarageBand, though making music is probably a bit optimistic for me currently, recording an audio podcast might be possible. I’d like to be able to record audio segments and drop them into some form of template, and render out an episode, complete with intro and segment audio jingles.

  I’m guessing the whole template thing probably isn’t possible, but having a rudimentary way to put together a podcast from some audio clips might be.

  Speaking of which, wouldn’t it be awesome if you could add annotations in the Podcasts app while you were listening to a podcast, and a way to easily crop out short clips, so that you could insert them into a podcast you were creating?

  I like the idea of being able to have an async conversation via the medium of podcasts, for fun but also could be very useful in a work setting too. Anyhow just wanted to mention briefly my recent experiences with iOS apps, frustrating, but I can see potential possibilities.

• Nginx and the Ukraine-Russia war

  2022-03-15 07:56:00 +07:00 by Mark Smith

  With the war in the Ukraine unfolding, I started wondering about tools and libraries that might be affected by the crisis. What happens to open source projects that are caught in the cross fire of war?

  The first such item that sprung to mind was Nginx. It’s used by an enormous amount of the modern web as a reverse proxy and load balancer.

  Nginx is one of the biggest open source successes in recent memory, and as far as I know it’s developed by russian based developers. Looking at their website it appears they self host all development rather than use a git SaaS platform like Github.

  Nginx has already been in the news in the past couple of years for similar issues. Thankfully the code is open source, but it’s clear that developing safe and reliable software for a world wide ecosystem is not very straight forward.

  With the US and many countries imposing sanctions on Russia, how will that affect the open source communities online?

  I don’t have an answer to that question, but it’s something I’m looking out for.

  My best wishes to both Ukrainian and Russian developers, I hope you aren’t caught up in the madness of war.

• The Mirror (A poem of cruelty)

  2022-02-24 20:15:00 +07:00 by Mark Smith

  They: Be the mirror!

  I: I won’t be the mirror!

  They: we hate that you won’t do what we say, so we hate you!

  [some time passes]

  [everybody is a bit sad]

  They: what do we do?

  I: I don’t know

  ...

  [some time passes]

  ...

  They: Be the mirror!

  I: I’m the mirror!

  They: we hate the mirror, so we hate you!

  [some time passes]

  [everybody is a bit sad]

  They: what do we do?

  I: I don’t know

  ...

  [some more time passes]

  ...

  I: I guess I’ll go then, again. Maybe some others have food that I can eat and water that I can drink, so that I don’t die

  Update: This poem will be the last that I ever write so as to try to avoid even more abuse than is currentky beibg inflicted upon me. Cruelty should not be encouraged. Cruelty in the world is very real, I am absokutely certain if that now.

  Update: Lots of very unnusual internet connectivity issues today

• Next generation tools and workflows for the creator economy

  2022-02-23 10:45:00 +07:00 by Mark Smith

  There was an Interesting Smashing Magazine piece earlier in the week, Thoughts On Markdown. It does a really good review of the transformative effect Markdown has had in tech, especially by developers, but also by creators.

  Markdown unlocked a whole ecosystem of workflows, that have been generally centered around git version control platforms such as Github. Reason being that they now all offer CI/CD tools, i.e ways to run shell programs that can do things to and with the files in your repository when you do different actions like push/commit/merge files.

  I wrote about this trend a little over a year ago in my piece about Github Actions for custom content workflows. I later wrote about Mozilla MDN Docs are going full Jamstack, which was a high profile example of the trend. I was seeing in my own work how the combination of automation, git version control and a simple authoring format were transformative in what I was able to do. I think the fact that Mozilla went all in on such a workflow is a big sign of things to come.

  Along with the praise about the impact, the Smashing piece’s thesis is that although good things have come about because of markdown, that it isn’t well suited for editors, writers and creators. They go into a lot of depth in their article, it’s really worth the read.

  I’m sympathetic to their point of view because although it’s fantastic for quickly writing documentation for coding projects, I have found it a bit tedious for writing, but especially editing essay style pieces.

  I love how easy it is in Markdown to add URLs, lists, for boldening/italicising text, and for adding titles and sub headings. Guess what you can also just add HTML directly in the file for those situations where the syntax falls short. All that is great, and it doesn’t bother me that you end up with some slightly ugly syntax scattered through the text. I’m fine with that, in fact for URLs you get used to it very quickly, and it’s actually, in my opinion, argusbly better to be able to see the full URL text, so you can easily spot mistakes before publishing, and it encourages you to favour well written URLs. It’s often overlooked because it’s somewhat subtle, but structured and nicely formatted URLs make browsing and sharing on the web a much nicer experience.

  The biggest annoyance for me is in the editing, because the way corrections are displayed in a Github PR are sort of close to unreadable. Each paragraph is one line in a markdown file, and if you change 1 word, the entire paragraph is highlighted in a way that you lose the flow of the whole document. I’m constantly having to preview, commit, review. It’s that feeling where you can’t see the woods for the trees, and it makes writing prose much more difficult than it should be.

  The topic came up in a recent Shop Talk Show. Dave is keen on exploring Markdown editors when he's old and Chris is bullish on block editors from his mostly very positive experience with Wordpress’s newish Gutenberg editor. He’s always going on about it (in a good way), so there must be something to it.

  And that’s were the Smashing piece ends up, talking about the concept of block editors, which I am totally unfamiliar with. It sounds interesting, though I wonder how much of the automation and collaboration features of a Github+Markdown workflow you would lose by moving to block editors. Also, and this is a big one: portability. I personally am willing to put up with some of the annoyances of Markdown because, I know it works everywhere that supports git repos. Where is the portability in block editors?

  There are signs that some of the benefits of modern webdev blocks and components are making their way into Markdown, with for example Mdx, which is a markdown variant that makes it possible to embed React and Vue components directly in your markdown files. So maybe we will get both markdown and block editing in the new embedable web that the folks at Smashing envision.

  It’s a topic that I’m watching closely because it will have a big effect on what I like doing. I’ll finish with a Markdown Pro/Cons list as I currently see it:

  Markdown Pros:

  • Portability
  • Collaboration
  • Git friendly
  • Compatibility
  • Huge ecosystem of tools
  • Being able to stick it in a PR, then stick it into a workflow

  Markdown Cons:

  • Not the best writing experience
  • Lousy editing experience

• Changes to the blog and newsletter

  2022-02-22 11:45:00 +07:00 by Mark Smith

  Towards the end of last year, I stopped posting to the linkblog. It wasn’t by choice, I still enjoy reading and curating content, but life configured itself for me in such a way that it was basically impossible to do in any meaningful way. I had just reached the 10 year anniversary. I’m glad I made it that far, but the world said no, loud and clear.

  I have no way to post privately using the linkblog tool. I never had the chance to build that feature. It was on the roadmap, but there were always more important features that popped up, and I had made the decision to implement that feature later to try to make things a bit easier. There was already so much to worry about, and it seemed to me that storing less private data would eliminate a whole class of security concerns for my little sass web application.

  With no links, I didn’t have any material for the newsletter, so that had to be put on hold. I eventually did send out a short new year’s edition. It was fun putting it together, and to be honest I quite liked the brevity. I put out another, and another. I also started adding a short unique one line title to each edition. The idea was to have something memorable, a sort of virtual hook to hang it on, so it doesn’t just fade into obscurity of time. I’m liking the new format.

  I also have a bit of a goal of writing more blog posts, but I want it to be fun, and not get too stuck in crafting the perfect post each time. I might check out some writing tools I keep hearing and readibg about like Grammerly and Hemingway at some point. Until then though I’m just going to try writing and editing quickly. So expect the posts to be a bit rough around the edges, with spelling mistakes, and annoying repeated bits. Sorry. Those bits annoy me too, but I think quantity and general narrative is more important than minutia at the minute. Hopefully my writing will get better over time. It’s a somewhat different way of thinking to writing code, and my brain just doesn’t do it all that well yet.

  I’m still liking the idea of publishing on Saturday, but I’m going to be more flexible. I might sometimes do a week day issue, and I’m not commiting to every Saturday, life is too darn complicated at the minute. I’d love to have a consistent schedule, everyone says that’s the best way to increase readership, but it’s just not realistic for me right now. It’s the "if the world allows" attitude, I’m going to try but I’m not going to kill myself over it.

  Ok that’s it, nothing Earth shattering but I’m feeling good about the changes. I hope you like the sound of them.

  Update - Wow I just noticed how many 2s there are in today's date. Hello world! 🤷‍♂️

  TODO add links when world allows

• Component and configuration based UIs

  2022-02-09 08:26:00 +07:00 by Mark Smith

  The past few days, following on from my investigations into the rendering process of modern javascript frameworks, I’ve had component and configuration based UIs on my mind. I’ve been wondering what the big deal is, why all the fuss, when it seems to just boil down to co-locating your templates and code together.

  I kept thinking there must be more to it. But then I thought, what if that really is all there is to it, there are probably other reasons, but what if it is just about co-locating the templates and the code, would that be so bad?

  One major annoyance with templating is making sure all the functions you are using in your templates are in the render context at render time. If any of them aren’t there, the rendering is going to bomb because the template library will do exactly as you have written in your template logic, except some of the functions you told the library to call just aren’t there, and when your template reaches one of those, it results in a runtime error. If you don’t use many functions in your templates, this is likely a non-issue. On the other hand if you do, then it is a pain you face.

  The way it’s done with regular template rendering, is that you have to look down your entire template hierarchy, note down which functions get used, and pass these into the top level template, as part of the data object, and you have to make sure that each time you include a template/partial, to pass the right functions into the render method invocation for that template/partial. So again, if you don’t use many nested templates, it’s probably a non-issue. But if you do, then welcome to code bombsville.

  For people that write highly nested, highly functioned templates, collocating code would perhaps be beneficial, just to make preparation of render contexts less error prone.

  With all that in mind, I decided to just try adding component rendering to my ssg in a feature branch. Worst case is that it doesn’t work and I could just delete the feature branch.

  I thought about it for a few hours, looking at my code, looking at javascript’s implementation of classes, which bundle functions, over and over again. It's still very new code, so I wasn’t totally sure about it, but I think I could see a way to modify the ssg that would result in component rendering.

  It was simple enough, just add a renderComponent method to the renderer, and that method would grab all the functions from the passed component, merge them into the data object, and pass that into the regular render method. Could it be that simple?

  {% highlight javascript %} renderComponent(component, data) { // Add component functions to render context const templateBody = component.getTemplate(); const templateData = Object.assign({}, component.getData(), this.getComponentFns(component));

  return this.render(templateBody, templateData); }

  getComponentFns(component) { return Object.getOwnPropertyNames(component) .reduce(function(acc, current) { acc[current] = component[current]; return acc; }, {}); } {% endhighlight %}

  In the javascript community such implementations are often referred to as syntactic sugar. Might be the same in other language communities, I don’t know (update - turns out it is the same in other programming languages). Essentially the underlying process remains the same, but a new structure is available which is useful in many situations. Examples of this approach are Classes, syntactic sugar on top of prototype based inheritance, and Async/Await which still uses traditional callbacks underneath, but via promises.

  It’s pretty clear to me that if you can write your template alongside the functions it uses, and be sure that at render time, all the functions will be available, then that’s a net positive. And it might very well be that there are other, more frontend centric reasons to use components.

  Anyway, guess what, it worked!

  It seems sort of obvious now, but sometimes when you are in the fog of technology, mental models, implementations, buzzwords, build tools, peoples opinions, and just the complexities, mutilations, humiliations, thirsts, hungers, dirtyness, impossible situations, contradictions, blocking, pain, starvation, cruelty and madness of life, it totally doesn’t seem obvious!

  In the end I added about 10 lines of code, and a very simple Component class, and it worked, the component I included in my template rendered! And since my ssg has a way to define custom renderers, most libraries that support partials/includes should now be able to render components. It's early days and I havent done much testing, but it seems as though it is possible.

  It was quite a moment, when I ran the code, I sort of knew it would work, but it was still a pretty big wow moment. I was like, "wow it worked". That doesn’t happen very often in programming, at least not at that scale.

  Regular template rendering still works as before, and for templates that don’t use lots of functions, that’s probably going to be the better, simpler way to go. But if you do write templates that use lots of functions, using components is a neat way to bundle them with your template, so that at render time the functions are right there in the context, no need to do any special preparation.

  The only current limitation is that your top level can’t be a component, it has to be a template. The ssg renders components inside templates. There might be a way to have top level components in the future, I haven’t figured out an easy way to do that yet.

  One other thing, not related, but related, if there is one thing I have learnt here in Vietnam, it's that the people are cruel, and their cruelty will only increase over time. This is my opinion that has been formed after living here several years. They aren't all cruel, but when all is said snd done the net cruelty only increases. I really feel I have to say that now, while I still am able.

  They should consider renaming it the People's Cruelty of Vietnam. It would be a much more accurate description of the place.

• What’s up with templating in modern javascript frameworks?

  2022-02-08 13:30:00 +07:00 by Mark Smith

  I’ve been writing a static site generator recently. It’s a rewrite of a utility that I use to build my statically generated linkblog. That utility has been working well, but it’s implementation isn’t exactly very elegant. It was, after all, my first attempt at building the linkblog statically, which had previously been a typical MongoDB backed Express app.

  I’ve been really delighted using many of the latest javascript features. Specifically Classes, Array functions (especially map and reduce) and Async/Await. That combination IMO has dramatically improved the development experience. The code is massively easier to reason about, and just looks so much better.

  class Dispatcher {
    constructor(app, cfg, renderers, collections, srcDir, outputDir) {
      this.app = app;
      this.cfg = cfg;
      this.renderers = renderers;
      this.collections = collections;
      this.srcDir = srcDir;
      this.outputDir = outputDir;
      
      this.renderersByName = undefined;
      this.renderersByExt = undefined;
      this.outputPaths = [];
    }
    
    async init(templatePaths) {
      debug(`Initialising dispatcher with templatePaths: [${templatePaths}]`);
  
      // Arrange things for easy selection
      this.renderersByName = this.getRenderersByName();
      this.renderersByExt = this.getRenderersByExt();
      
      for (var i = 0; i < templatePaths.length; i++) {
        // ... do some rendering ...
      }
    }
    
    getRenderersByName() {
      const renderersByNameReducer = function(acc, current) {
        acc[current.name] = current;
        return acc;
      }
  
      return this.renderers.reduce(renderersByNameReducer, {});
    }
    
    // ...
  }
  

  The latest ssg tool now has all the basic features I need to rebuild my linkblog. It uses what I call "old school templates" to render all the pages. In my case it’s EJS, but it could work equally well using any of the many other template libraries like handlebars, liquid, mustache etc. I wanted to get that working before venturing into rendering pages using some of the more "modern" component based javascript frameworks like React, View and Svelte.

  This past week, I’ve been looking at these frameworks more closely, trying to figure out how the rendering works. I’m trying to get a sense of what modifications I’ll need to make to my ssg so that it can generate component based websites. I’ve read so much about these frameworks over the years, it all sounds so wonderful, but also quite mysterious.

  With old school templating rendered server-side, though there are lots of peripheral features, it essentially boils down to passing a template string together with an object containing data to the templating library. The library takes the data and the template string and gives you back an HTML string, a.k.a the rendered page. You write that to a file. Rinse and repeat, upload all the files to your CDN / hosting provider, and that’s your site live.

  debug('Renderer fn - ejs: rendering template');
    
  const renderedContent = ejs.render(
    templateBody,
    context,
    options
  );
      
  return renderedContent;
  

  The template libraries have lots of neat features to make it easier to create your pages. One such feature is the ability to include templates inside templates. You create templates for small pieces that you can reuse across all your pages. On my Jekyll based blog for example, I have among others, includes for the Google analytics snippet, as well as messages promoting my development services that appear on each page. I can update the included templates and the messages update across all the pages that use those templates. The feature is sometimes called partials, it’s been a standard feature of ssg’s for many years. Each library has a syntax for describing the include that you insert directly into the template HTML, and you often additionally pass the include a data object which is used by the library to render the included template.

  <h1>Hello World!</h1>
  
  <h2>Some included content:</h2>
  
  <%- render(includes.helloIncludes, locals) %>
  
  <h2>Some regular content:</h2>
  
  <p><%= locals.loremipsum.content1 %></p>
  

  It’s worth noting that in most implementations you can pass functions in the data object, and use those functions directly in your templates. The libraries execute these functions as part of the rendering of the page.

  The templates contain the logic for creating the HTML files, your javascript code which you use to fetch and prepare template data is in separate files. What about the modern frameworks, what are they doing?

  Well there’s a lot of buzzwords and fancy sounding terminology, but after reviewing several component based projects, it seems to me they are doing essentially the same thing, except the template code has been moved into the javascript files inside a return statement, components are essentially the same thing as includes, and even props, which is one of the key concepts are really just the same as the data object that you pass to includes. All this mystery, and really it’s just another template library. I’m probably missing some crucial detail, but after a few days of digging, that’s what I’m seeing.

  class ShoppingList extends React.Component {
    render() {
      return (
        <div className="shopping-list">
          <h1>Shopping List for {this.props.name}</h1>
          <ul>
            <li>Instagram</li>
            <li>WhatsApp</li>
            <li>Oculus</li>
          </ul>
        </div>
      );
    }
  }
  
  // Example usage: <ShoppingList name="Mark" />
  

  There must be something else that’s useful over and above collocating related code together. At the present moment I personally find that sloping everything together causes me more confusion, because it’s difficult to conceptualise where the boundaries of the app are located. It’s just this vast amorphous everything code, instead of code that’s divided into rendering logic and data preparation logic.

  I’m aware that I’m coming at this from a very server-side way of thinking about things, and probably component based websites make a lot more sense once they are running in a browser environment, with event handlers and the like, but nonetheless, I find these observations interesting, even if no one else does.

  What’s the major advantage that you get from sticking the template into the javascript?

• Static site generator development continues...

  2022-01-01 14:00:00 +07:00 by Mark Smith

  Here are the notes I mentioned in my previous post. You’ll get some idea of what I’ve been up to in my personal development projects, even if it’s not a nicely crafted piece, I’ve made some progress on my static site generator, and I wanted to blog about it. Still blogging... :)

  • Initial version which I wrote as a sort of life raft when the linkblog.io ship was sinking
  • Jamstack, serverless, Netlify, Github Actions, CI/CD, and git repo powered development were all becoming super popular
  • It’s running my current linkblog, it works, it’s all running in the cloud, quite awesome
  • But...the code is kind of fragile
    • quite a lot of duck tape
    • the structure of the code is convoluted in places
    • lots of callback hell
    • hey async/await was new when I wrote it and I hadn’t gotten comfortable with it yet
  • It uses plain javascript objects rather than classes, which is mostly fine, but I’ve seen quite a few implementations of tools that use classes
    • it’s much clearer to me now how to think about and mentally manipulate such concepts
    • there’s a lot of flexibility and structural benefits when you have the right abstractions
    • you can more easily get out of the weeds in some places
    • You can more easily build something that can be refactored to suit your current needs
  • Classes do present a new set of challenges though, but on balance, I feel like they are the right structure to be using a lot of the time
  • I wrote an initial version back before the big lockdown happened, I had been thinking about it for a while and was able to get something working over a few days
  • The main ideas were
    • Just render templates, stick to EJS and Markdown, keep it simple
    • Path base page routing
    • Maybe rendering websites using templates could be similarish to how a render farm operates, something I have a lot of experience from my time in the feature film visual effects (VFX) industry
    • I had been reading a lot about various more formal data structures, and it was clear to me that a queue would be beneficial to handle all the template rendering
    • Just get something working that runs locally, but maybe if the architecture was done right it would be easy to get things working in a serverless environment later
    • Initial specific requirements for phase 1 - Implemented these crucial examples very early, and so can easily check that structural changes don’t break these
      • EJS using data files
      • EJS with includes
      • Markdown
    • Phase 2 implemented after the big lockdown, once again I’d been thinking about the best structure for many weeks and was able to get these changes made over 3-4 days
    • A set of core components have emerged that feel really nice, a structurally sound way to render templates, with possibility to extend functionality
    • Things I have in the back of my mind
      • How to handle config
      • How to handle more advanced template rendering, for instance how to render many different output pages from a single template, with some kind of iteration logic, necessary for rendering the linkblog calendar folder structure
      • Concentrate on EJS, but try to architect a solution where different ways of rendering templates could be accommodated in future, at least try to go in that vague direction even if it’s not a completely working multi-rendering solution at first
      • Create the right abstractions so that changing template rendering input and output locations is trivial (i.e. local file system, S3 like cloud storage etc)
  • Inspiration
    • Jekyll - the ssg I currently use for my blog
    • Eleventy - awesome ssg that uses classes in a really nice way, great community
    • Pixar’s Renderman - the standard for running large scale render farms in the VFX industry
    • Render farms - [memories I have of my time working with VFX render farms, mostly sysadmin/devops stuff, but also some of software development
    • Serverless and Jamstack the shiney future at the end of the tunnel
    • Wordpress, & open source
  • Open source
    • Something I’ve wanted to do for many many years
    • There’s been a lot of Linux in all my previous jobs, I feel like I’ve been living and breathing in open source in one way or another all throughout my career
    • But I must mention though that the closer I get to having something that I feel is good enough to ship, the more ham strung and hands tied behind my back the world around me has me feeling, not so great considering the whole point of open source
    • Which license to choose, MIT, GPL or Apache, or something else
    • How to handle the competing, often contradictory aspects of the intersection between developing software in the open, and having a balanced personal life
  • Somewhat esoteric, feels like I should mention, without dwelling on them for too long at this point in time
    • Why does it feel like the choice of open source license somehow impacts the freedoms I experience in my personal life?
    • Why does it always feel like right after I do one of these 3-4 days development sprints on a personal project, that I get absolutely clobbered by the world around me? It’s like clockwork, happens every single time, I’m not particularly religious, but is this the metaverse of the future, appearing in the present? Seriously, it really worries me, and especially that it’s impossible to talk about without sounding like you’ve lost your marbles
  • The try to stay alive cascade
    1. survival
    2. self-preservation
    3. try not to negatively affect others

• Hi it’s me. I’m still alive.

  2022-01-01 13:20:00 +07:00 by Mark Smith

  I just wanted to say hi and wish a happy new year to all.

  The world is pretty complicated for me at the minute, I ended up in some sort of metaphorical/metaphysical (but in a lot of ways, very real) alive/dead loop in my life, so I just stopped posting because I sensed that it was going to continue for a while, and the thought of what that would look like from the outside had me thinking "No thank you very much".

  I’ve done some development on my linkblog ssg, and I’ve just written up some notes.

  Hopefully it will turn into a blog post, though I think I’m just going to post the unedited bullet points rather than turn it into a beautiful written article. That’s all the world is allowing for me at the minute, but I want to put something out there, because life goes on even if it’s difficult sometimes.

  So once again, happy new year, best wishes for 2022.

2021 #

• The Reddit Account Saga

  2021-04-16 19:30:00 +07:00 by Mark Smith

  I have been a reader of Reddit for many years but hadn’t until late last year posted anything.

  I created my Reddit account sometime back in November 2020. 153 days ago according to my profile page. In my first post, I shared a link to a blog post I wrote about Robust NodeJS architectures. 138 days ago I posted a link to my newsletter in r/javascript and received a private message that it had been removed because it didn’t follow the subreddit guidelines.

  Some time went by and I noticed that my account page appeared strangely when I was logged out. At times it showed a 404 page but other times it rendered a page where my username was replaced with the string ‘undefined’. I figured it was a bug in the rendering of the page, and kept using the account, everything else seemed fine. I could log in, read threads and post to subreddits.

  I continued to occasionally post links to my newsletter, and mostly it was fine though I did receive one other take down notice. I eventually emailed Reddit support several times to ask them about the account, but only ever received automated replies that didn’t solve the problem. It really felt like I was being disappeared, because at this point I started to suspect that none of my posts were being seen by other users.

  I posted on Indie Hackers to ask for advice. I also posted to some help subreddits. It was a landslide diagnosis: I had been shadow banned for posting too many links to my own content. In the process received quite a lot of criticism about how I was posting on Reddit, but also on other sites like Indie Hackers and Hacker News.

  It was quite a blow, I had been sharing my newsletter that I spend many hours every week preparing, and just received criticism in return.

  This wasn’t helped by the fact that the support team wouldn’t answer my emails aside from auto replies. I tried posting to a special subreddit that tells you if your account is banned, but it just didn’t reply anything. People seemed to think this was because I was banned.

  In a separate Indue Hackers thread I had received some advice that Reddit was a good place to look for jobs, but I still couldn’t post. Eventually I decided to post publicly on Twitter and try to get the attention of the Reddit support team. The post never showed up in my timeline, and the only way I found to get it to show-up was to pin it to my Twitter main page. I ended up creating a Twitter thread to chronicle the things I had tried.

  I never received any replies from the support team. Meanwhile my Twitter profile now had a strange Twitter thread attached to it that every job prospect was now reading. Not the best kind of look when you are looking for work.

  Looking at the timeline of events, it still wasn’t obvious to me that my account was banned, because it would have had to be banned since before I ever posted anything, which doesn’t make any sense.

  At some stage I found another Reddit page where you can request that your banned account is re-instated, so I went ahead and filled out the form. After submitting it, the page showed a popup saying that the account was NOT banned.

  I emailed support yet again and this time got a human reply. A few emails later they informed me that my account had been caught in a spam filter, and that the issue had been fixed. I never received any notice from them that this had happened. The account finally started working many months after the issue first appeared.

  Shortly after I posted to a jobs subreddit, my post was immediately removed and I was given conflicting information. I posted again on Indie Hackers to ask for advice. Basically one message said ask the mods, and another said that if I asked the mods I could be banned. I opted to just not use Reddit for job search, and I’ve mostly stopped posting to other subreddits.

  It was a really horrible and drawn out situation that affected my life in many ways, where I was tried, judged, convicted and hung out to dry by the community, where there was no recourse, no procedures to restore my account. It’s really tarnished how I view community and the web. I’ve since had account issues with several other sites, including Freelancer.com, Coinbase.com, big named sites where you would expect account issues not to happen.

  There were a few folks on Indie Hackers that helped and talked to me, interacted on some of my other posts, giving me good advice about my blog, and I’m great-full for their support during very difficult times.

  I’m not out of the woods yet, but I will at least be removing the pinned thread from my Twitter main page later today.

  Here’s hoping 2021 starts to get a bit better, even though the rainy season started a few days ago! There’s nothing better than rain thunder and lightning to remind yourself that you are still alive :)

• Experiments with the newsletter structure and content

  2021-04-09 15:53:00 +07:00 by Mark Smith

  In the last few newsletters I have been experimenting with the structure and content quite a bit. The result is a much better newsletter that is easier to parse, where it’s easier to find interesting content, and where it’s possible to see the context around individual links.

  My newsletter is evolving, slowly but surely, things are starting to take shape.

  The current newsletter structure:

  • Intro
    • News
    • Tutorials
    • Technologies
    • Write-ups
  • Special mentions
  • The best links from the linkblog’s last 7 days
  • Footer

  The last 2 editions have been particularly strong, with a focus on the news section, which is further subdivided into the latest trends, and changes week to week based on what I’ve been seeing.

  Last week for example the trends were Congressional Hearing, Social Media Features, Chips, NFT and crypto currencies getting real, Other Bits and Pieces. These were different to the week before, but there is some similarity, because the topics developed and evolved.

  The other sections are re-occurring and a way to more quickly find content you might be interested in. These categories seem to cover most things, but they might change a bit in the future. There’s structure but I’m not going to be militant about it and will add / remove sections as needed.

  Navigating through a massive link dump is tedious, so the intro section aims to be a sort of meta index, to make it a more pleasant experience finding articles to read. It vaguely mirrors what was going on in my head when I found and posted the link, though that evolves throughout the week, culminating in a synthesising of the topics and trends.

  Generally I’ve been very happy with the news narratives that emerged and also with the sprinkling of interesting javascript, technology and web development articles. I’m still working on getting the balance right, but I’m into the new format.

  The other thing I’ve been experimenting with is to use linkblog hash links in the intro section. The idea here is that I want to somehow make accessible the context around a linkblog link, hopefully over time you’ll get a better sense for how the narratives emerged.

  Since I choose all the links and write the copy, I have a lot of editorial control, but you can see the reasoning and source material behind a particular narrative and so you can more easily judge for yourself what to think on a particular topic.

  Something that has been mentioned is the ‘double jump’, i.e. you have to click twice to get to the article: once to get to the linkblog link and then again to get to the actual article. It’s a little unusual, and there might be a better way to make the context available and have a way to get to the article more quickly. Who knows, it might even become ‘the thing’ that sets it apart from other newsletters.

  Another aspect I experimented with in February was a Javascript Core Special Edition. That was a fun edition to put together.

  As for the future, I’ve really been enjoying Twitter Tweet Threads recently, IMO it’s one of the platforms best features. I want a similar feature! :)

  I’ve been pondering how to update the linkblog to create something similar, a way to group a collection of linkblog links on a timeline. I’ve got some ideas of how this could enhance the newsletter. There are a couple of wrinkles still to iron out, but that might be happening sometime in the future.

  The newsletter is a work in progress, I’d love to hear your thoughts and suggestions via email or in social media comments.

• The evolution of my javascript, technology and web development newsletter

  2021-04-09 15:39:00 +07:00 by Mark Smith

  I started the newsletter back in November 2020, and I’ve published an edition every week since. That’s 21 editions so far!

  In the beginning, I would extract the best links from the linkblog, prepend them with a short single paragraph intro section, usually a bit about what I had been up to that week, and that was it.

  However It was apparent to me that 20-30 links in an email was quite a lot to parse through, so to draw attention to some of the links that had made an impression on me that week, I started to link to them from the intro. This worked well, though over time, the collection of links grew in size, and so did the intro section.

  Eventually, though the content remained high quality, the intro section became a bit unwieldy in length and had a ‘wall of text’ problem. Readers commented that it was difficult to parse and find the content they were interested in. Since the intro was turning into a sort of meta index to the links, I added some subheadings, a few re-occurring ones and a news section that changes based on the latest trends.

  Ideally I would link internally from the intro section to the relevant link further down the page, but Substack doesn’t yet have a way to do internal linking. I contacted their support to ask if it was possible, they have passed on the suggestion to the product team.

  In the interim all the intro links are to the relevant post on the linkblog, which contains any comments I made when I posted it, and the link to the article. By looking at the surrounding links you can also get a sense for the ‘context’ when that link was posted.

  It’s a work in progress, I’d love to hear your thoughts and suggestions via email or in social media comments.

• My approach to software planning and estimation

  2021-02-20 12:22:00 +07:00 by Mark Smith

  While there isn’t a single approach that works for all projects, doing some planning and estimation goes a long way to increasing the chances of a successful project. I wanted to give my general approach and share some useful resources on the subject.

  Here’s the general approach I'm going to take in this blog post:

  • Why planning and estimation
  • Create a high level system design
  • Create a sprint plan
  • Useful tools

  Why planning and estimation

  For smaller projects, a high level overview and a few diagrams might suffice, but for longer projects with more unknowns, it’s important to take more time, filling out a more detailed plan before commencing. At the same time you want to avoid getting stuck in endless details or a plan that isn’t flexible to changing conditions.

  Identifying problematic areas, and finding an appropriate balance between planning and implementation is key, with good communication between the people involved in the project.

  Create a high level system design

  • Overview
  • Project parts
  • Architecture
  • Issues / items that need to be discussed and solved

  See the Facebook architecture planning videos for some good examples of high level system design:

  • Top Facebook System Design Interview Questions (Part 1)
  • Top Facebook System Design Interview Questions (Part 2)

  Create a sprint plan

  General approach is agile scrum methodology. Break down the project parts into subtasks (called stories), add time to each, gather all these into a backlog. These will be completed during time blocks called sprints, which last a set amount of time (typically 1 or 2 weeks).

  During sprint planning assign stories to be completed in each sprint:

  • high level sprint planning for several months at a time, and
  • more detailed sprint plan each week for the next sprint

  In pseudocode:

  Take each project part
  
  - Create stories
  - Add story estimates
      - amount
      - confidence
      - measured
      - accuracy 
  - Add stories to backlog
  - Total all the estimate amounts
  - Update measured & accuracy throughout project
  

  The following project planning article goes into a lot more depth on the subject:

  • Definite guide to estimating software projects

  Useful tools

  There are lots of great agile development tools available, many of which have excellent collaboration features. You should probably use the best available at the time your project is happening.

  Planning is such an important aspect of web development that I have build an agile blog tool that you can use just in case there’s nothing else available.

  It’s open source, uses text files as storage, very unlikely to break, and will enable you to manage a backlog of tasks.

  • Eleventy Agile Blog

• Hash links for individual linkblog items

  2021-02-17 21:18:00 +07:00 by Mark Smith

  If you read the linkblog you might have noticed a change today.

  The linkblog, by the way is where I share links I find online, it’s published every day at midnight.

  Sometimes I add a bit of commentary, other times it’s just the link and page title. It’s mostly links to javascript, technology and web development content, but there’s often other stuff in there too. There’s an RSS feed.

  At the end of the week I do a roundup of the best links and send it out in the newsletter.

  One of the features that I lost when migrating from the old linkblog SaaS (which is sadly no more) to the statically generated serverless linkblog, was the ability to link directly to an item in a day. I’ve added this feature back, so you should see hash links next to each item in the linkblog. The day hash links remain unchanged.

  The item hash links point to the archives rather than on the main page, because the main page only shows the previous 50 days, so it makes more sense to link to the archives, so that the link continues to point to something even after the item drops out of the most recent 50 days.

  The feature is a little different to the SaaS version, it’s less fancy, but since it just uses a standard url hash fragment it works without javascript, which makes it arguably more robust.

  It’s definitely useful, I’ve been experimenting with inserting linkblog items into blog posts as quotes, and having a direct link to an item will make that a lot easier.

  Hash links are standard these days, quite well understood by users, people are used to seeing them on blogs and social media sites. I’ve also chosen a font size, weight and color that blends quite well with the page without getting in the way of the reading experience.

  Though they add quite a lot of extra characters to the UI, I think on balance the page still has the minimalist aesthetic.

  That’s the change, it’s nothing earth shattering, but the main linkblog page doesn’t change very often so I wanted to describe the change. I think you’ll find it a useful feature.

• Cloud Native web application development: what's it all about?

  2021-01-12 07:28:00 +07:00 by Mark Smith

  I've been reading and listening to a lot of content related to Cloud Native web applications recently, so I wanted to gather together some thoughts and links related to the topic.

  I previously built a robust NodeJS SaaS application, a Serverless Books API and some Jamstack Architecture Content Sites, so I have experience in both conventional and cloud based web application programming.

  A lot of these technologies and paradigms are still somewhat in flux, but I think it’s worth adding my perspective to the conversation.

  Here’s the general approach I'm going to take in this blog post:

  • The Cloud Native trend
    • History
    • Benefits
    • Challenges
  • Alternatives
    • Infrastructure
    • Software Architectures
  • Wrapping up

  I'll also be adding in relevant posts from my linkblog. That's where I've been gathering links. There will be a link to that place in the linkblog so you can see the context around that link, in case you are interested, and also a link to the linked article. Those bits will appear as quotes and look like this:

  From my linkblog 12-01-2021:

  Lenovo unveils ThinkReality A3 smartglasses with Snapdragon XR1 - This makes me wonder is VFX artists are going to be really into AR tech, especially all the 3D folks neowin.net

  That's the very latest link on my linkblog as I write this, it doesn't have anything todo with this article, it's just an example to show you what those bits will look like.

  Hopefully by the end you’ll have a better sense of where this trend is heading and thus a better idea of what the future of web applications could look like.

  The Cloud Native trend

  History

  One of the big developer trends for 2021 is the move to Cloud Native applications. Historically applications have been built to run on standard Virtual Private Servers (VPS) hosting, often running a flavour of Linux. This was a step up from having to run applications on bare metal servers, adding flexibility and decreased costs.

  Over the past 10 years hosting providers have expanded their product catalogue from VPS hosting to offering a huge variety of services which can be integrated relatively easily into your application over the internet via APIs.

  Another related trend in web development has been the resurgence of statically generated sites that make extensive use of APIs to carry out backend tasks. That trend is called the Jamstack, which is a broad term to describe websites that make use of javascript, APIs and markup to create sites that are performant, scalable and very secure. Providers like Netlify and Vercel offer file hosting packages bundled with serverless functions, making it possible to build websites without the need for classic VPS hosting.

  Overall the move to the cloud has been gradual. When Amazon released S3 it’s simple file storage offering, it wasn’t at all obvious to most where the industry was heading. At the time, I was working in the visual effects industry at a company where all our tech was hosted on-premises. That was the norm, especially in post production. The idea that you would store files over the internet just didn’t seem realistic. There certainly wasn’t a general awareness of the explosion in APIs that was about to happen.

  The technologies improved, and in recent years there has been broad adoption of cloud technologies across all industries, with application developers modifying their applications to offload certain tasks to these cloud based services. Amazon is the biggest player and has services such as DynamoDB (database), S3 (file storage), Route53 (DNS), Lambda (event driven serverless compute), SNS (notifications) and SQS (message queuing) to name a few. Other large hosting providers such as Microsoft Azure, Google App Engine, aswell as smaller ones such as Linode and Digital Ocean have similar offerings. These services provide highly available and effortlessly scalable solutions to common computing problems.

  Cloud Native is the logical conclusion of this gradual move, with applications being designed and implemented to run completely on cloud services.

  From my linkblog 31/12/2020:

  Software Engineering Podcast - Cloud-Native Applications with Cornelia Davis (Repeat) - Looks at applications architected and built to run exclusively in cloud environments, covers event driven architectures, functional programming, infrastructure as code, Kubernetes, immutability and workloads, cloud failure domains, statelessness, microservices vs monoliths, and the new cloud abstractions such as Lambda and Big Query softwareengineeringdaily.com

  Benefits

  There are several drivers of this move to Cloud Native applications.

  A big one is cost. These hosting providers are running their operations at such scale that they can offer very competitive pricing. For example in 2019 Amazon AWS data centres covered 14 million square feet. They have continued to invest in both hardware and specialist virtualisation software lowering costs and increasing utilisation, and those cost savings have been passed in to customers.

  Related to cost is the move to usage based billing. Whereas with VPS hosting you had to at least pay for an entire server running 24/7, with cloud computing you can pay for exactly what you use. Each service is billed in slightly different ways depending on where the cost is associated. You can for example pay for exactly the storage in GBytes you are using, exactly the volume of inbound and outbound data in Gbits you are transferring, and exactly the number of requests you are making to a service. The services pricing are usually divided up into tiers, and often include a free tier for low volume usage.

  It’s a model that has grown very popular with developers as it can accommodate for large variability in load. Whereas in the past it would have been necessary to manage a lot of infrastructure, all that complexity is handled by the hosting provider. And their infrastructure is the best in town, since they’ve been developing it for many years. It’s as if you had your own highly available, infinitely scalable, robust and flexible infrastructure. It’s possible to build Cloud Native applications very rapidly. You are quite literally building on the shoulders of giants in the industry.

  Amazon is investing heavily this vision of the future of computing, recently announcing it plans to train 29 million people for free on cloud computing over the next few years.

  Challenges

  With all this progress it’s important to step back every once and a while to assess the situation. How resilient and robust is Cloud Native really? Let’s look at some areas that should be considered.

  Well from a technology perspective it is undoubtably very resilient and robust. Most of these platforms are at the stage where a lot of the technologies have been operational for many years, supporting customers from a wide range of industries. They have been fine tuned and battle tested, and as a result are some of the best technology stacks available.

  One technology area that I feel still needs improvement is the developer experience. Since all the services that you are using are cloud based, it becomes a bit of a challenge setting up local developer environments. There are tools and libraries available to create local versions of the services that you are using. For example Amazon has the Serverless Application Model and the aws-sam-cli, similar developer tools exist for other providers, but there isn’t a whole lot of standardisation, so this area can be a bit confusing. I expect the tooling to improve over time.

  Some libraries and frameworks I’ve found useful for serverless development:

  serverless-http - Wraps your entire server framework so you can use it in a lambda function, supports a load of server frameworks including express, koa, hapi, fastify and restify. Useful for creating apps that can run locally and in a serverless function.

  netlify-cli - CLI tool for interacting with the Netlify API, great for deploying to Netlify when you aren't building your site on their infrastructure.

  netlify-lambda - CLI tool for building and running your Netlify functions locally.

  serverless framework - A CLI and web tool to build and deploy serverless architectures, includes supports for Amazon, Azure, Google among others. Makes it easy to configure and provision all the components of your Cloud Native application.

  Another area to consider is long term costs. So far we have seen costs mostly getting cheaper, but that might not always be the case, it’s still relatively early days for the cloud industry as a whole. One example where costs suddenly went the other way is Google Maps. Back in 2018 they raised prices by ~1,400% overnight. Application developers were not happy, but it still happened. Price hikes have happened in the past and will likely happen again somewhere in the future.

  Also it’s worth noting that the pricing structure can have a direct effect on your business that isn’t always obvious. Engineers might start to carry out their duties in a different, non-expected, non-optimal way in order to avoid hitting different pricing tiers. These effects can be difficult to uncover.

  From my linkblog 13/12/2020:

  The author explores the landscape of buy vs build in detail, lots of food for thought, the arguments for building are generally much stronger when it’s a solution that solves something that is core to the business - I’m linking to the HN thread because there are some interesting comments about how vendor pricing models can effect growth in ways that aren’t immediately obvious ycombinator.com.

  As is always the case with the tech industry, there will be mergers and acquisitions, that is a certainty. When that happens there are large shifts in power dynamics throughout the industry. The next 10 years are going to see some big changes throughout the stack with activity already happening right at the hardware level with a move from x86 to ARM chip architectures, most notably with the recent announcement of Apple’s M1 chips.

  From my linkblog 21/12/2020:

  Apple M1 foreshadows Rise of RISC-V - Another piece about the future of chip architectures, it appears like a general move towards ARM cpus surrounded by specialised corprocessors running RISC-V with special extensions to the base instruction set, also discusses the possibility of using RISC-V for the cpu medium.com

  That’s a massive change, and it’s not just for consumer hardware either. There are plans to have M1 chips running in data centres to take advantage of their much lower power to watt ratio, and their new architecture that is very well suited to AI compute tasks. Shifts such as Nvidia attempting to buy Arm are going to continue and their effect will be felt throughout the industry.

  From my linkblog 02/12/2920:

  Amazon unveils 5 new EC2 instance types, the most interesting being the Amazon-made Graviton2 powered instances for compute heavy applications which uses the ARM chip architecture, can deliver a whopping 100 Gbps network speeds at 40% price performance over existing Intel x86 chip architectures - Lots of movement in the chip architecture space at the minute zdnet.com.

  Also worth considering: The forces of government have been stirred in recent years, with the introduction of new rules and regulations for technology companies, especially in the EU with legislation for GDPR, EU’s Article 11 & Article 13, and Section 230 to name but a few. Similar legislation is being drafted in countries all around the world. How these will play out is unknown at this time, but as some of these regulations get passed, there will be pressure on hosting providers to enforce new rules, and that could affect some hosting provider customers.

  Cloud Native architectures are technologically robust, and will enable you to get up and running quickly, but there are aspects of the bigger picture that are worth being aware of. It will be crucial to be ready to pivot if necessary, know where your cost centres are, and perhaps consider building some infrastructure that would enable a move in a worst case scenario event.

  Alternatives

  Infrastructure

  One of the biggest developments in the recent past when it comes to infrastructure is Kubernetes. It was originally written by Google for managing its data centres, and is now open source and maintained by the Cloud Native Computing Foundation. It is a platform for automating deployment, scaling, and operations of application containers across clusters of hosts.

  Containers can be thought of as slimmed down versions of VPS hosts that instead of containing an entire operating system, only contain exactly the operating system pieces that your application requires. With Kubernetes it’s easy to create and configure the resources your application depends on, deploy the latest application code, and ensure that your application is distributed in such as way that it is always available. It automatically does a lot of the heavy lifting that is necessary when you are running on standard VPS hosting.

  Relevant: Software Engineering Podcast - Kubernetes vs. Serverless with Matt Ward (Repeat)

  An interesting aspect is that since Kubernetes is open source and platform agnostic, we are starting to see offerings from all the major services providers. What this means is that if you do need to move between providers it’s a lot easier since the application will run on any Kubernetes cluster, regardless of provider.

  As well as the ability to move, we are starting to see companies investing into building multi-cloud infrastructure. Software such as Cloud Manager, makes it possible for your application to run seamlessly across hosting providers. There will likely be a lot of development in this area as more and more companies seek to build out resilient and robust infrastructure.

  Software Architectures

  Whether you have opted for a classic VPS style infrastructure or the more modern Kubernetes based infrastructure, the question of how best to architect your application will come up. In the early days it’s often best to keep things simple, have minimal layers, use just enough code to get the task complete. You will be learning about the intricacies of your problem domain and having too much code can get in the way.

  Once you start integrating a few cloud services though, and especially if you want the ability to easily switch providers you will want to architect your application in such a way that makes it easier to do that.

  A popular way to do just that is to use Domain Driven Design techniques, an example of this is Microservices Clean Architecture Patterns. This approach uses entities, controllers, adapters and frameworks/devices layers, resulting in an architecture where changing the elements in the outer layers is much easier because you mostly just need to create new adapters, while much of the core business logic will remain unchanged.

  {:refdef: style="margin-top:14px; margin-bottom:14px;"} {: refdef}

  I have had good results using some of these techniques in my Serverless Books API application. In my case the layers are named slightly differently, with handlers, utilities and adapter layers, but the idea is the same. Each layer encapsulates some bits of information so that the core handler functions don’t need to know exactly how the storage has been implemented.

  The core layer handler functions use the addBook, getBook, getAllBooks, updateBook and deleteBook utility functions, passing in the query data, but don’t know any details about how and where the data is stored.

  The utility functions, which are the next level out, know only the name of a table passed in via an environment variable. They use the create, get, getAll, delete and update db adapter functions passing in the query data and the table name.

  The db adapter functions connect to the backend cloud database to perform the operation, do any necessary post processing to the data and return it through the layers back to the core, which eventually sends the data back to the end user.

  Since the db adapter is generic it can be used by any utility function to read and write data to any database table, as long as the utility function passes in the table name.

  Changing to a service hosted by another provider is then just a question of implementing a new adapter, the business logic in the core should remain unchanged, and only the storage identifier (i.e. the table name) might have to be updated to work with the new adapter.

  Thinking about it logically, the most resilient way to develop applications in this way would be to always start by implementing a version of the service yourself, even if it’s paired down in functionality and scale, and only when you have a basic version running, write an adapter for the hosting provider’s version of the service. That way in the worst case you always have something to fall back on, though there would be some degradation to the performance.

  Everyone’s situation is different though, and your circumstances might be such that you don’t have time to go through ultra-cautious route, in which case build out cloud integrations first, and at a later date back track and build your own fallback version of the service. There might even be cases where that approach is quicker overall because you will likely learn a lot about how to build yours by using that of the cloud provider.

  Wrapping Up

  Although it’s early days for cloud computing, even large companies such as the British Broadcasting Company (BBC) in the UK are building huge cloud native infrastructures. It’s interesting to read their high level goals and see how cloud technologies align very well with their mission.

  From my linkblog 03/01/2021:

  Moving BBC Online to the cloud - The engineering team writeup of their recent move from on-premises infrastructure to mostly cloud based where they are using serverless technologies extensively - Very clear articulation of the project high level goals, a description of the layered approach that enables code re-use but also keeps the flexibility to create custom specialised solutions, the re-organisation into teams focussed on page types and common concerns such as development methodology and hosting, interspersed with lots of development principles and guidelines - I have worked on several big projects at the BBC, it’s a staggeringly large organisation, so I am aware of how massive an undertaking this re-architecting of their infrastructure must have been, kudos to all the teams that made it happen medium.com

  However remember they are a very big company, so they will have high leverage with their hosting providers and it’s likely that they will have service level agreements giving them preferred access to the hosting provider’s support staff. And of course they might very well be busy building their own fallback services too. Just because large companies go full cloud doesn’t necessarily mean that it’s the right choice for every business, the trade-offs and peculiarities to consider in the decision making process are going to be different in each situation.

  Cloud Native is a tremendous leap forward for building applications. You will be able to rapidly build out applications that are performant, robust and that can scale to handle any load. Keep in mind the risks, think about resiliency, try to have contingency plans for events that can affect your business.

  The road to building successful web applications is long and windy, and many things can happen over the life of an organisation. You have to make compromises, strategic decisions and there will likely be times when you have to take some risks, but with some thought and planning it’s possible to make that risk more manageable.

• Web development technologies bucketlist for 2021

  2021-01-03 15:10:00 +07:00 by Mark Smith

  It’s the start of the year, so what better way to start the year than to make a list of technologies I’d like to learn and get experience with over the next 12 months:

  Frontend

  • Vue & React - I have done a bunch of tutorials for both of these frameworks and I’ve read the docs, and lots of blog articles, so I’ve got a good idea how they work, but I haven’t had a chance yet to build or work on a full sized production application using these
  • Figma & Sketch - I’ve concentrated most of my web development on backend technologies, and most of the frontend work I have done has been using vanilla javascript and JQuery, with quite a lot of Bootstrap to get the UIs looking reasonable, so I’d really like to learn the basics of a design tool so as to be able to create wireframes and mock-ups, these two appear to be the most popular at the minute
  • Web components - I’d like to build some UIs using the platform’s component framework, so that I can build robust UIs that will last many years without needing much refactoring and upgrades, I have a good idea how they work but haven’t had the time yet to build real world production apps that use them, so I don’t yet have a good sense of when they are the best choice for a frontend project
  • Electron - In the context of web development it's a frontend technology because it would be communicating with a NodeJS backend, but it doesn't necessarily need to connect to a backend, it would be awesome to be able to create Desktop apps

  Backend

  • Elastic search - The text search I’ve done to date has been using the features offered by the MongoDB toolset which work alright, but for really great search, which is something most websites need, the best technology in town appears to be elastic search
  • Kubernetes - Though I’m interested in serverless for some tasks/scenarios where it makes sense, I like to self host applications, usually on standard Linux VPS servers, however the new technology in town is clearly Kubernetes, which solves a lot of the infrastructure provisioning, and workload assignment challenges in a way that can be multi-cloud and well integrated with CI/CD systems
  • Raspberry Pi - I’m super interested in the internet of things space, this could be just as a hobby initially, but I think these is the potential for a lot of interesting web development work in this space, I also have some long term pipe dreams of creating a hardware product at some stage in the future, Raspberry Pi seems like a great place to start

  I might add some more if I think of any in the next few days/weeks. If I do I will post the update to the linkblog which you can follow using the RSS Feed.

  Happy new year and I wish you all the best for 2021!

2020 #

• Building websites and workflows

  2020-12-29 23:54:00 +07:00 by Mark Smith

  I like to be up to date on the latest tech trends, and I read a lot of online tech publications. I post many of the interesting articles I find to my linkblog every day. Some of these trends start to become a bit buzzwordy, like artificial intelligence (AI), augmented reality (AR), blockchain and autonomous vehicles. All these technologies are everywhere these days, and they are great, but I realised today that I’m happy just building websites and workflows. When it comes down to it, that’s what I do.

  The world of websites is massive, with different approaches necessary for both client-side and server-side code. What I love is that both in their own way require creative ways of solving problems. It’s incredible to me how much progress has been made in just the past decade on both sides of web development. New frameworks, libraries, cloud infrastructure, architectures, design patterns, tooling, governance models. It’s amazing.

  And workflows have become so central to what and how we do things in a digital world that we hardly notice them anymore. It’s a very broad category, sometimes it’s just called automation, but the essence is the same, analysing how we are doing things, then streamlining and in some cases creating entirely new processes by stitching together a variety of off-the-shelf and custom software. Whether it’s file based media production, extract-transform-load (ETL) pipelines, continuous integration & continuous delivery (CI/CD) build systems, or infrastructure provisioning systems, there’s an enormous variety.

  The boundary between the two disciplines is fuzzy, with quite a lot of cross-over, and new trends like the Jamstack and static site generators, because you can build websites using workflows! The boundary is an interesting place to be.

  It’s totally possible that I might get involved in some of the buzzwordy trends in the future, but I’m happy creating efficiency, stability, robustness, and growth through technology by building websites and workflows.

• Web design that focusses on text content is the best

  2020-12-24 08:58:00 +07:00 by Mark Smith

  I love text focussed website. There are no annoying consent forms, paywalls or popups. The reading experience is the best. If there is an ad it’s usually relevant and unobtrusive. They load extremely fast, and are great for technical content.

  The best ones:

  • Load extremely fast, and are readable immediately
  • Have a font that is well balanced and easy to read
  • Have spacing between paragraphs, images and lists that looks nice
  • Don’t re-render several times as javascript loads and fonts get swapped out
  • Render really well on mobile devices without needing to zoom or pan the screen
  • Render on mobile devices in such a way that the text takes up the entire screen with just a small margin at the edges
  • Have the right font size that is easy to read
  • Content layout that resizes correctly if you do need to zoom
  • Have regular text that is visibly distinct from hyperlinked text
  • Have tag or category pages making it easy to find other content
  • Have easily findable links to information about the author

  There are some big publications that have these types of websites but lots of smaller sites too. They aren’t always perfect, but that’s okay. It’s actually quite difficult to successfully create such a simple design. I do my best with my blog but I know the reading experience could be improved, perhaps with a different font.

  Anyway I’m going to keep an updated list here of such sites that I really enjoyed reading:

  • Drew DeVault's blog
  • Samizdat
  • Web History by Jay Hoffmann
  • Silicon Angle
  • Openwarr
  • BBC News - Science & Environment

  Sexy design is great, but I find purely text sites refreshing these days.

• The art of the minimal example

  2020-12-18 15:00:00 +07:00 by Mark Smith

  I’ve been putting together a Portfolio of my work this past week. It’s been really interesting re-visiting the web development, workflow/automation and devops/sysadmin projects I’ve been involved with over the years. One thing that stood out was all the minimal examples I’ve built in order to either learn a technology or debug a feature.

  Creating minimal examples is particularly enlightening, it’s actually quite a skill to be able to extract just the code you need to demonstrate a problem you are experiencing. It’s useful because you get rid of much of the complexity of the code you are working on and can focus in on discovering the root cause of an issue.

  Stackoverfkow even has a special minimal reproducible example page which has guidelines on how to create one. For stackoverflow questions you would likely have just a few small snippets of code to demonstrate an issue

  I’ve included a Minimals section in my portfolio, that links to many of the repos I’ve created over the years when I was looking to learn a particular feature or troubleshooting an issue I was experiencing. These aren’t strictly speaking minimal examples in the stackoverflow sense of the term, they tend to be a little bit more involved. In my case they are often small apps that implement just the feature I was exploring or debugging. They get rid of the complexity of the surrounding code, making it easier to reason about. I find myself often revisiting these minimals when I need to implement similar functionality somewhere else.

  The ability to easily create runnable minimal examples I feel is one place NodeJS really shines, especially when you are building Express based web applications. This makes it easier to debug complex problems and communicate your findings to others you are working with.

  Related post: Reasons to use NodeJS for developing your backend systems

• Choosing your web development stack

  2020-12-13 17:40:00 +07:00 by Mark Smith

  This is a bit of an aggregation piece, pulling together several thematically related posts that I think work quite well together to inform on the topic of choosing your web development stack. I’m a NodeJS developer so that’s my focus, but even if you are using a different language, a lot of the information might be useful in terms of building a broader understanding of the web development landscape.

  Make it personal - When you are just starting out, a good place to start is to build your own website. It’s your chance to create your own place on the web. Nice recent css-tricks article that’s gets to the core of why building your own personal site is a worthwhile endeavour.

  Deciding when to build a custom solution in web development - Sometimes referred to as the Build vs Buy dilemma, if you are developing websites, it’s important to be able to focus your efforts in the right place, this article gives you a mental model to help you decide when to build and when to use off-the-shelf products.

  Reasons to use NodeJS for developing your backend systems - Self explanatory title, worth a read if you want to learn why people specifically use NodeJS.

  Robust NodeJS Deployment Architecture - If you are leaning towards self-hosting, this article might be of interest, it’s more effort initially but you gain considerable portability.

  I'm going to keep updating this post with other relevant articles I write or find online and I'll post to the linkblog when the article gets updated, so you might want to add the linkblog RSS to your feed reader.

• Deciding when to build a custom solution in web development

  2020-12-11 13:53:00 +07:00 by Mark Smith

  The web has been around for just over 3 decades now, with the capabilities of the sites we build increasing every year. With that forward movement there’s a corresponding rise in complexity. For website builders this presents somewhat of a dilemma. When you start out, you don’t have the experience to build things for yourself, but you still want to have an online presence. Luckily it’s easier than ever to get started quickly with many website builder tools available. But how do you know when building something custom is appropriate?

  Answering that question when you are deep in the weeds of designing and building a site, can be challenging. I think it’s useful to have a mental model that enables you to step back and see the woods for the trees.

  Building a website is a bit like building a shed. Right off the bat you can choose to buy a prefab shed, and that’s a perfectly good solution. The suppliers of these often have several models to choose from, so if one of them meets your needs, it’s going to be quick to get up and running. There is of course also the option to customise your prefab shed, make it look a bit more like your favourite style. However you might have some very specialist requirements, because of where the shed will be located, or maybe you need some electrical wiring for your home office which you plan to have in your new shed. In those situations it might make sense to build a complete custom shed that fits your requirements exactly.

  It’s similar with software. There are many off-the-shelf solutions that might fit your needs. There are hosted services and also open source frameworks like Wordpress that will make it easy to get a site up and running, and enable tweaks using themes and plugins. In situations where this meets your needs, that’s probably going to be the best route.

  Of course the web isn’t just sheds. The analogy scales up too. There are houses, hotels, community centres, sky scrapers, towns, cities and we can continue scaling up. As your infrastructure grows, your requirements will evolve. You’ll want to create resiliency, by splitting your backend into several components. To some extent you might be able to modify your Wordpress sites to fill these needs. But don’t assume that Wordpress will solve all your problems, it’s totally possible to build a custom Wordpress monstrosity, same goes for any custom software.

  One of the downsides of the prefab solutions that large frameworks offer is complexity. Along with all the useful heavy lifting that the frameworks offer, comes a big increase in the amount of code. If you ever need to get into the code yourself, rather than have a developer modify it for you, it might be a challenge.

  Having a custom solution that is very focussed on solving exactly the problem you are trying to solve, has the possibility to be much much more streamlined, less code, easier to understand if you ever need to get into it yourself. Starting small and growing progressively as your needs change can ensure that you don’t suddenly find yourself in a sea of complexity, overwhelming your efforts to get a site live.

  If you are considering a custom solution, you might want to look at NodeJS. It’s designed specifically for network based applications, and it’s possible to build very focussed low code applications that are extremely performant and easy to maintain. I recently wrote about the reasons for using NodeJS for building your backend systems, you might find that interesting.

  The analogy in this post was inspired by a recent Shop Talk episode. Great podcast that covers frontend and increasingly backend web development topics.

  Hopefully this gives you a bit of a better idea of the software development landscape, and a neat macro way to think about your web development projects.

  This post is part the choosing your web development stack series.

• The coming revolution in freelance web development

  2020-12-09 15:41:00 +07:00 by Mark Smith

  For most of the early web, freelance web development has been the norm. If you needed a website you hired a Wordpress developer and they hand crafted a beautiful website that met your needs. These developers had some backend development skills, but their real strength was frontend development, wrangling the html, css and javascript necessary to manifest your soon to be online property.

  The web has evolved, and the websites people are building are in some cases becoming more complex, they aren’t only concerned with a nice place to present things digitally, but in addition with backend systems to handle various tasks, either through integrations with 3rd party cloud web services or fully fledged SaaS web applications.

  Whereas previously these types of websites and applications were only built by large organisations' in-house developers, the tooling, frameworks and technologies have progressed to the stage where it’s possible to embark on such projects with freelance developers. The world of freelance web development is broadening, and this is a great progression.

  The wider industry still has to mature and reshape to accept this new reality. Developers doing this type of development have been siloed inside organisations, who’s hiring practices are anti-freelancer. This is something you realise pretty quickly as a freelancer when you apply for regular jobs. The recruitment process is structured in such a way that the freelancer has to pay for it. Organisations expect applicants to perform tests and take-home projects in addition to many rounds of interviews, and they don’t pay a penny. It’s a totally unsustainable backwards situation. Large organisations are expecting their recruitment process to be funded by individuals who are often struggling to get by. They are completely shocked when you suggest that they should be paying for your time. This keeps freelancers poor and in-house developers locked-in.

  In my opinion freelancing should be the norm everywhere. Joining a company more permanently should be possible but the norm should be to start as a freelancer and then transition to a full time position. The work force should be more mobile, it should be easier for individuals to move between companies, or to operate independently, and to plant roots when it makes sense to do so.

  The move towards remote work is accelerating this shift. Dan Andrews and Ian Schoen serial entrepreneurs and hosts of the TMBA podcast, in a recent Q&A episode (24:00) spoke about the myth that freelancers are more likely to leave, freelancer and business owner incentives and the improved innovation that freelancers can offer.

  Dan is “Long freelancers”, he thinks the role that freelancers play is being redefined, and that in many ways the interests of business owners and freelancers are more aligned, with long term partnerships growing from initial low friction and flexible freelance engagements.

  I’m on the freelance road, and it’s especially tough in these uncertain times, but I thought it might be useful to others to share my strategy as I move forward. I wrote about it on Indie Hackers. It’s very much like marketing and promoting a product, a multi pronged campaign to highlight my freelance services. I’ve also implemented a job interview policy that I am sticking to. It’s something that I have to do. It’s not sustainable otherwise. I encourage you to take a similar approach that makes sense for you.

  I aspire one day to have an organisation that can fund it’s own recruitment process. One which works with freelancers as a normal way of doing business. I believe new ways of working have the possibility to improve the whole ecosystem. I think it’s an important topic that we should be discussing.

  I would love to hear your thoughts about how you see freelancing will evolve over the next 10 years.

• Reasons to use NodeJS for developing your backend systems

  2020-12-04 21:00:00 +07:00 by Mark Smith

  There are a lot of programming languages to choose from when it comes to writing server-side code. What makes NodeJS a good choice for this task? Whether you are embarking on a new project or extending an existing one, it’s a very relevant question. In this post I’m going to cover the main reasons for choosing NodeJS for your backend application.

  Javascript the programming language runs in two main environments:

  • The web browser which runs client-side javascript
  • The server which runs server-side javascript

  When programmers talk about server-side javascript they call it NodeJS, and when they write a NodeJS application they are actually writing javascript code. NodeJS is more than just the language though, it’s a whole environment (called a runtime) that, among other things, can execute javascript code, but it also provides a way for the code to access hardware like the storage and the network adapters.

  The NodeJS runtime achieves this using an architecture that is optimised for creating network applications. This architecture is called the event loop.

  The event loop looks very much like a collection of queues. As the code runs, anytime something has to happen asynchronously, i.e. that will take a while to complete, the code to be run on completion, called a callback, is placed in a queue, so that the remaining code to be run can continue executing without blocking.

  Asynchronous tasks would be for example writing to storage or making an API call across the internet. The event loop architecture makes it possible for the NodeJS runtime to be single threaded, and we say it’s event driven because the data input/output (I/O) from the hardware doesn’t block the code. Instead of blocking, events are triggered once the result of the I/O operation is ready.

  This means that NodeJS is particularly good for applications that operate over a network, because it can handle many simultaneous requests very easily.

  That’s the big architectural advantage that NodeJS offers, but there are quite a few other reasons to build your backend using NodeJS.

  Here is a summary:

  • Runtime environment is particularly well suited and optimised to network applications, the event loop architecture makes it possible for single threaded code to execute in a non-blocking way, resulting in a high capacity for handling concurrent requests and realtime data
  • The NodeJS foundation is particularly well organised with a very regular release cycle, including long term support (LTS) releases that have a published schedule
  • There are working groups such as next10 and web-server-frameworks that collaborate publically, they have a clear strategic direction for the NodeJS project
  • Npm hosts a vast collection of community built modules, with mostly open source licenses, greatly speeding up development
  • Of all the interpreted languages it is one of the fastest. It uses the v8 javascript engine written in C++ by Google also used in the Chrome web browser, very regularly updated, always being improved
  • There are a huge number of javascript developers worldwide since javascript also runs in the browser. This means finding developers is easier
  • It’s possible to use javascript throughout the entire application stack, from the client-side code that runs in the browser, to the code running on the server, and also with NoSQL databases many of which use javascript as the query language. The elimination of context switching results in massive boost in developer productivity
  • There is a vibrant tooling ecosystem with developers around the world continuously building and sharing the best development tools
  • NodeJS is cross platform, running on Windows, Mac and Linux
  • Using libraries like Electron and Cordova it’s possible to write desktop and mobile apps using NodeJS so you can have a single code base across mobile, desktop and web applications
  • Typically promotes fast development, robust testing and code refactoring
  • All serverless hosting providers have NodeJS implementations, microservices are very often written in NodeJS

  It’s also worth reading the nodejs website about page for more details.

  Companies such as Netflix, PayPal, Trellio, LinkedIn, Uber, eBay, Groupon, NASA, Mozilla, Twitter and Walmart are examples of big tech organisations that run significant amounts of their infrastructure on NodeJS. More details here and here.

  NodeJS makes it possible to build extremely robust network applications quickly and at low cost, that are then easier and cheaper to refactor, extend and maintain.

  This post is part the choosing your web development stack series.

• How to become an official sponsor of the linkblog

  2020-12-04 01:22:00 +07:00 by Mark Smith

  UPDATE (2020-12-20) - This blog post has been turned into a separate page, check out the sponsorships page for the latest.

  It’s now possible to become an official sponsor of the linkblog!

  I’ve spent time working for feature film visual effects companies and an enterprise software startup, and I’ve been running the linkblog for close to 10 years. It's one of the longest running personal linkblogs on the internet.

  The readership is very tech and developer focussed, lots of smart people working for big tech related companies.

  What do you get as a sponsor?

  • Your logo displayed on each linkblog day that you sponsor
  • A special blog post at the end of the week thanking you and linking to your website (for 7 day sponsorships)
  • A special mention in the newsletter that gets sent out on Saturday (for 7 day sponsorships)

  The logo is right under all the links for the day, grouped with the patreon button.

  It will appear on the main linkblog page which displays the most recent 50 days. It will also live on in the day and month archive pages.

  Here’s an example of what it would look like for the month and day archive pages. It will look very similar on the main linkblog page

  If you’d like to become a sponsor then get in touch with me via email.

• What it’s like working for an enterprise software startup

  2020-11-30 08:00:00 +07:00 by Mark Smith

  I wrote recently about my time working in the VFX industry during the height of the analogue to digital transition. I wanted to write another similar piece remembering my time working for an enterprise software startup in the file delivery space, so that I have something to refer to in future, but also because it might be useful to others to get a picture of what it’s like working in an enterprise software company startup.

  First I worked as a Technical Consultant, and then as a Solutions Architect as part of the Global Operations team. I then joined the engineering team as a Software Developer, contributing to the development of a hybrid cloud file delivery SaaS. Having worked with many software vendors as a client in my time working for visual effects companies, it was eye opening being on the other side of the fence, with the folks building, marketing and selling vendor software.

  When I joined they had a head office in Boston (US) and the engineering team in Ottawa, (Canada). Europe, Middle East & Africa (EMEA) was a new region for them, so there was no office. I worked from home in London (UK), or onsite at client locations, everything was done remotely. Initially we had to build the customer base, so there was a lot of travel throughout the EMEA region. There were times where I was travelling to several different countries every other week. I got a taste for being on the road and I liked it.

  The company was pivoting from making source code replication software, to more generalised file movement software and they were focusing on the Media & Entertainment sector because the analogue to digital conversion was resulting in a need to move very large files. I was brought in to help grow the EMEA region and I worked closely with the regional Sales Director and Sales Engineer, meeting with clients, doing technical pre-sales, and later post-sales integration work, architecting and implementing a variety of media workflows.

  I met with people from all levels, from engineers, developers, producers, journalists, managing directors, CEOs, CTOs to name a few. This was within visual effects companies but also broadcasters, channel aggregators, playout centres, network carriers, news organisations, radio stations, advertising agencies, system integrators, other software vendors, cloud providers, and occasionally clients in other sectors like oil and gas and automotive. Having exposure to a much wider section of the media landscape was incredibly interesting. I learnt first hand how these companies moved and transformed files, how they structured their organisations so that people could collaborate effectively, and the challenges they faced in moving to digital.

  Another side that was super interesting was working closely with sales and marketing.

  We had weekly sales calls at the end of the week, everyone in Global Ops would dial into the conference call no matter where we were. This was mostly the sales reps sharing how we were getting on with prospects, if they were on target to hit their number for the quarter, as well as any roadblocks they were up against. Occasionally the Technical Consultants / Solution Architects would be called upon. Us techies also had our own separate weekly conference call for more involved technical discussions. Although there was some competition between regions, we all collaborated together really effectively because we all wanted the entire company to do well.

  At the time it blew my mind that we were all separately moving around while dialled into the sales call on our Blackberrys (and later iPhones). People transiting between clients in cabs, waiting for planes in airports, snacking in hotel lounges, zipping around on trains, or just in their home office, all across the globe, every continent. The first few years our phone bills were pretty insane.

  The language in sales and marketing focussed companies is completely different to what I had experienced before. Everything revolves around the year being divided up into quarters, with Q1 starting in beginning of January and Q4 finishing end of December. It makes it easier to talk quickly about approximate timelines.

  There are a lot of 3 letter acronyms, POCs (proof of concept), SOWs (statement of work), NDAs (non-disclosure agreements), RFPs (Request for proposal), QBRs (quarterly business review), ROI (return on investment), and many more. They make communication more efficient. The other thing there was a lot of was conferences, attending IBC in Amsterdam and BVE in London every year, hosting a booth where we did demos and presentations throughout the day.

  As can be expected, there was a big focus on numbers, on your team hitting it’s quarterly quotas, with extensive use of Salesforce to track and forecast the sales process. It was noticeable to me the motivating effect of having a commission component to my compensation package.

  I was often really impressed with how very complex customer relationships were navigated, with creative ways to move the sales process along, and the formation of strategic partnerships with people and companies with whom we had synergies. It was enlightening to see how much effort was put into pricing, and into relaying customer feedback to the product and engineering teams, and then to later see changes in the product that had been added in response.

  As the EMEA region customer base grew, there was a lot more post-sales integration work. This is where I would work closely with the customer to architect the deployment of the software they had purchased. There was a lot of variety but generally they wanted a robust, secure and highly available infrastructure, and they often wanted to integrate our file movement software with their existing internal systems.

  I wrote custom components in Perl that connected our software to existing systems using APIs or hot folder + XML file integrations. I then used these and pre-existing components in the product's workflow engine to build multi step workflows to automate the processing of inbound and outbound files, sending them to 3rd party tools or to operators and digital artists for manual steps. Sometimes replacing old manual workflows, other times creating entirely new workflows that had never existed or been possible before. I would then install the software and custom workflows, test everything was working, get the customer to sign-off on the deployment and train the users on how to use the system.

  These were some of the workflows I designed and/or built:

  • ProsiebenSat – Ingest via Mx/Agent/Smartjog/Ftp, automated validation then manual validation, checking into MAM, export via Mx/Agent/Smartjog/Ftp
  • Canal Plus - Ingest via FTP Pull/MX, media check, Agility transcoding, import into Sonaps and Avid editors
  • Chellomedia - Ingest via Agent, movement of file bundles, Antivirus check, splitting of bundle on arrive in Lan, delivery of each piece to the right place
  • TPC – Swiss Television - Avid export, transcoding, Sound recording, manual steps offered via Mx GUI, delivery to Playout
  • Media City UK - Avid Gateway - easy ingest and outgest of avid projects
  • Discovery UK - VOD preparation with automatic adding of bumpers, logos and subtitles, and automated quality assurance check

  It was at times very chaotic. I remember completing a pre-sales demo workflow that used AWS SQS to receive files in the back of the taxi on the way to the client demo. I got it working on my laptop over 3G just as we pulled into the carpark. The live demo worked without a hitch. The total number of hours worked per week was very high, it was difficult to separate work and personal life, and the constant travel, though fun, did start to take it's toll on me after several years.

  Overtime we introduced processes, training programs, new staff, we started using a concierge to organise our travel and hotel bookings, we built cloud infrastructure to host many concurrent client POCs on VMs. Things got very streamlined and we were bringing in about a 1/3 of the company revenue.

  I then moved over to Ottawa and joined the engineering team as a Software Developer, working on the team that was building a new SaaS file movement product. It was a nice change of pace to have an office again, and to be on a team of developers.

  We followed an agile software development methodology, working on features in 2 week sprints, with daily standups, spikes, retrospectives and sprint planning. I worked closely with QA engineers and devops to ensure the features were thoroughly tested and deployed without issues, and with the support team to identify and fix bugs. The team was very well organised, and we shipped a lot of features. The product was adopted rapidly with the user base growing to 400000 users, made up of both existing and new clients. During that period I learned a lot about web development, it was a fantastic experience.

  Aside from the occasional merge conflict, things ran pretty smoothly, there weren’t that many emergencies. We had weekly meetings where all the teams across engineering shared what they were working on. It was fascinating to be a part of the development process of enterprise grade software. It was a great foundation in developing web based software as part of an agile team.

  I also experienced what it’s like to live in a very cold climate, with winter temperatures often down to -20 Celsius, having to use special snow tires, and learn how to drive in snow storms and ice rain. We had a team outing to go ice skating on a nearby lake, that was a lot of fun. I went over the provincial border to Quebec, met and spoke with french Canadians. They have a very different accent to people from France. I got to go snowboarding quite a bit and can now board pretty well in either direction.

• Announcing my new Custom Training Service

  2020-11-27 11:44:00 +07:00 by Mark Smith

  I’m adding a new package to my professional services called the Custom Training Service.

  Here’s the description from the services description page:

  The custom training service covers the creation and delivery of custom training on web development and automation/workflow design and implementation. These can be made to according to customer needs and delivered either onsite or remotely via video conferencing tools. Other training topics might be possible.

  I have previous experience developing custom training programs for

  • Onboarding artists at vfx companies
  • Rapidly documenting feature film custom workflows using video interviews
  • Training users on enterprise software products
  • Training users on workflow and automation integrations

  My specialities are NodeJS web development and automation/workflows, but I could also create custom training programs on other topics.

  In my previous work in this area, some of the projects involved onsite visits and consulting where I analysed the existing systems and processes prior to constructing the training program. I then worked worked with designers to create training material that was specifically tailored in look and content to the client needs.

  If you have a need for some custom training, then don’t hesitate to contact me via email.

• Looking back at linkblog.io

  2020-11-26 13:30:00:00 +07:00 by Mark Smith

  I recently announced the end of linkblog.io. I go into a bit of detail in this short indie hackers discussion thread. I wanted to do a quick retrospective to have something to refer to in future.

  I’ve been running a linkblog for close to 10 years. In a lot of ways all the links I’ve posted chronicle my web development journey, but also just life in general. I found it a very useful tool and still do to this day. Somewhere along the way I decided to build a linkblogging SaaS product.

  These were the high level goals of the system that emerged over time:

  • Robust implemenation that does what it was designed to do and does it well
  • Have a minimalist user interface, emphasis on text content
  • Easily scalable to handle growth
  • Deployable to standard VPS hosting
  • Fault tolerant and able to have zero downtime deploys
  • Resilient so that any server could be easily rebuilt from scratch using backups where necessary

  From a technical standpoint I achieved all of these goals.

  I wrote about Robust NodeJS Architectures earlier in the month, and this describes very closely what the linkblog.io infrastructure looked like.

  Some other application level features and capabilities that are worth mentioning:

  • MongoDB sessions support
  • Redis sessions support
  • Rate limiting using Redis
  • JWT API authentication
  • API Caching using Redis
  • Backend job scheduling using Agenda
  • Backend message queues using MongoDB
  • Custom domains
  • SCA compliant billing system (Stripe + webhooks)
  • Production and staging environments

  Implemented using bash scripts:

  • Repeatable server provisioning system
  • Application build and deploy system
  • Backup and restore of certificates and databases
  • Log file backup and cleanup
  • Certificate renewal

  Most of these features were added as a necessity in response to real world events that happened during development.

  Though the UI is quite minimalist and mostly text based, there was quite a lot going on underneath. The system could have been quite easily and safely extended.

  Finally here are some screenshots of the UI:

  

  Landing Page - Header

  

  Landing Page - Description

  

  Landing Page - Features

  

  Landing Page Latest News

  

  User Linkblog Page

  

  User Post Message Page

  

  User Archives Page

  

  User Search Page

  

  User Edit Mode Page

  

  FAQ Page

  

  User Billing Page - Fresh Load

  

  User Billing Page - Add or Update Card Extended

  

  User Profile Page

  

  User RSS Feed

  That was linkblog.io!

• Linkblog.io is dead, long live linkblogging

  2020-11-25 14:55:00 +07:00 by Mark Smith

  Linkblog.io the minimalist link curation tool, the Saas I spent 3 years building, is no longer live. All active users have been contacted and I’ve decommissioned the site.

  Though the site was a tremendous success in terms of meeting my personal learning and development goals, building a robust SaaS from soup to nuts, ultimately I didn’t get the traction as far as user base that I had hoped for. Continuing to maintain the site didn’t make sense because it wasn’t generating any income.

  I’m still linkblogging!

  Update: the linkblog is now located on my main website domain.

  I’ve been experimenting with Jamstack, Serverless and GitHub Actions and have built a prototype linkblog static site generator that I’m using every day. It builds 10 years of content in just a few seconds. It’s totally serverless. It’s pretty sweet.

  I’ve been blown away with the possibilities that the jamstack opens up for creating custom content workflows.

  I’ve also started a weekly newsletter where I share the best posts from the linkblog.

  The general theme is javascript / tech / web development, but I also blend in some other topics from time to time.

  I hope you like it, consider subscribing, it's completely free!

  All the best.

• What it’s like working in tech in the visual effects industry

  2020-11-24 16:25:00 +07:00 by Mark Smith

  I’ve built and supported many workflows for visual effects productions on big blockbuster movies.

  Movies such as

  • Mirrormask
  • Are we there yet?
  • Charlie and the Chocolate Factory
  • Children of Men
  • Harry Potter and the Goblet of Fire
  • The Golden Compass

  And hundreds more like those, working alongside digital artists and producers to build novel ways of collaborating. It’s been a few years now, I’ve since moved into enterprise software and now web development, but I thought it would be interesting to write a piece about my experiences in the visual effects (vfx) industry.

  The landscape of the vfx industry is made up of a wide variety of companies, big and small, that work together to perform the post production on motion pictures. Ultimately they have the same goal - get the film finished on time for a theatrical release. There are often big budgets involved, and lots and lots of people doing a huge variety of jobs.

  If you’re curious about how things are structured watch the credits of any film right until the very end. You can figure out a lot just from how the names are grouped together. Usually it’s the actors first then production staff, then all the post-production staff. But there’s a lot of nuance, with people being divided up into units based around certain pieces of the movie, say if it was filmed in different main locations.

  It’s often surprising how many different visual effects companies are listed, each with their own teams of artists, engineers and producers. Of course animation based movies like the stuff that Pixar makes are nearly all post-production people, also huge amounts of post staff for all the superhero movies, but often films that don’t appear to have many visual effects shots, actually have loads of them.

  The film credits of a movie is an art form in itself, with companies specialising in creating both the front and end rollers. If you stay right until the end of the credits in a movie theatre, a lot of the people remaining at the end, work in post. They want to see how things are structured at other companies, but they also just want to see their names or the names of their friends appear on the big screen. It’s quite a big thrill to see your name rise up across the screen when you’ve spent many months working on something.

  I’ve worked in two different areas of the overall post-production movie pipeline.

  First was as part of the engineering team in a reasonably classic vfx house, where we would work on a handful of movies in parallel, typically working on lots of individual shots, adding effects, then delivering these to companies upstream, who assemble all the vfx shots with the regular shots to create the full length movie. The work tended to happen right after production, or in some cases, at the same time as the movie is being filmed. There were often long days, and weekends, with some people working night shifts. These projects felt big, taking many months, with great wrap parties on completion. They also had a whole department that built old school physical visual effects, robotic monsters and weird creatures. Very cool but a bit scary when you are on the night shift.

  There was a rather big render farm to render the effects onto the images. Hundreds of pizza box style nodes in racks, all managed by Pixar’s Renderman software. Organising the farm in an optimum way was complicated by the fact that 2D and 3D artists had very different types of jobs running. The 2D jobs were generally rather quick taking minutes, whereas some 3D jobs could take hours. You never wanted the 3D jobs clogging up the farm, but you also didn’t want the farm to be empty, and of course everyone’s job is very important. After a long time analysing the software and use cases we came up with a pretty effective configuration for the system.

  There was a heavy focus towards Linux based systems, because it was easier to create multi-machine complex workflows using scripts written mostly in shell (bash/csh/tcsh) and Python. We would setup artists machines using shell environments in a neat way that enabled them to quickly switch between projects, with shot information pulled from a database, so they didn’t have to think about where the data was being stored, that was all automatic as long as they knew which shot they were working on, and everything was versioned. There were lots of cli system tools that for example did things with long sequences of image files.

  For quite lot of the time I was there we had free toast, jams, Nutella, tea and coffee. That was nice. There was also a table football table and we held a yearly World Cup style competition, engineers vs 2D artists vs 3D artist. Good times.

  The second vfx shop I worked for was further downstream, I was also part of the engineering team, receiving the finished shots from all the classic vfx houses. The work happened right at the start of post-production with film scanning and then right at the very end, after all the vfx shots had been completed by the classic vfx shops. We received all the shots and did various tasks such as colour correction, colour grading, beginning and end rollers, language versioning, subtitling, recording back to film, and creation of digital cinema distribution masters. The film scanning / recording and colour management tasks are often referred to as digital intermediate (DI), with the correction and grading being done by artists working with the directors and producers in specially constructed completely blacked out theatres.

  We often had to meet impossible deadlines compounded by the fact that we were right at the end of the post-production process, impacted by any delays that happened at the other vfx houses. Missing our deadlines was just not an option, the penalties for which were in the order of millions of dollars per day, since the release date of the film would have to be moved.

  This happened quite a lot:

  Boss: How long will X take to do?

  Engineers: 3 months

  Boss: Ok we need you to do it in 3 weeks

  Amazingly we always figured out a way to meet the impossible deadlines.

  In this second place there was a much bigger volume of films passing through and typically the work on each film was on the order of weeks, the parties were mostly just for the producers interacting directly with the clients.

  I mention the parties because I think it helps to illustrate that the overall vibe at different vfx shops can be very different depending on the type of work that is done. One isn’t necessarily better or worse, it’s just different. In broadcast media, where I’ve also had some experience, it’s yet another kind of different, more corporate in a way.

  As far as tech goes there was a lot of cool hardware. Super specialist colour management boxes with unbelievably fast infiniband network connected storage, amazing 2D & 3D projectors, top of the line audio setups. The DI part of the facility could be hired out directly to clients, but our in-house artists also used it for the jobs we were working on.

  Finished feature films took up terabytes (12TB for a full length 4k resolution film) of storage space, and the artists often worked on multiple versions of shots. Lots and lots of storage. Anytime new storage was added it got gobbled up pretty quickly. Problems with the storage had a big impact and cost a lot of money because no one could work. Stressful at times but generally there was a good camaraderie between everyone.

  When I was there much of the movement of finished shots was done via USB drives, we had hundreds of them moving around town at any one time. I bet they still use drives quite a bit. Very manual but effective. You do what’s necessary to get the job done. I’ve been in situations where we had to send someone transatlantic on a plane just to hand carry some drives.

  These days a lot of the finished shots are moved between companies digitally. There are high bandwidth networks between a lot of the post facilities. At the time, we were already experimenting with some very involved transatlantic workflows on some big films.where the director was in LA and the colourist was where we were in London. Making sure that the colour science equipment in both grading theatres was setup correctly was crucial. It was also quite fun because we got to work with the US team that was flown in for the duration of the project.

  The film scanners which were about the size of two fridges side by side, were able to scan reels of film at around 8 frames per second, and they had a suite of Perl scripts used for automating the post processing of the all images created, so you could trigger custom scripts to process the scanned images on the render farm or other specialised hardware. Some of the reels of film would come in quite badly scratched, so we setup workflows to automatically digitally remove the scratches using specialised software. The only other way was to do it by hand frame by frame. The render farm there was smaller and ran on very fast blade servers. It was managed by a piece of software called Rush which is written in Perl.

  There was always lots of things happening in parallel, so it was a little tricky not to step on each other’s toes. We were all trying to do this complex thing of creating digital effects all at the same time as constantly rebuilding and upgrading everything around us. If you went away on holiday it was quite usual for your desk to have moved by the time you got back. There were lots of emergencies, the business people were always trying to land new contracts with important clients visiting to see the latest version of their project, a sort of organised chaos. But we sure did make a lot of films.

  The other thing that was happening at the time I was there, on top of the general day to day internal tech chaos, was that the whole wider industry was freaking out about the introduction of digital cinema. Every part of the ecosystem was being written and re-written all at the same time.

  Was it cool? Yeah it was pretty cool.

  I gained a solid foundation in using cool technologies, building infrastructure and workflows, figuring out new ways of collaborating, streamlining existing processes using both off-the-shelf / open source tools, and writing our own when necessary.

• Why are people chopping off their legs with consent forms?

  2020-11-22 11:51:00 +07:00 by Mark Smith

  A recent HN thread was discussing Why Japanese web design is so different. It’s based around an article written in 2013 but both the article and the thread discussion are an interesting read.

  I believe that consent forms pose one of the worst usability problems the internet has ever seen, with big companies that have alternative news reading experiences benefitting immensely from the situation. Their products are slick and unbelievably lovely to use in comparison, but fundamentally non-weblike, featuring only big media brands and with no ability to link to articles.

  I wondered if the US users knew how bad it had become, I asked about the prevalence of consent forms in Japan closing with:

  Probably US users don’t know how bad it is because the US sites only add them to requests originating in the EU.

  A canadian user commented that he didn’t think it was only EU users that saw the consent forms because he is seeing a lot of them too.

  I don't think that's the case, if only anecdotally, as I have seen a huge uptick in these consent modals in Canada.

  Most of these sites needed or wanted them implemented on the cheap. Restricting it to EU customers would require extra work...

  If this truely is happening, it makes no sense whatsoever.

  Why would a website owner that makes money from people viewing their website risk that by blocking users from viewing their website even when they are not legally required to do so?

  Are they living in a strange parallel universe where money doesn’t affect them? (serious question btw)

  There’s must be some crucial bit of missing information here, because it really makes no sense.

• I've started a Newsletter!

  2020-11-21 10:59:00 +07:00 by Mark Smith

  Earlier in the week I met-up with James Clark from Nomadic Notes for coffee. He’s a long time travel blogger and digital nomad known for his excellent city travel guides. Really interesting guy with lots of insight into the travel blogging scene. It was great to meet him.

  Yesterday his newsletter hit my inbox and I noticed that it’s powered by Substack. They’ve recently become very popular for their easy to use newsletter web application. I had coincidentally sent them a job application last week, turns out they use NodeJS. Anyhow I’ve been thinking about starting a newsletter for a while now, and this felt like a good time to do it!

  So after deliberating a bit about which day to send it out, I settled on Saturday midday. Post-lunch BBQ time in Australia, tastey lunch time soup time in Asia, croissant and sausage sandwich time in Europe, in the Americas the midnight feast folks are going to love it, and for the others they’ll have something waiting for them to go with their breakfast burritos, cheerios and pancakes when they wake up.

  It’s also a good way for me to delineate the start of my weekend. Something I’ve been missing for a while.

  The newsletter is a roundup of the best links I’ve posted to my linkblog over the week. The general flavour is javascript / tech / web development but I also mix in other bits and pieces to get a good balance.

  Subscribe for free in just a few clicks, the first issue is going out later today.

• My policy on job interviews

  2020-11-20 12:54:00 +07:00 by Mark Smith

  UPDATE (2020-12-20) - This blog post has been turned into a separate page, check out the job interviews policy page for the latest policy.

  I wanted to describe my policy on job interviews so that I have something to refer to in the future.

  Over the years I’ve interacted with many prospects discovered online and done quite a few of these and I have had a number of bad experiences, which ultimately end up wasting my time and money.

  I’m just one person and I cannot support people for all the myriad of exotic reasons they want to spend their time doing interviews. It’s exhausting and never ending.

  For new accounts I have 1 short call to meet and get a high level overview of the project.

  My policy then is quite simple. My hands don’t touch the keyboard until I am getting paid. Also if you want consulting services, it’s possible, but again you need to pay me for that.

  For clients that want extended discussions, I suggest the consultancy service package which is delivered via video chat. We can verbally discuss code in any way you would like.

  You can send me short code samples to read in advance if you’d like so we have something concrete to discuss.

  What I suggest to clients that want to work with me live is to start with a 1-3 day development service package where you can have the time to work directly with me in a live setting.

  I am open to long term arrangements and would be happy to transition to a more permanent role, but I could also continue on as a freelancer, whichever suits your needs best.

  Check out the service package descriptions and price list for more details.

• Mozilla MDN Docs are going full Jamstack

  2020-11-19 18:24:00 +07:00 by Mark Smith

  Earlier in the year MDN Web Docs turned 15 years old. They have been around since the early days of the web and made huge contributions to it’s evolution. More info on their Wikipedia page.

  But how big are they?

  Let’s look at a few stats:

  • 15 million views per month
  • 3,000 new articles in the last 3 years, 260,000 article edits
  • Grown in double-digit percentages, year over year, every year since 2015
  • Serving more than 15 million web developers on a monthly basis

  They are big!

  They’re also doing a phenomenal job of organising the web’s developer documentation, adding new site features to make learning easier, while building a vibrant community.

  So it’s big news when they decide to re-architect their platform, especially when it happens just after they announced 250 layoffs (1/4 of it's workforce).

  I posted on my linkblog a few weeks ago:

  MDN Web Docs evolves! - the folks at Mozilla are going Jamstack + GitHub for their new MDN docs content contribution workflows, great writeup of the planned architecture, this is definitely an interesting space to keep an eye on, it will be really cool to see the collaboration workflows they build

  Here is a link to the original article on hacks.mozilla.org.

  Their writeup does a really good job of explaining the architecture changes they are making.

  Essentially it boils down to this:

  We are updating the platform to move the content from a MySQL database to being hosted in a GitHub repository (codename: Project Yari)

  And the key takeway:

  We are replacing the current MDN Wiki platform with a JAMStack approach, which publishes the content managed in a GitHub repo

  In the piece they list many reasons including:

  • Making it much easier to add new features to the platform
  • Better contribution workflow, moving from a Wiki model to a pull request (PR) model
  • Better community building by adding discussions, feedback, review and approve steps
  • Simplified frontend architecture improving accessibility

  And they go on to describe the planned architecture in a lot of detail with some illustrative diagrams. Jamstack website architectures are officially where new custom content workflows are being built.

  I think this is perhaps a sign that there are some big shifts happening in web development, where the architectures that have powered the web for the past 10 years are being rewritten, enabling new ways of collaborating together to make digital things.

  Which other large websites will make similar moves?

• Robust NodeJS Deployment Architecture

  2020-11-13 19:17:00 +07:00 by Mark Smith

  The aim of this post is to succinctly describe an effective and robust architecture for self hosting your NodeJS web applications. I’m going to stay relatively high level, describing the technologies, and components, by the end of it you will have a good idea of what such a system looks like. There is a focus on standard well tested pieces rather than the latest shiny cloud / containerisation offerings. It is well suited for running small to medium size applications.

  

  Features of the architecture

  • Runs on standard VPS hosts
  • Possibility to scale
  • Secure
  • Easy to maintain
  • Fault tolerant
  • Low cost
  • Backed up and easy to restore
  • Easy machine provisioning
  • Easy to deploy code
  • Support multiple databases

  3 main components

  • Load balancer
  • Web and API application servers
  • Datastore

  During it’s life cycle, a client web request travels over the internet and eventually arrives at the load balancer where any SSL/TLS connections are terminated, then re-encrypted using self-signed certs and sent to an available application server. That application server performs the tasks it needs to do, persisting information on a shared datastore. Responses are sent directly from the application servers to the client.

  The SSL/TLS termination happens on the load balancer because it makes managing the certificates much easier, with only a single place to renew, create, update and backup certificates.

  Having a load balancer ensures that you can have several application servers running in parallel, which means you can scale by just adding more application servers, but it also means you can reboot servers without impacting site uptime.

  As for the application servers, you can separate out web servers from API servers, but for ease of maintenance you can also just run both on the same machine on different ports, with a reverse proxy on the machine directing the requests to the right application. In this way you have one discrete unit which makes it much easier to add capacity. In the vaste majority of cases this setup is good enough, though could be optimised later.

  Having a shared datastore is key to being able to run the application servers in parallel. This is a single machine that has a large storage volume mounted. It runs all the databases which write their data to the storage volume. The datastore can also run on a clustered set of machines for high availability, though this adds quite a lot of complexity, so initially it’s probably best to run one machine with good backups, so if anything goes wrong you can be restored and running with a minimum of downtime.

  Technologies

  • Nginx - Load balancer and reverse proxy
  • Redis - Key/value very fast database often used for storing sessions and caching
  • Mongodb - NoSQL database
  • Postgres - SQL database
  • Letsencrypt certbot - for generating and maintaining certificates
  • Linux Ubuntu - Operating system for all 3 components
  • Pm2 - NodeJS process manager, runs the applications, handles logging and a variety of other runtime activities
  • RabbitMQ - Message queue software very important for fault tolerant backend systems
  • Mongodb-queue - Message queue implemented via a NodeJS library backed by MongoDB

  Provisioning infrastructure

  You can keep things quite simple in this regard, using a Bash script for each of the 3 main components. The script would need to do the following:

  • Install latest OS updates
  • Install necessary software
  • Configure users and groups
  • Write/update software configuration files
  • Start and stop various services

  These are some of the important Linux items you would need to know about:

  • sshd - server for ssh connections
  • stunnel - creates secure connections, used on datastore for applications without built in SSL - e.g. Redis
  • ufw / iptables - firewalls
  • PKI and creating self-signed certificates
  • logrotate - manage rotating and backing up application log files
  • cron - schedule the running of maintenance scripts like backups
  • certbot - generate and renew certs
  • rsync - securely synchronize files between machines

  It’s likely that your VPS hosting provider has an API and / or command line tools, making it possible to create a provisioning script that creates a VPS server, rsyncs the bash install script to the machine and runs it. So with a minimum of fuss you can provision fresh servers by running a script, so it’s completely repeatable.

  It’s worth noting that there are modern tools that use containerisation like Kubernetes, which are very powerful but can get quite complex.

  Deploying code

  This is another place where a simple bash script can be very effective.

  It would need to do the following:

  • Build your application to a deploy directory
  • Backup currently running app
  • Rsync the files to the application servers
  • Restart the application server

  There is a lot of variety in this area. Many modern workflows that use CI/CD systems use git to clone your entire application repository to the server, rather than rsyncing just the built files. Requirements vary a lot from project to project.

  The bash script route is great for simplicity, but there are often more manual steps involved, especially if your application has complex configuration. In the early days of a project it’s often good enough.

  Backups

  Backups are super important. You need to have all the important files backed up and ideally scripts to restore the backups in the event that a component fails and needs to be restored.

  Consider backing up:

  • Each deployed application version, along with configuration
  • Log files for databases, firewalls
  • Certificates
  • Contents of all databases
  • Configurations for every 3rd party application you are using

  It’s a good idea to use storage from big cloud providers, they are low cost and have good scripting tools.

  Security

  It’s important to configure your machines securely, set firewalls (local and cloud) appropriately. Always use TLS/SSL for inter machine communication. Follow the security advice from the various pieces of software you install, for example creating different users for specific purposes e.g. application access vs access for backups. Only give the minimum of access rights necessary to perform a given task.

  Staging and production environments

  Once the application is running in production, you will benefit a lot from having a staging environment. It’s a replica of the production environment where you can try out new code without being worried to break the live system. Never deploy directly to production, always test it out in staging first.

  Wrapping up

  The infrastructure side of running applications can get quite complex, but there are a lot of advantages to knowing how to construct these setups yourself:

  • Keep costs at a minimum
  • Be in full control of the infrastructure
  • Be able to deploy anywhere

  It’s also worth experimenting with integrating serverless technologies for aspects that are very high load, the low cost and high performance might be worth the portability trade-off, but be aware that a move might require rewriting parts of your application should you need to change providers.

  This post is part the choosing your web development stack series.

• Self-hosted web based tool to get tasks done efficiently

  2020-11-02 15:40:00 +07:00 by Mark Smith

  A short while ago I was wanting to have a self-hosted tool to manage sets of tasks in an efficient way, so that tasks actually get done. I wanted it to be simple enough not to require much maintenance, with minimal chances of something breaking because of software upgrades.

  I built a simple statically generated web based tool that is based on the agile software development methodology, which uses a “backlog” of “stories” that you complete during time periods called “sprints” (usually 2 weeks long).

  Another key idea is that writing things down is a great way to focus on the task at hand and a way to drive your way as you forge your path.

  It uses the Eleventy static site generator to render all the pages.

  Here is the Github repo and there is a demo site with a bit of example data.

  How does it work?

  From the repo docs:

  Use blog posts to describe your work, what you did, what you are about to do, then create "stories" that you add to the "backlog". Assign stories to "sprints", these last 1 week. Flesh out the stories, implement them, and then move these to "done" when you complete them.

  At the end of the week do a retrospective of what you did, and plan (i.e. create and assign stories to the next sprint) for the upcomming week.

  Whenever you are a bit unsure of your path, read the above 2 paragraphs. You probably need to write a blog post.

  Benefits:

  • Self hosted - run it in the cloud or just locally on you machine
  • Everything is a file, versioned using git
  • Create items written in markdown
  • Easily backup your repo using one of the many git hosting providers
  • Never have a migration problem, it’s all just text files
  • Works offline

  This is aimed at personal use rather than for large projects. I think for big projects, especially where you are collaborating with others, it’s probably better to use more comprehensive tools.

• GitHub Actions for custom content workflows

  2020-10-30 19:24:00 +07:00 by Mark Smith

  I’ve spent the past few weeks making some updates to the build system of the latest incarnation of my long running linkblog, now a statically generated website. In doing so I had my first chance to try out GitHub Actions and what I discovered is an extremely versatile tool that makes it possible to create a very wide variety of software developer workflows, centering predominantly around coding activities.

  I believe another area where Github Actions might be very effective is in modern collaborative workflows for content creators. And this is happening now because of the recent resurgence of building static websites in a way refered to as the Jamstack.

  In this post I’m going to describe the pieces of the puzzle, and the types of workflows that are possible, but without a lot of the complicated jargon.

  Github is most well known for hosting repositories, places where developers can collaborate on code. It’s really a web interface for the open source git command line tool, which can be used completely separately, but their interface has become very popular over the past few years, and they are a big reason why there has been a renaissance in open source software.

  Github Actions is their workflow automation tool, and it has a focus on automating tasks in and around repositories. It makes it possible to trigger custom actions when various repository events happen, such as adding new content, or when users have reviewed content, or when discussions mention a certain keyword, and lots more. It’s really very versatile.

  The Jamstack movement is a way to build websites that has become popular recently. It focusses on pre-rendering all the pages of a website so that they are static files, rather than have a server running an application that dynamically renders the pages when they are requested. There is a lot of variations and nuance to what it encompasses but that’s the general idea. A lot of the technology isn’t actually new but it’s the first time this way of thinking about and building sites has had a name, and it’s resulted in some very interesting forward motion in how websites are built.

  There are a lot of benefits to Jamstack sites including:

  • Security because there’s no server to hack
  • Speed because all the pages are pre-rendered
  • Being a naturally a good fit for automation

  And it's this last point that opens up new possibilites for collaborating.

  I’ve been running my linkblog for close to 10 years, there is a lot of content, but it’s possible to render out all the site pages in just a few seconds. I’ve built my own static site generator (SSG) tool in NodeJS because I’ve been able to optimise for that use case specifically, but there are many open source SSG tools out there.

  One of the major benefits of creating a content workflow that uses git, is that you gain all the safety that makes it a great code collaboration tool. By using similar, all be it simpler, workflows to developers you can be quite confident that you won’t loose any work. It’s all just text files, and everything is versioned. As well as safety you get a considerable amount of future proofing because in the end it’s all just text files, so much less danger of software update breakages, and since it’s based on standard git tools you can move to another hosting provider relatively easily. Most of the providers have their own workflow tools but there is some interoperability, Bitbucket for instance can run Github workflows, and making platform specific tweaks isn’t that difficult in a lot of cases.

  As an example, for my linkblog I have some HTML forms built using serverless cloud functions, that I use to easily add new content to a Github repo throughout the day. I have a scheduled workflow that runs at the end of every day and merges in the new content in a safe way using what is called a ‘Pull Request’, often referred to as a PR. This makes it easy to revert the merge if that’s ever necessary. There is then another workflow that detects newly merged content and triggers a site re-build, and then a deploy to the website hosting provider, and the site is live with the new content.

  It’s also possible to use some of the other tools that Github provides as part of the workflow, for example the Issues and Pull-Request pages, to create places where you can discuss new additions with collaborators, as well as setup notifications, and to only publish content that has been approved by a certain number of reviewers.

  It’s not strictly text based content either, as part of my updates I added a podcast to my website. The files are hosted outside of Github, but the GitHub action renders all the podcast pages and creates the RSS feed.

  If you want to listen to some very experimental audio, the show is in iTunes!

  Also if this type of content workflow is interesting to you, feel free to get in touch with me to let me know. I might post more details, tips and tricks about the workflows I am using.

  I’m also available for hire on projects you have. Check out my Github profile.

• Test Post Written Using Github

  2020-02-02 19:45:00 +07:00 by Mark Smith

  Just a quick test to see if it's possible to write a blog post directly in Github.

  Usually when you push to Github using git, it triggers Jekyll to rebuild the static site, but I don't know if that happens if you save the file directly to the repo without a push. Based on this stack overflow post looks like it probably doesn't but figured I would test it out just in case.

  Update 1: It worked! My first attempt didn't work because I forgot the .md file extension, once I got the file name right, the build was automatically triggered. Saving the file now should result in another build which will add this update.

  Update 2: The update resulted in a site build too :)

  Update 3: However I now see that though there is an edit button for files that already exist, there is no "create new file" button in the mobile version of guthub.com :(

• List of remote friendly web development job board websites

  2020-01-19 19:55:00 +07:00 by Mark Smith

  When looking for remote web development opportunities there are broadly two categories - freelancing and remote positions.

  Here is a list of websites for finding remote position opportunities:

  • Weworkremotely
  • Workingnomads
  • Remoteleads.io - Contract and part-time remote jobs
  • Remote.io
  • Angel.co
  • Remoteworkhub
  • Remote.co
  • Remotive.io
  • Skipthedrive
  • Outsourcely
  • Europeremotely
  • Remotelyawesomejobs
  • Landing.jobs
  • Remoteok.io
  • Stack Overflow Remote Developer Jobs
  • Nodesk.co
  • YCombinator - Look for monthly "Who's hiring?" and "Who wants to be hired?" posts

  List of remote friendly companies:

  • Remote Friendly Companies List
  • Remote Teams List
  • Remotive.io Remote Companies

  Also worth checking out is this list of resources for remote workers.

  For freelancing check out my list of freelance web development websites and marketplaces.

• List of freelance web development websites and marketplaces

  2020-01-19 19:51:00 +07:00 by Mark Smith

  When looking for remote web development opportunities there are broadly two categories - freelancing and remote positions.

  Here is a list of websites for finding freelancing opportunities:

  • YCombinator - Look for monthly "Freelancer? Seeking Freelancer?" posts
  • Upwork
  • Freelancer
  • Toptal
  • Gun
  • Freeeup
  • Codeable - Sepcialises in wordpress web development
  • And.co gig List
  • And.co freelance Jobs
  • Gigster
  • MoonlightWork - Has a Stripe integration
  • Authentic Jobs
  • Indeed
  • Guru
  • People Per Hour
  • Craiglist
  • Github
  • Find Bacon

  Also worth checking out is this article that has lots of details about freelance contracts.

  For remote positions check out my list of remote friendly web development job board websites.

• Setting up as a freelance web developer

  2020-01-17 18:48:00 +07:00 by Mark Smith

  There are lots of reasons for doing freelance work, whether it's for what people refer to as a side hustle or whether it's more of a full time freelancing goal, but in all cases it's necessary to have a minimal infrastructure setup. You fine tune this over time and adjust according to the direction of the projects. Broadly speaking the essentials are payments, contracts and some form of marketing.

  I got some of the details from this article from an audio on soundcloud called Web dev freelancing high level overview basic setup. The author makes some good points and is worth the listen.

  Payments

  Freelancers need a way to receive payments. There are a lot of online services that specialise in providing services for freelancers. Freshbooks seems to be a favorite for a lot of web developer freelancers. I hear it mentioned a lot on podcasts. Whichever service you choose, you will need to create an account, connect it to Paypal, and connect it to your checking account. It's a good idea to setup a seperate checking account for your freelancing work than your personal banking account. Then it's a good idea to test out the setup by sending a $1 invoice from Freshbooks, pay and ensure that the money reaches your paypal and then bank account.

  Another possibility here is to use a partly self hosted solution. You will need some web development skills, but you could use the Freelancer project (One of my projects) to host a payment site that uses Stripe as the backend payment provider. You can host the code on a public Github repo, so customers can inspect the code if they want.

  Joan from Toptal contacted me after reading this post to inform me that they have just released a freelance calculator tool that could be useful to easily figure out your hourly rate and yearly income. Thanks Joan!

  Contracts

  You will need some basic freelancer contracts that will clearly set out the project goals and responsibilities, and a way to send these contracts back and forth securly. Docusign has been recommended a lot, but there are other similar services available online, easily findable via Google search.

  As far as the contract google search 'standard exchange of services contract' or 'web development contract' and use one of these as the basis for your contract. Make sure that the contract contains at least the following:

  • State that you will build website, describe the website, pages, api etc
  • State the price for building the website
  • Client will pay 1/3 before, 1/3 midway, 1/3 at the end
  • After contract finishes, will leave project
  • Hourly rate applicable for maintenance after contract finishes, should be arranged as a seperate engagment

  Also worth checking out is this article that has lots of details about freelance contracts.

  Marketing

  Some way to promote your services, this doesn't need to be too envolved initially.

  A basic website with a logo, contact details, short description of the services offered. Something you can add to the footer of your emails, it will be useful for referals too, a way for other to point people towards you. You might want to setup a blog at some stage, but initially a basic 1 page website should be enough.

  How to deliver

  General advice is to deliver a working website using new accounts for hosting and give these to customer as a deliverable. Hosting your customers website is not advised.

  Now you have the basic infrastructure to make proposals, draft contracts, and receive payments.

  Related posts:

  List of freelance web development websites and marketplaces

  List of remote friendly web development job board websites

• New Linkblog feature: rss, atom and json feeds

  2020-01-17 15:21:00 +07:00 by Mark Smith

  The latest linkblog feature is feeds. These are a versions of your linkblog that are easily readable by a computer. This makes it easier for people to read your linkblog but also enables you to do neat things like auto post your links to social media such as Twitter and Facebook. Each time a day finishes where you posted links, the hash link of that day gets added to your feed.

  

  To access your feed, right click on the icon in the toolbar (it looks a bit like the Wifi symbol rotated by 45 degrees) and select copy link address. This should copy the url of your feed to your computer's clipboard.

  If you then paste it into some form of text editor you will see what the feed url looks like. Here are my feeds for my regular linkblog and for my custom domain linkblog:

  https://linkblog.io/users/mark/feeds/daily/rss
  https://links.markjgsmith.com/feeds/daily/rss
  

  The advantge of using your custom domain url is that theorectically if you ever needed to move your linkblog to another provider, then as long as you still own the custom domain, that could be done without the people already using your feed having to update to a new url.

  If you try to load these pages in a browser you'll see lots of strange looking text with HTML tags, but you will also notice that your daily posts are within <item> elements. Computers can read these feeds easily. The vaste majority of the time though you never need to read the actual feed, you only really have to copy and paste the urls of the feed.

  The most common type of feed is RSS so that's what is used in your navbar. Atom and JSON versions of the feed are also available by replacing 'rss' in the url with 'atom' or 'json'.

  A very typical thing people do with feeds is add them to their feed reader, which makes it possible to read many sites from 1 place rather than have to remember to visit all the sites individually. Seen like this, feeds are very similar to the follow feature in many social media sites, they have the advantage of working across many different sites, though they are a bit more complicated.

  Feeds also make it possible to do things automatically with your posts, like posting them to sites like Twitter or Facebook. For example I have setup my linkblog so days are posted to my Twitter account @markjgsmith. There are lots of 3rd party online sites that offer rss-to-[insert favorite social media site] type services.

  For a more detailed description about feeds check out the You Need Feeds site. Also useful is this list of rss feeds for social media sites which might give you some ideas about what is possible.

  If you want to try out running a linkblog then signup for the 14 day free trial, after the trial it's just a few dollars per year.

2019 #

• New Linkblog feature: highlightable messages

  2019-12-08 16:32:00 +07:00 by Mark Smith

  A big part of linkblog is the ability to see the context taround a link, to see the other links that were posted on that day. It’s been possible to link directly to a day in your linkblog since the beginning. But until now, it hasn’t been possible to link directly to a specific message in a day.

  For example here is the link to yesterday in my linkblog. These day urls are copyable from the hash link (#) next to each day.

  The latest feature is the ability to link directly to a specific message in a day. When the page loads from a highlightable message url, the message that is specified in the url will momentarily be highlighted. After a few seconds the highlight will fade back to the regular linkblog page. This strikes a good balance between being able to point directly at a specific message while also keeping the context around the message focussed.

  

  After loading a linkblog using a highlightable message url you will know which message was being referenced but also be able to see clearly the context of that message. Note that you have to have javascript enabled in your browser for the feature to work.

  One challenge implementing this feature was how to do it while minimising the effects on the linkblog page. Adding a clickable element next to each message on the page to copy the url to the clipboard would negatively impact the minimalism and readability of the page. For the moment the way to get a highlightable message url is from the search page.

  Each message in the search page results has a highlightable message url as the hash link. Since most often users will probably be searching for the most recent messages, I’ve updated the search page so that performing a search with an empty search box returns all the messages with latest listed first.

  

  I hope you enjoy the new feature!

• Steady and stable progress

  2019-12-07 14:40:00 +07:00 by Mark Smith

  Along with the linkblog site redesign and more recent foundations building, I’ve made steady progress with other items too, including:

  • Improved useability during signup, activation and onboarding
  • Built foundation for upcoming SSL for Custom Domains feature
  • Added tags to profile page data exports
  • Optimizations for custom domains
  • Database connection improvements
  • Kept 3rd party modules up to date
  • Faq and privacy pages updates
  • Code build and deploy system overhaul
  • Server provisioning system overhaul
  • Refactored analytics for better anonymization
  • Improved security with CSP policy
  • Improved integration and unit tests
  • Frontend optimizations by using compiled templates on client
  • Better data validation

  Because of previous work done to build a staging environment and load balance the site across several nodes I've been able to make all these changes while keeping downtimes to a minimum.

  Here is the status page for the previous month:

  

• Building the foundations for the future of linkblog

  2019-12-06 11:50:00 +07:00 by Mark Smith

  Earlier in the year I did a pretty big redesign of the site which lead to upgrading the style framework which ended up being rather a lot of work, but the site looks great now and still minimalist. The redesign resulted in me building and introducing 3 new core components:

  • Redesigned billing system - Compliance with EU Regulation
  • Scheduler - Periodically runs jobs completely separate from handling website requests
  • Queues - Mechanism to co-ordinate scheduled jobs in a fault tolerant way

  This post gives a bit of description around these components, because although they are not directly visible in the linkblog UI, you might find it interesting to know some of the details of how the site operates under the hood, and maybe give you a bit of an idea of the possible directions for the future.

  If you’ve been following the news you might have noticed that there has been a wave of introductions of new internet regulations all around the world. In the EU they have introduced Strong Customer Authentication (SCA) which affects linkblog because the servers on which the site runs are hosted in the UK. The new rules meant a complete redesign of the billing system and with only 60 days notice. The previous integration with the payment processor Stripe was using their Checkout product, which was simple to setup but doesn't meet the new EU regulations. The re-architected integration is more complex and uses a combination of Stripe Elements for the UI and the new Stripe Intents API which complies with the latest EU regulations.

  The billing system is now active and whereas previously it could only handle 1 type of subscription, it’s now ready to handle multiple subscription types, the first addition to the Linkblog Basic subscription is likely going to be SSL for Custom Domains, the plan is to slowly add new services to compliment the basic functionality.

  As part of the billing system redesign, it became apparent that I needed a way to receive status updates from Stripe called webhooks. Handling the webhooks is non-trivial because an acknowledgment has to be returned immediately, the same webhooks sometimes get sent multiple times, the order isn't always guarantied, the task necessary to process the webhook differs depending on the type of webhook received and processing tasks take varying lengths of time to complete.

  The strategy I settled on was to record the incoming webhooks in the database, respond to Stripe, and have a scheduler component run jobs to periodically process the recorded webhooks. I had to restructure the app quite a bit to be able to handle regular website requests and also separately run scheduled jobs. This is a good strategy but problems occurred because when a cluster node was rebooted, there was the possibility of jobs failing midway through and not completing.

  To make the setup fault tolerant I had to build a queuing system to help orchestrate the scheduled jobs. The queues are hosted on a separate machine to the website cluster nodes, and the scheduled jobs which periodically run on these read and write messages to the queues during execution. Queues are useful because if a website node goes down midway through a job, another node can automatically pick up the job and finish the processing when it sees the item in the queue didn’t complete within a certain timeout period.

  So a webhook arrives, is recorded in the database, and scheduled jobs run and add messages to the appropriate processing queue. These queues are monitored by more specific scheduled jobs running on the nodes which take new items off the queues and carry out the appropriate processing tasks, for example sending out email notifications.

  The cool thing about queues is that they can be used for lots of tasks that don't need to be part of handling website requests. I’m hoping to migrate a bunch of tasks like data exports and generating custom domain certificates to use the queuing system in the near future. Being able to off-load tasks to the queues will keep the site performing well.

  The new billing system, the scheduler and queues are 3 core components that are fundamental going forward.

• Linkblog new look

  2019-12-04 20:00:00 +07:00 by Mark Smith

  Linkblog has had a site redesign!

  The redesign was actually done and released several months ago, but there were quite a few changes needed after the redesign so this is the first chance I’ve had to blog about it.

  Part of what I learnt from the launch on Indie Hackers was that people really wanted a way to try out the software before committing to a subscription. So as part of the redesign I also updated the signup process so that the initial signup creates a 14 day free trial (no credit card required).

  

  

  

  The redesign itself is based on one of the free bootstrap themes from Themes For App, I customised one of the themes to create the main landing page, signup and signin pages.

  

  The rest of the site needed a bit of a refurbish so I decided it was a good time to upgrade the style framework I use called Bootstrap from v3 to the latest v4. I was able to get rid of some rather crufty old frontend code and replace it with new bootstrap components that use solid and modern CSS. Visually the site is a lot cleaner, the most obvious change is the toolbar along the top of the page which has changed from solid black to a lighter grey, which I think along with the other improvements in the latest Bootstrap results in a site that is nice to use and keeps with the minimalist ethos.

  One big issue was that the latest Bootstrap broke the site content security policy (CSP) which ensures that only the right 3rd party libraries get loaded into the pages. There was a related CSP issue on the repo and with the help of the commenters I found a workaround to make sure that the site is still secure. According to the bootstrap devs, the CSP issues will be addressed in v5.

  Little did I know that the redesign was only the beginning of a long voyage of discovery, though there wasn't a whole lot of actual movement so it's more a voyage of the mind and keyboard. I'll be following up this post with a summary of the recent major changes in the linkblog foundations.

  I hope you like the new look!

2018 #

• Linkblog featured on 10words

  2018-08-14 12:38:20 +07:00 by Mark Smith

  Linkblog is featured today on 10words.io, currently on the homepage and will also go out via their newsletter and twitter!

  To all the 10words readers - thanks for stoping by!

  {:refdef: style="text-align: center;"} {: refdef}

  A good example of a linkblog is my linkblog which I also publish using the custom domain feature so that it nicely fits in with my other web presences.

  Some other examples of use are posting links to blog posts and finding a video you watched last year and the latest feature addition is the navbar globe icon. Running a linkblog is a great way to build some context around what you are doing.

  This blog post wouldn't be complete without a coupon code. If you've read this far then you're at least curious, if you use coupon code 10WORDS at check out get 30% discount on your first year, and it's a 30 day money back guaranty.

• New Linkblog feature: navbar globe icon

  2018-08-09 14:11:17 +07:00 by Mark Smith

  The navbar globe icon feature adds a way to link to your other online sites. To set it up, simply update the url form field on your user profile page. Then a globe icon will render in your public linkblog's navbar that loads that url when clicked.

  It's a minimalist way to link to your other sites from your linkblog, so visitors can find your other presences online. In the screenshot below notice the globe icon in the top right just next to the search icon, and in my case it links to my homepage, which has links to all my other online sites, so visitors have a way to find among others my blog, twitter, linkedin.

  See it in action here.

  

• The dead simple todos system

  2018-07-07 20:51:06 +07:00 by Mark Smith

  Over the years I've tried a variety of sofware solutions to todo lists but I always find that eventually I end up just opening a empty file and typing a text list. It's just so straight forward. Almost as if writing a list in a notebook, which of course is the ultimate todo list solution.

  Anyhow I've been adding quite a lot of aliases to my dotfiles recently and I wondered if I could add just a few that would make the bare essentials of a todo list system. These are the aliases that I came up with:

  {% highlight bash %} alias 'slugd=date +%Y-%m-%d' alias 'todos=cd $TODOS_DIR' alias 'tdf=echo $TODOS_DIR/$(ls $TODOS_DIR | tail -n 1)' alias 'tdt=echo $TODOS_DIR/$(slugd).txt' alias 'tde=e $(tdf)' alias 'tdd=echo "### $(date "+%A %d %B, %Y") ###"' alias 'tda=cat $(tdf); echo' alias 'tdc=cat $(tdf) | grep "[x]"' alias 'tdi=cat $(tdf) | grep "[ ]"' alias 'tdn=TODOS=$(tdi); ! test -f $(tdt) && tdd > $TODOS_DIR/$(slugd).txt && echo >> $(tdf) && echo "$TODOS" >> $(tdf) && tda' {% endhighlight %}

  It's all very standard shell scripting, the aliases get loaded from .bash_aliases or similar. The only complicated one is tdn which only creates the new todos file if one doesn't already exist for the current day to avoid accidentally overwritting an existing list.

  • slugd - create slug using date
  • todos - fast navigation to $TODO_DIR
  • tdf - (file) prints latest existing todo file path
  • tdt - (today) prints file path using todays date
  • tde - (edit) opens latest todo file in editor
  • tdd - (date) prints todays date nicely formated
  • tda - (all) prints all todos from latest todo file
  • tdc - (complete) prints all completed todos from latest todo file
  • tdi - (incomplete) prints all incomplete todos from latest todo file
  • tdn - (new) new todo file extracting all incomplete from previous

  In practive the only aliases you actually use are tdn, tda and tde. That's it just 3 aliases to remember and it's pretty close to using a notebook. The only configuration necessary is to set TODOS_DIR environment variable somewhere that gets loaded by the shell automically like your shell's .bashrc file.

  Here is what a todo list file looks like:

  {% highlight bash %} ~ $ tda

  Saturday 07 July, 2018

  [ ] Install new theme on blog [ ] Deploy live keys to payments pages [ ] Troubleshoot failed mail deliveries [ ] Add links to freelancer github repo on payment pages [x] Troubleshoot github remotes issue on markjgsmith.com [x] Troubleshoot freelancer left pane image centering issue [ ] Re-organise dotfiles and dotfiles local [x] Add todo aliases to dotfiles [ ] New blog post: The dead simple todos system

  ~ $ {% endhighlight %}

  The real test of course is tomorrow morning when I create a new todo list.

• Commands I ran to install rbenv and upgrade ruby versions

  2018-07-05 18:03:58 +07:00 by Mark Smith

  I was able to upgrade ruby versions. These are the commands I had to run. I will spare you the error messages and detours.

  {% highlight bash %} ~ $ ruby -v ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin14]

  Commands to install rbenv and latest ruby

  brew doctor brew update brew install rbenv echo export PATH=$HOME/.rbenv/shims:$PATH >> $HOME/.bashrc brew upgrade ruby-build rbenv install --list rbenv install 2.5.1 echo 2.5.1 > ~/.rbenv/version rbenv rehash rbenv versions gem env gem install bundler

  ~ $ ruby -v ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin15]

  Re-install the gems in the blog installation directory

  bundle install bundle show jekyll bundle exec jekyll serve {% endhighlight %}

  These are essentially the commands I ran, roughly in that order but it got a bit confusing somewhere in the middle.

  I decided not to run rbenv init, the only mandatory thing it does is add the shims folder to the PATH. So I just did that myself.

• Ruby jargon to Nodejs jargon translation

  2018-07-05 15:11:41 +07:00 by Mark Smith

  I've been making quite a lot of modifications to the blog recently. The blog is generated by Jekyll a tool written in the ruby programming language. I have been seeing a lot of warning messages about ruby versions during builds, so it's time to try to figure out what's going on.

  Everytime I've ventured into this area it's been full of confusing articles and definitions that don't seem to quite match up. I found an article comparing the rvm and rbenv ruby version managers, which along with my browsing various ruby sites and blogs and quite a lot of head scratching resulted in the followng translation which might be of use to other nodejs developers:

  • ruby === node
  • rubbies === versions of node
  • RubyGems.org === npmjs.com
  • gems === node modules (packages hosted on npmjs.com)
  • gemsets === node_modules folders (there can be many)
  • Gemfile === package.json
  • gem === npm
  • bundler === yarn
  • rvm === nvm
  • rbenv === simpler rvm

  Something to be aware of is that rvm apparently also overides the system cd command to automatically set ruby on directory change which might not be desirable.

  I don't claim that the list of correct, it's just what I've come up with so far. Email me with any corrections.

• Description of my freelance NodeJS and automation software services

  2018-07-04 23:18:00 +07:00 by Mark Smith

  Update - This is the blog post that constituted the original services description, it is no longer being updated and has been replaced by the services page.

  The consultancy service is very straight forward, it covers any aspect of building cloud web applications and is delivered via Skype call. It is customizable to the clients needs but generally covers architecture, building and deployment as well as things like efficient setup of developer environments.

  The infrastructure service deliverable is a running cloud based system ready for web application deployment. There are lot of tweaks necessary over and above the vanilla Ubuntu install that your cloud provider offers, such as machine access, cloud provider environment setup, firewall rules, logs, 3rd party applications installation and configuration, user accounts, aliases, SSL certificates, cron jobs to name a few. This services covers all these and does it at 3 complexity levels depending on the size of your installation.

  The maintenance service is aimed at maintenance and support of a web application infrastructure. It should be used for upgrades, improvements and customizations to an existing infrastructure.

  The development service covers the development of NodeJS based web applications and could cover any type of application. It also covers the development of software based automation and workflows. I recommend doing some consultancy services sessions first to determine the application specifics before starting a development services engagement.

  The custom training service covers the creation and delivery of custom training on web development and automation/workflow design and implementation. These can be made according to customer needs, could involve a systems discovery and consulting phase, and delivered either onsite or remotely via video conferencing tools. Other training topics might be possible.

  The writing service is aimed at internet publications that wish to create content for their websites. It could cover any aspects of web development, and might be in the form of a tutorial or perhaps an essay expressing an opinion on a particular aspect of modern technology. The details could be fleshed out via a short consultancy service.

  The pricing page has the full list of the packages for each service. If you would like to purchase some software services or products visit the payments page.

• How I use my linkblog - posting a link to my latest blog post

  2018-06-30 19:52:00 +07:00 by Mark Smith

  Situation:

  Earlier I wrote a how to blog post. It would be nice to add it to my linkblog timeline in case anyone stumbles across my linkblog. Then they might get a better idea of how a linkblog might be useful. It might also be interesting to others that are building software and writing documentation to see my flow.

  Solution:

  I open the post in my web browser, load the blog post I just wrote and click the bookmarklet in my browser bookmarks bar. I added the popup bookmarklet to my browser bookmarks bar when I setup my linkblog account.

  

  The popup window appears and I add the text "New Post:" to the start of the message text. When I am linking to a blog post I wrote, I always add this prefix so that I can easily search for them later, and it also draws a bit of attention to the link for people that might read my linkblog.

  

  I click on the 'Meta' tab and add some tags. While typing the tags, a drop down appears under the tag box to suggest tags that I have previously used in posts. I click the tag in the drop down and it autocompletes the tag in the tag boz or just continue typing and hit enter when I am done typing the tag. Tags can have spaces in them but no underscores.

  I jump back to the main tab and click the "Post Message" button. The window disappears as if it was never there. I open my linkblog and find that a new item has been posted linking to the blog post.

  

• How I use my linkblog - finding a git tutorial I watched last year

  2018-06-30 18:17:17 +07:00 by Mark Smith

  Situation:

  I need to commit some changes to my dotfiles, when I do a git diff I see that there are two distinct changes that have been made because I must have forgotten to check the previous change in, no doubt I was distracted by something much more impotant at the time. The world is like that sometimes. Other people are just so me me me sometimes. :)

  I could check these commits in one commit, it's no big deal, they are my dotfiles in any case, it won't matter really. On the other hand I know there is a git command just for this called patch, but I don't use it very often and I don't remember the flow. However I do remember that I watched a tutorial on youtube but it was a long time ago, at least a year. I did post a link to it on my linkblog because it was quite a good video. Might as well learn it now, plus I can write a how to blog post.

  Solution:

  So I open up my linkblog on the search page. I search for: git patch "youtube.com"

  

  Adding the url in quotes returns exact matches for the url. I hit the search button and receive a load of results. I use the browser in page search by doing ctr-f and type 'patch' in the browser search box window that opens in the top right of the browser. I hit enter and all occurrances of patch are highlighted in yellow. Hitting enter a few more times and the focus jumps down the page and BAM there is the tutorial: "Intro to git patch mode tutorial". As an added bonus I notice that there is another video I posted about git patch right above it. Cool!

  

  I click the cmd-click the domain at the end of the line and a new browser window tab opens up loading the youtube video. Oh so THAT's how to git patch!

  Here is a link to that day in my linkblog timeline. The video was by a chap named John Karey. Thanks for the video John.

• Things I learnt as a solo developer building Linkblog.io

  2018-06-26 13:08:17 +07:00 by Mark Smith

  Building a web application as a solo developer ain't easy. There are an insane amount of things that need to be done and an almost unimaginable amount of decisions that need to be made...but it's possible.

  I wanted to take a few minutes to reflect on some of the things that I've learnt along the way:

  • selecting the right technologies
  • setting up consistent development environment
  • using unix/linux tools effectively
  • building a deployment pipeline
  • architecting the app
  • building a scaleable infrastructure
  • testing, linting, logging, debugging
  • securing components
  • scripting and automation
  • setting up server side and client side analytics

  If you are a solo developer / small team just starting out on your building journey, feel free to get in touch tell me a bit about yourself. All these things are very fresh in my mind right now, and I am available for consulting gigs! Investing a bit of time and money now might save you weeks if not months of headaches later down the line.

• Launch on Indie Hackers

  2018-06-26 13:08:17 +07:00 by Mark Smith

  I unnoficially launched Linkblog.io yesterday on Indie Hackers. It's been a long road with a seemingly endless onslaught of showstoppers, but the site is up and running and it's built with a strong architecture, and running on a stable infrastructure with the possibility to scale if necessary.

  

  There was modest amount of hits from around the globe and I even got a comment, which was at least partially positive. After a few hours I realised that I had no way to differentiate between server-side side and client-side analytics. Oh noes!

  

  So today I setup some new views and filters using custom dimensions to show each type of data. Things are looking much better now. :)

  If you're intersted in signing up, there is a promo code on the indie hackers post, your entire first year for the price of a few cups of coffee.

• How to get an old jekyll blog active again

  2018-06-10 17:27:00 +07:00 by Mark Smith

  It was actually pretty straight forward. Check that the git remote is still configured, install the jekyll software, follow the instructions in the error messages. I had the dev version of the site back up within a few minutes.

  {% highlight bash %} $ cd $WEBSITES_DIR/blog.markjgsmith.com $ git status On branch master Your branch is up-to-date with 'origin/master'. nothing to commit, working tree clean $ cat .git/config | grep -A 2 remote\ "origin" [remote "origin"] url = https://github.com/mjgs/mjgs.github.io.git fetch = +refs/heads/:refs/remotes/origin/ $ which jekyl $ gem install jekyll bundler $ bundle exec jekyll serve Could not find RedCloth-4.2.9 in any of the sources Run bundle install to install missing gems. $ bundle install $ bundle exec jekyll serve Configuration file: [WEBSITES_DIR]/blog.markjgsmith.com/_config.yml No post given to analyze. Try with -h Source: [WEBSITES_DIR]/blog.markjgsmith.com Destination: [WEBSITES_DIR]/blog.markjgsmith.com/_site Generating... done. Auto-regeneration: enabled for '[WEBSITES_DIR]/blog.markjgsmith.com' Configuration file: [WEBSITES_DIR]/blog.markjgsmith.com/_config.yml Server address: http://0.0.0.0:4000/ Server running... press ctrl-c to stop. {% endhighlight %}

  I'm still a little fuzy on how to add posts. I tried to login to prose.io but the site wanted full access to all my repos on Github...a little excessive. Oh well editing in vim is good enough.

  Last but not least push the changes to github...

  {% highlight bash %} git add * git commit -m "New post: How to get an old jekyll blog active again" git push {% endhighlight %}

2015 #

• Minimal Setup for Blogging with Jekyll

  2015-10-25 18:40:00 +07:00 by Mark Smith

  This initial post is to document how to configure the base Jekyll installation so that it's ready for blogging, with posts displaying on the main page, an archives page that lists all the posts, an about page for a personal description and social media info in the footer.

  

  The actual Jekyll installation is covered in the docs. It's pretty straight forward. Setting up free hosting with Github is covered here.

  For details of the modifications I made to the vanilla install have a look through the commits in the Github repo up to this commit.

  

  I'm using Prose.io to edit posts in my web browser. Prose knows about Jekyll so you can create drafts and publish posts, it's also open source.

• Welcome to Jekyll!

  2015-10-23 20:05:48 +07:00 by Mark Smith

  You’ll find this post in your _posts directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run jekyll serve, which launches a web server and auto-regenerates your site when a file is updated.

  To add new posts, simply add a file in the _posts directory that follows the convention YYYY-MM-DD-name-of-post.ext and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.

  Jekyll also offers powerful support for code snippets:

  {% highlight ruby %} def print_hi(name) puts "Hi, #{name}" end print_hi('Tom') #=> prints 'Hi, Tom' to STDOUT. {% endhighlight %}

  Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo. If you have questions, you can ask them on Jekyll’s dedicated Help repository.