markjgsmith

2020/12/31

2020/12/30

2020/12/29

Building websites and workflows

I like to be up to date on the latest tech trends, and I read a lot of online tech publications. I post many of the interesting articles I find to my linkblog every day. Some of these trends start to become a bit buzzwordy, like artificial intelligence (AI), augmented reality (AR), blockchain and autonomous vehicles. All these technologies are everywhere these days, and they are great, but I realised today that I’m happy just building websites and workflows. When it comes down to it, that’s what I do.

The world of websites is massive, with different approaches necessary for both client-side and server-side code. What I love is that both in their own way require creative ways of solving problems. It’s incredible to me how much progress has been made in just the past decade on both sides of web development. New frameworks, libraries, cloud infrastructure, architectures, design patterns, tooling, governance models. It’s amazing.

And workflows have become so central to what and how we do things in a digital world that we hardly notice them anymore. It’s a very broad category, sometimes it’s just called automation, but the essence is the same, analysing how we are doing things, then streamlining and in some cases creating entirely new processes by stitching together a variety of off-the-shelf and custom software. Whether it’s file based media production, extract-transform-load (ETL) pipelines, continuous integration & continuous delivery (CI/CD) build systems, or infrastructure provisioning systems, there’s an enormous variety.

The boundary between the two disciplines is fuzzy, with quite a lot of cross-over, and new trends like the Jamstack and static site generators, because you can build websites using workflows! The boundary is an interesting place to be.

It’s totally possible that I might get involved in some of the buzzwordy trends in the future, but I’m happy creating efficiency, stability, robustness, and growth through technology by building websites and workflows.

2020/12/28

2020/12/27

2020/12/26

2020/12/25

2020/12/24

Web design that focusses on text content is the best

I love text focussed website. There are no annoying consent forms, paywalls or popups. The reading experience is the best. If there is an ad it’s usually relevant and unobtrusive. They load extremely fast, and are great for technical content.

The best ones:

  • Load extremely fast, and are readable immediately
  • Have a font that is well balanced and easy to read
  • Have spacing between paragraphs, images and lists that looks nice
  • Don’t re-render several times as javascript loads and fonts get swapped out
  • Render really well on mobile devices without needing to zoom or pan the screen
  • Render on mobile devices in such a way that the text takes up the entire screen with just a small margin at the edges
  • Have the right font size that is easy to read
  • Content layout that resizes correctly if you do need to zoom
  • Have regular text that is visibly distinct from hyperlinked text
  • Have tag or category pages making it easy to find other content
  • Have easily findable links to information about the author

There are some big publications that have these types of websites but lots of smaller sites too. They aren’t always perfect, but that’s okay. It’s actually quite difficult to successfully create such a simple design. I do my best with my blog but I know the reading experience could be improved, perhaps with a different font.

Anyway I’m going to keep an updated list here of such sites that I really enjoyed reading:

Sexy design is great, but I find purely text sites refreshing these days.

2020/12/23

2020/12/22

2020/12/21

2020/12/20

2020/12/19

2020/12/18

The art of the minimal example

I’ve been putting together a Portfolio of my work this past week. It’s been really interesting re-visiting the web development, workflow/automation and devops/sysadmin projects I’ve been involved with over the years. One thing that stood out was all the minimal examples I’ve built in order to either learn a technology or debug a feature.

Creating minimal examples is particularly enlightening, it’s actually quite a skill to be able to extract just the code you need to demonstrate a problem you are experiencing. It’s useful because you get rid of much of the complexity of the code you are working on and can focus in on discovering the root cause of an issue.

Stackoverfkow even has a special minimal reproducible example page which has guidelines on how to create one. For stackoverflow questions you would likely have just a few small snippets of code to demonstrate an issue

I’ve included a Minimals section in my portfolio, that links to many of the repos I’ve created over the years when I was looking to learn a particular feature or troubleshooting an issue I was experiencing. These aren’t strictly speaking minimal examples in the stackoverflow sense of the term, they tend to be a little bit more involved. In my case they are often small apps that implement just the feature I was exploring or debugging. They get rid of the complexity of the surrounding code, making it easier to reason about. I find myself often revisiting these minimals when I need to implement similar functionality somewhere else.

The ability to easily create runnable minimal examples I feel is one place NodeJS really shines, especially when you are building Express based web applications. This makes it easier to debug complex problems and communicate your findings to others you are working with.

Related post: Reasons to use NodeJS for developing your backend systems

2020/12/17

2020/12/16

2020/12/15

2020/12/14

2020/12/13

Choosing your web development stack

This is a bit of an aggregation piece, pulling together several thematically related posts that I think work quite well together to inform on the topic of choosing your web development stack. I’m a NodeJS developer so that’s my focus, but even if you are using a different language, a lot of the information might be useful in terms of building a broader understanding of the web development landscape.

Make it personal - When you are just starting out, a good place to start is to build your own website. It’s your chance to create your own place on the web. Nice recent css-tricks article that’s gets to the core of why building your own personal site is a worthwhile endeavour.

Deciding when to build a custom solution in web development - Sometimes referred to as the Build vs Buy dilemma, if you are developing websites, it’s important to be able to focus your efforts in the right place, this article gives you a mental model to help you decide when to build and when to use off-the-shelf products.

Reasons to use NodeJS for developing your backend systems - Self explanatory title, worth a read if you want to learn why people specifically use NodeJS.

Robust NodeJS Deployment Architecture - If you are leaning towards self-hosting, this article might be of interest, it’s more effort initially but you gain considerable portability.

I'm going to keep updating this post with other relevant articles I write or find online and I'll post to the linkblog when the article gets updated, so you might want to add the linkblog RSS to your feed reader.

2020/12/12

2020/12/11

Deciding when to build a custom solution in web development

The web has been around for just over 3 decades now, with the capabilities of the sites we build increasing every year. With that forward movement there’s a corresponding rise in complexity. For website builders this presents somewhat of a dilemma. When you start out, you don’t have the experience to build things for yourself, but you still want to have an online presence. Luckily it’s easier than ever to get started quickly with many website builder tools available. But how do you know when building something custom is appropriate?

Answering that question when you are deep in the weeds of designing and building a site, can be challenging. I think it’s useful to have a mental model that enables you to step back and see the woods for the trees.

Building a website is a bit like building a shed. Right off the bat you can choose to buy a prefab shed, and that’s a perfectly good solution. The suppliers of these often have several models to choose from, so if one of them meets your needs, it’s going to be quick to get up and running. There is of course also the option to customise your prefab shed, make it look a bit more like your favourite style. However you might have some very specialist requirements, because of where the shed will be located, or maybe you need some electrical wiring for your home office which you plan to have in your new shed. In those situations it might make sense to build a complete custom shed that fits your requirements exactly.

It’s similar with software. There are many off-the-shelf solutions that might fit your needs. There are hosted services and also open source frameworks like Wordpress that will make it easy to get a site up and running, and enable tweaks using themes and plugins. In situations where this meets your needs, that’s probably going to be the best route.

Of course the web isn’t just sheds. The analogy scales up too. There are houses, hotels, community centres, sky scrapers, towns, cities and we can continue scaling up. As your infrastructure grows, your requirements will evolve. You’ll want to create resiliency, by splitting your backend into several components. To some extent you might be able to modify your Wordpress sites to fill these needs. But don’t assume that Wordpress will solve all your problems, it’s totally possible to build a custom Wordpress monstrosity, same goes for any custom software.

One of the downsides of the prefab solutions that large frameworks offer is complexity. Along with all the useful heavy lifting that the frameworks offer, comes a big increase in the amount of code. If you ever need to get into the code yourself, rather than have a developer modify it for you, it might be a challenge.

Having a custom solution that is very focussed on solving exactly the problem you are trying to solve, has the possibility to be much much more streamlined, less code, easier to understand if you ever need to get into it yourself. Starting small and growing progressively as your needs change can ensure that you don’t suddenly find yourself in a sea of complexity, overwhelming your efforts to get a site live.

If you are considering a custom solution, you might want to look at NodeJS. It’s designed specifically for network based applications, and it’s possible to build very focussed low code applications that are extremely performant and easy to maintain. I recently wrote about the reasons for using NodeJS for building your backend systems, you might find that interesting.

The analogy in this post was inspired by a recent Shop Talk episode. Great podcast that covers frontend and increasingly backend web development topics.

Hopefully this gives you a bit of a better idea of the software development landscape, and a neat macro way to think about your web development projects.

This post is part the choosing your web development stack series.

2020/12/10

2020/12/09

The coming revolution in freelance web development

For most of the early web, freelance web development has been the norm. If you needed a website you hired a Wordpress developer and they hand crafted a beautiful website that met your needs. These developers had some backend development skills, but their real strength was frontend development, wrangling the html, css and javascript necessary to manifest your soon to be online property.

The web has evolved, and the websites people are building are in some cases becoming more complex, they aren’t only concerned with a nice place to present things digitally, but in addition with backend systems to handle various tasks, either through integrations with 3rd party cloud web services or fully fledged SaaS web applications.

Whereas previously these types of websites and applications were only built by large organisations' in-house developers, the tooling, frameworks and technologies have progressed to the stage where it’s possible to embark on such projects with freelance developers. The world of freelance web development is broadening, and this is a great progression.

The wider industry still has to mature and reshape to accept this new reality. Developers doing this type of development have been siloed inside organisations, who’s hiring practices are anti-freelancer. This is something you realise pretty quickly as a freelancer when you apply for regular jobs. The recruitment process is structured in such a way that the freelancer has to pay for it. Organisations expect applicants to perform tests and take-home projects in addition to many rounds of interviews, and they don’t pay a penny. It’s a totally unsustainable backwards situation. Large organisations are expecting their recruitment process to be funded by individuals who are often struggling to get by. They are completely shocked when you suggest that they should be paying for your time. This keeps freelancers poor and in-house developers locked-in.

In my opinion freelancing should be the norm everywhere. Joining a company more permanently should be possible but the norm should be to start as a freelancer and then transition to a full time position. The work force should be more mobile, it should be easier for individuals to move between companies, or to operate independently, and to plant roots when it makes sense to do so.

The move towards remote work is accelerating this shift. Dan Andrews and Ian Schoen serial entrepreneurs and hosts of the TMBA podcast, in a recent Q&A episode (24:00) spoke about the myth that freelancers are more likely to leave, freelancer and business owner incentives and the improved innovation that freelancers can offer.

Dan is “Long freelancers”, he thinks the role that freelancers play is being redefined, and that in many ways the interests of business owners and freelancers are more aligned, with long term partnerships growing from initial low friction and flexible freelance engagements.

I’m on the freelance road, and it’s especially tough in these uncertain times, but I thought it might be useful to others to share my strategy as I move forward. I wrote about it on Indie Hackers. It’s very much like marketing and promoting a product, a multi pronged campaign to highlight my freelance services. I’ve also implemented a job interview policy that I am sticking to. It’s something that I have to do. It’s not sustainable otherwise. I encourage you to take a similar approach that makes sense for you.

I aspire one day to have an organisation that can fund it’s own recruitment process. One which works with freelancers as a normal way of doing business. I believe new ways of working have the possibility to improve the whole ecosystem. I think it’s an important topic that we should be discussing.

I would love to hear your thoughts about how you see freelancing will evolve over the next 10 years.

2020/12/08

2020/12/07

2020/12/06

Motorbike Food Guy 1

A mobile food stand on a motorbike.

2020/12/05

2020/12/04

How to become an official sponsor of the linkblog

UPDATE (2020-12-20) - This blog post has been turned into a separate page, check out the sponsorships page for the latest.

It’s now possible to become an official sponsor of the linkblog!

I’ve spent time working for feature film visual effects companies and an enterprise software startup, and I’ve been running the linkblog for close to 10 years. It's one of the longest running personal linkblogs on the internet.

The readership is very tech and developer focussed, lots of smart people working for big tech related companies.

What do you get as a sponsor?

  • Your logo displayed on each linkblog day that you sponsor
  • A special blog post at the end of the week thanking you and linking to your website (for 7 day sponsorships)
  • A special mention in the newsletter that gets sent out on Saturday (for 7 day sponsorships)

The logo is right under all the links for the day, grouped with the patreon button.

It will appear on the main linkblog page which displays the most recent 50 days. It will also live on in the day and month archive pages.

Here’s an example of what it would look like for the month and day archive pages. It will look very similar on the main linkblog page

If you’d like to become a sponsor then get in touch with me via email.

Reasons to use NodeJS for developing your backend systems

There are a lot of programming languages to choose from when it comes to writing server-side code. What makes NodeJS a good choice for this task? Whether you are embarking on a new project or extending an existing one, it’s a very relevant question. In this post I’m going to cover the main reasons for choosing NodeJS for your backend application.

Javascript the programming language runs in two main environments:

  • The web browser which runs client-side javascript
  • The server which runs server-side javascript

When programmers talk about server-side javascript they call it NodeJS, and when they write a NodeJS application they are actually writing javascript code. NodeJS is more than just the language though, it’s a whole environment (called a runtime) that, among other things, can execute javascript code, but it also provides a way for the code to access hardware like the storage and the network adapters.

The NodeJS runtime achieves this using an architecture that is optimised for creating network applications. This architecture is called the event loop.

The event loop looks very much like a collection of queues. As the code runs, anytime something has to happen asynchronously, i.e. that will take a while to complete, the code to be run on completion, called a callback, is placed in a queue, so that the remaining code to be run can continue executing without blocking.

Asynchronous tasks would be for example writing to storage or making an API call across the internet. The event loop architecture makes it possible for the NodeJS runtime to be single threaded, and we say it’s event driven because the data input/output (I/O) from the hardware doesn’t block the code. Instead of blocking, events are triggered once the result of the I/O operation is ready.

This means that NodeJS is particularly good for applications that operate over a network, because it can handle many simultaneous requests very easily.

That’s the big architectural advantage that NodeJS offers, but there are quite a few other reasons to build your backend using NodeJS.

Here is a summary:

  • Runtime environment is particularly well suited and optimised to network applications, the event loop architecture makes it possible for single threaded code to execute in a non-blocking way, resulting in a high capacity for handling concurrent requests and realtime data
  • The NodeJS foundation is particularly well organised with a very regular release cycle, including long term support (LTS) releases that have a published schedule
  • There are working groups such as next10 and web-server-frameworks that collaborate publically, they have a clear strategic direction for the NodeJS project
  • Npm hosts a vast collection of community built modules, with mostly open source licenses, greatly speeding up development
  • Of all the interpreted languages it is one of the fastest. It uses the v8 javascript engine written in C++ by Google also used in the Chrome web browser, very regularly updated, always being improved
  • There are a huge number of javascript developers worldwide since javascript also runs in the browser. This means finding developers is easier
  • It’s possible to use javascript throughout the entire application stack, from the client-side code that runs in the browser, to the code running on the server, and also with NoSQL databases many of which use javascript as the query language. The elimination of context switching results in massive boost in developer productivity
  • There is a vibrant tooling ecosystem with developers around the world continuously building and sharing the best development tools
  • NodeJS is cross platform, running on Windows, Mac and Linux
  • Using libraries like Electron and Cordova it’s possible to write desktop and mobile apps using NodeJS so you can have a single code base across mobile, desktop and web applications
  • Typically promotes fast development, robust testing and code refactoring
  • All serverless hosting providers have NodeJS implementations, microservices are very often written in NodeJS

It’s also worth reading the nodejs website about page for more details.

Companies such as Netflix, PayPal, Trellio, LinkedIn, Uber, eBay, Groupon, NASA, Mozilla, Twitter and Walmart are examples of big tech organisations that run significant amounts of their infrastructure on NodeJS. More details here and here.

NodeJS makes it possible to build extremely robust network applications quickly and at low cost, that are then easier and cheaper to refactor, extend and maintain.

This post is part the choosing your web development stack series.

2020/12/03

2020/12/02

2020/12/01

Live Music From Small Boat 2

More live music from the guy on the small river boat.

2020/11/30

What it’s like working for an enterprise software startup

I wrote recently about my time working in the VFX industry during the height of the analogue to digital transition. I wanted to write another similar piece remembering my time working for an enterprise software startup in the file delivery space, so that I have something to refer to in future, but also because it might be useful to others to get a picture of what it’s like working in an enterprise software company startup.

First I worked as a Technical Consultant, and then as a Solutions Architect as part of the Global Operations team. I then joined the engineering team as a Software Developer, contributing to the development of a hybrid cloud file delivery SaaS. Having worked with many software vendors as a client in my time working for visual effects companies, it was eye opening being on the other side of the fence, with the folks building, marketing and selling vendor software.

When I joined they had a head office in Boston (US) and the engineering team in Ottawa, (Canada). Europe, Middle East & Africa (EMEA) was a new region for them, so there was no office. I worked from home in London (UK), or onsite at client locations, everything was done remotely. Initially we had to build the customer base, so there was a lot of travel throughout the EMEA region. There were times where I was travelling to several different countries every other week. I got a taste for being on the road and I liked it.

The company was pivoting from making source code replication software, to more generalised file movement software and they were focusing on the Media & Entertainment sector because the analogue to digital conversion was resulting in a need to move very large files. I was brought in to help grow the EMEA region and I worked closely with the regional Sales Director and Sales Engineer, meeting with clients, doing technical pre-sales, and later post-sales integration work, architecting and implementing a variety of media workflows.

I met with people from all levels, from engineers, developers, producers, journalists, managing directors, CEOs, CTOs to name a few. This was within visual effects companies but also broadcasters, channel aggregators, playout centres, network carriers, news organisations, radio stations, advertising agencies, system integrators, other software vendors, cloud providers, and occasionally clients in other sectors like oil and gas and automotive. Having exposure to a much wider section of the media landscape was incredibly interesting. I learnt first hand how these companies moved and transformed files, how they structured their organisations so that people could collaborate effectively, and the challenges they faced in moving to digital.

Another side that was super interesting was working closely with sales and marketing.

We had weekly sales calls at the end of the week, everyone in Global Ops would dial into the conference call no matter where we were. This was mostly the sales reps sharing how we were getting on with prospects, if they were on target to hit their number for the quarter, as well as any roadblocks they were up against. Occasionally the Technical Consultants / Solution Architects would be called upon. Us techies also had our own separate weekly conference call for more involved technical discussions. Although there was some competition between regions, we all collaborated together really effectively because we all wanted the entire company to do well.

At the time it blew my mind that we were all separately moving around while dialled into the sales call on our Blackberrys (and later iPhones). People transiting between clients in cabs, waiting for planes in airports, snacking in hotel lounges, zipping around on trains, or just in their home office, all across the globe, every continent. The first few years our phone bills were pretty insane.

The language in sales and marketing focussed companies is completely different to what I had experienced before. Everything revolves around the year being divided up into quarters, with Q1 starting in beginning of January and Q4 finishing end of December. It makes it easier to talk quickly about approximate timelines.

There are a lot of 3 letter acronyms, POCs (proof of concept), SOWs (statement of work), NDAs (non-disclosure agreements), RFPs (Request for proposal), QBRs (quarterly business review), ROI (return on investment), and many more. They make communication more efficient. The other thing there was a lot of was conferences, attending IBC in Amsterdam and BVE in London every year, hosting a booth where we did demos and presentations throughout the day.

As can be expected, there was a big focus on numbers, on your team hitting it’s quarterly quotas, with extensive use of Salesforce to track and forecast the sales process. It was noticeable to me the motivating effect of having a commission component to my compensation package.

I was often really impressed with how very complex customer relationships were navigated, with creative ways to move the sales process along, and the formation of strategic partnerships with people and companies with whom we had synergies. It was enlightening to see how much effort was put into pricing, and into relaying customer feedback to the product and engineering teams, and then to later see changes in the product that had been added in response.

As the EMEA region customer base grew, there was a lot more post-sales integration work. This is where I would work closely with the customer to architect the deployment of the software they had purchased. There was a lot of variety but generally they wanted a robust, secure and highly available infrastructure, and they often wanted to integrate our file movement software with their existing internal systems.

I wrote custom components in Perl that connected our software to existing systems using APIs or hot folder + XML file integrations. I then used these and pre-existing components in the product's workflow engine to build multi step workflows to automate the processing of inbound and outbound files, sending them to 3rd party tools or to operators and digital artists for manual steps. Sometimes replacing old manual workflows, other times creating entirely new workflows that had never existed or been possible before. I would then install the software and custom workflows, test everything was working, get the customer to sign-off on the deployment and train the users on how to use the system.

These were some of the workflows I designed and/or built:

  • ProsiebenSat – Ingest via Mx/Agent/Smartjog/Ftp, automated validation then manual validation, checking into MAM, export via Mx/Agent/Smartjog/Ftp
  • Canal Plus - Ingest via FTP Pull/MX, media check, Agility transcoding, import into Sonaps and Avid editors
  • Chellomedia - Ingest via Agent, movement of file bundles, Antivirus check, splitting of bundle on arrive in Lan, delivery of each piece to the right place
  • TPC – Swiss Television - Avid export, transcoding, Sound recording, manual steps offered via Mx GUI, delivery to Playout
  • Media City UK - Avid Gateway - easy ingest and outgest of avid projects
  • Discovery UK - VOD preparation with automatic adding of bumpers, logos and subtitles, and automated quality assurance check

It was at times very chaotic. I remember completing a pre-sales demo workflow that used AWS SQS to receive files in the back of the taxi on the way to the client demo. I got it working on my laptop over 3G just as we pulled into the carpark. The live demo worked without a hitch. The total number of hours worked per week was very high, it was difficult to separate work and personal life, and the constant travel, though fun, did start to take it's toll on me after several years.

Overtime we introduced processes, training programs, new staff, we started using a concierge to organise our travel and hotel bookings, we built cloud infrastructure to host many concurrent client POCs on VMs. Things got very streamlined and we were bringing in about a 1/3 of the company revenue.

I then moved over to Ottawa and joined the engineering team as a Software Developer, working on the team that was building a new SaaS file movement product. It was a nice change of pace to have an office again, and to be on a team of developers.

We followed an agile software development methodology, working on features in 2 week sprints, with daily standups, spikes, retrospectives and sprint planning. I worked closely with QA engineers and devops to ensure the features were thoroughly tested and deployed without issues, and with the support team to identify and fix bugs. The team was very well organised, and we shipped a lot of features. The product was adopted rapidly with the user base growing to 400000 users, made up of both existing and new clients. During that period I learned a lot about web development, it was a fantastic experience.

Aside from the occasional merge conflict, things ran pretty smoothly, there weren’t that many emergencies. We had weekly meetings where all the teams across engineering shared what they were working on. It was fascinating to be a part of the development process of enterprise grade software. It was a great foundation in developing web based software as part of an agile team.

I also experienced what it’s like to live in a very cold climate, with winter temperatures often down to -20 Celsius, having to use special snow tires, and learn how to drive in snow storms and ice rain. We had a team outing to go ice skating on a nearby lake, that was a lot of fun. I went over the provincial border to Quebec, met and spoke with french Canadians. They have a very different accent to people from France. I got to go snowboarding quite a bit and can now board pretty well in either direction.

2020/11/29

2020/11/28

2020/11/27

Announcing my new Custom Training Service

I’m adding a new package to my professional services called the Custom Training Service.

Here’s the description from the services description page:

The custom training service covers the creation and delivery of custom training on web development and automation/workflow design and implementation. These can be made to according to customer needs and delivered either onsite or remotely via video conferencing tools. Other training topics might be possible.

I have previous experience developing custom training programs for

  • Onboarding artists at vfx companies
  • Rapidly documenting feature film custom workflows using video interviews
  • Training users on enterprise software products
  • Training users on workflow and automation integrations

My specialities are NodeJS web development and automation/workflows, but I could also create custom training programs on other topics.

In my previous work in this area, some of the projects involved onsite visits and consulting where I analysed the existing systems and processes prior to constructing the training program. I then worked worked with designers to create training material that was specifically tailored in look and content to the client needs.

If you have a need for some custom training, then don’t hesitate to contact me via email.

2020/11/26

Looking back at linkblog.io

I recently announced the end of linkblog.io. I go into a bit of detail in this short indie hackers discussion thread. I wanted to do a quick retrospective to have something to refer to in future.

I’ve been running a linkblog for close to 10 years. In a lot of ways all the links I’ve posted chronicle my web development journey, but also just life in general. I found it a very useful tool and still do to this day. Somewhere along the way I decided to build a linkblogging SaaS product.

These were the high level goals of the system that emerged over time:

  • Robust implemenation that does what it was designed to do and does it well
  • Have a minimalist user interface, emphasis on text content
  • Easily scalable to handle growth
  • Deployable to standard VPS hosting
  • Fault tolerant and able to have zero downtime deploys
  • Resilient so that any server could be easily rebuilt from scratch using backups where necessary

From a technical standpoint I achieved all of these goals.

I wrote about Robust NodeJS Architectures earlier in the month, and this describes very closely what the linkblog.io infrastructure looked like.

Some other application level features and capabilities that are worth mentioning:

  • MongoDB sessions support
  • Redis sessions support
  • Rate limiting using Redis
  • JWT API authentication
  • API Caching using Redis
  • Backend job scheduling using Agenda
  • Backend message queues using MongoDB
  • Custom domains
  • SCA compliant billing system (Stripe + webhooks)
  • Production and staging environments

Implemented using bash scripts:

  • Repeatable server provisioning system
  • Application build and deploy system
  • Backup and restore of certificates and databases
  • Log file backup and cleanup
  • Certificate renewal

Most of these features were added as a necessity in response to real world events that happened during development.

Though the UI is quite minimalist and mostly text based, there was quite a lot going on underneath. The system could have been quite easily and safely extended.

Finally here are some screenshots of the UI:

Landing Page - Header

Landing Page - Header

Landing Page - Description

Landing Page - Description

Landing Page - Features

Landing Page - Features

Landing Page Latest News

Landing Page Latest News

User Linkblog Page

User Linkblog Page

User Post Message Page

User Post Message Page

User Archives Page

User Archives Page

User Search Page

User Search Page

User Edit Mode Page

User Edit Mode Page

FAQ Page

FAQ Page

User Billing Page - Fresh Load

User Billing Page - Fresh Load

User Billing Page - Add or Update Card Extended

User Billing Page - Add or Update Card Extended

User Profile Page

User Profile Page

User RSS Feed

User RSS Feed

That was linkblog.io!

2020/11/25

Linkblog.io is dead, long live linkblogging

Linkblog.io the minimalist link curation tool, the Saas I spent 3 years building, is no longer live. All active users have been contacted and I’ve decommissioned the site.

Though the site was a tremendous success in terms of meeting my personal learning and development goals, building a robust SaaS from soup to nuts, ultimately I didn’t get the traction as far as user base that I had hoped for. Continuing to maintain the site didn’t make sense because it wasn’t generating any income.

I’m still linkblogging!

Update: the linkblog is now located on my main website domain.

I’ve been experimenting with Jamstack, Serverless and GitHub Actions and have built a prototype linkblog static site generator that I’m using every day. It builds 10 years of content in just a few seconds. It’s totally serverless. It’s pretty sweet.

I’ve been blown away with the possibilities that the jamstack opens up for creating custom content workflows.

I’ve also started a weekly newsletter where I share the best posts from the linkblog.

The general theme is javascript / tech / web development, but I also blend in some other topics from time to time.

I hope you like it, consider subscribing, it's completely free!

All the best.

2020/11/24

What it’s like working in tech in the visual effects industry

I’ve built and supported many workflows for visual effects productions on big blockbuster movies.

Movies such as

And hundreds more like those, working alongside digital artists and producers to build novel ways of collaborating. It’s been a few years now, I’ve since moved into enterprise software and now web development, but I thought it would be interesting to write a piece about my experiences in the visual effects (vfx) industry.

The landscape of the vfx industry is made up of a wide variety of companies, big and small, that work together to perform the post production on motion pictures. Ultimately they have the same goal - get the film finished on time for a theatrical release. There are often big budgets involved, and lots and lots of people doing a huge variety of jobs.

If you’re curious about how things are structured watch the credits of any film right until the very end. You can figure out a lot just from how the names are grouped together. Usually it’s the actors first then production staff, then all the post-production staff. But there’s a lot of nuance, with people being divided up into units based around certain pieces of the movie, say if it was filmed in different main locations.

It’s often surprising how many different visual effects companies are listed, each with their own teams of artists, engineers and producers. Of course animation based movies like the stuff that Pixar makes are nearly all post-production people, also huge amounts of post staff for all the superhero movies, but often films that don’t appear to have many visual effects shots, actually have loads of them.

The film credits of a movie is an art form in itself, with companies specialising in creating both the front and end rollers. If you stay right until the end of the credits in a movie theatre, a lot of the people remaining at the end, work in post. They want to see how things are structured at other companies, but they also just want to see their names or the names of their friends appear on the big screen. It’s quite a big thrill to see your name rise up across the screen when you’ve spent many months working on something.

I’ve worked in two different areas of the overall post-production movie pipeline.

First was as part of the engineering team in a reasonably classic vfx house, where we would work on a handful of movies in parallel, typically working on lots of individual shots, adding effects, then delivering these to companies upstream, who assemble all the vfx shots with the regular shots to create the full length movie. The work tended to happen right after production, or in some cases, at the same time as the movie is being filmed. There were often long days, and weekends, with some people working night shifts. These projects felt big, taking many months, with great wrap parties on completion. They also had a whole department that built old school physical visual effects, robotic monsters and weird creatures. Very cool but a bit scary when you are on the night shift.

There was a rather big render farm to render the effects onto the images. Hundreds of pizza box style nodes in racks, all managed by Pixar’s Renderman software. Organising the farm in an optimum way was complicated by the fact that 2D and 3D artists had very different types of jobs running. The 2D jobs were generally rather quick taking minutes, whereas some 3D jobs could take hours. You never wanted the 3D jobs clogging up the farm, but you also didn’t want the farm to be empty, and of course everyone’s job is very important. After a long time analysing the software and use cases we came up with a pretty effective configuration for the system.

There was a heavy focus towards Linux based systems, because it was easier to create multi-machine complex workflows using scripts written mostly in shell (bash/csh/tcsh) and Python. We would setup artists machines using shell environments in a neat way that enabled them to quickly switch between projects, with shot information pulled from a database, so they didn’t have to think about where the data was being stored, that was all automatic as long as they knew which shot they were working on, and everything was versioned. There were lots of cli system tools that for example did things with long sequences of image files.

For quite lot of the time I was there we had free toast, jams, Nutella, tea and coffee. That was nice. There was also a table football table and we held a yearly World Cup style competition, engineers vs 2D artists vs 3D artist. Good times.

The second vfx shop I worked for was further downstream, I was also part of the engineering team, receiving the finished shots from all the classic vfx houses. The work happened right at the start of post-production with film scanning and then right at the very end, after all the vfx shots had been completed by the classic vfx shops. We received all the shots and did various tasks such as colour correction, colour grading, beginning and end rollers, language versioning, subtitling, recording back to film, and creation of digital cinema distribution masters. The film scanning / recording and colour management tasks are often referred to as digital intermediate (DI), with the correction and grading being done by artists working with the directors and producers in specially constructed completely blacked out theatres.

We often had to meet impossible deadlines compounded by the fact that we were right at the end of the post-production process, impacted by any delays that happened at the other vfx houses. Missing our deadlines was just not an option, the penalties for which were in the order of millions of dollars per day, since the release date of the film would have to be moved.

This happened quite a lot:

Boss: How long will X take to do?

Engineers: 3 months

Boss: Ok we need you to do it in 3 weeks

Amazingly we always figured out a way to meet the impossible deadlines.

In this second place there was a much bigger volume of films passing through and typically the work on each film was on the order of weeks, the parties were mostly just for the producers interacting directly with the clients.

I mention the parties because I think it helps to illustrate that the overall vibe at different vfx shops can be very different depending on the type of work that is done. One isn’t necessarily better or worse, it’s just different. In broadcast media, where I’ve also had some experience, it’s yet another kind of different, more corporate in a way.

As far as tech goes there was a lot of cool hardware. Super specialist colour management boxes with unbelievably fast infiniband network connected storage, amazing 2D & 3D projectors, top of the line audio setups. The DI part of the facility could be hired out directly to clients, but our in-house artists also used it for the jobs we were working on.

Finished feature films took up terabytes (12TB for a full length 4k resolution film) of storage space, and the artists often worked on multiple versions of shots. Lots and lots of storage. Anytime new storage was added it got gobbled up pretty quickly. Problems with the storage had a big impact and cost a lot of money because no one could work. Stressful at times but generally there was a good camaraderie between everyone.

When I was there much of the movement of finished shots was done via USB drives, we had hundreds of them moving around town at any one time. I bet they still use drives quite a bit. Very manual but effective. You do what’s necessary to get the job done. I’ve been in situations where we had to send someone transatlantic on a plane just to hand carry some drives.

These days a lot of the finished shots are moved between companies digitally. There are high bandwidth networks between a lot of the post facilities. At the time, we were already experimenting with some very involved transatlantic workflows on some big films.where the director was in LA and the colourist was where we were in London. Making sure that the colour science equipment in both grading theatres was setup correctly was crucial. It was also quite fun because we got to work with the US team that was flown in for the duration of the project.

The film scanners which were about the size of two fridges side by side, were able to scan reels of film at around 8 frames per second, and they had a suite of Perl scripts used for automating the post processing of the all images created, so you could trigger custom scripts to process the scanned images on the render farm or other specialised hardware. Some of the reels of film would come in quite badly scratched, so we setup workflows to automatically digitally remove the scratches using specialised software. The only other way was to do it by hand frame by frame. The render farm there was smaller and ran on very fast blade servers. It was managed by a piece of software called Rush which is written in Perl.

There was always lots of things happening in parallel, so it was a little tricky not to step on each other’s toes. We were all trying to do this complex thing of creating digital effects all at the same time as constantly rebuilding and upgrading everything around us. If you went away on holiday it was quite usual for your desk to have moved by the time you got back. There were lots of emergencies, the business people were always trying to land new contracts with important clients visiting to see the latest version of their project, a sort of organised chaos. But we sure did make a lot of films.

The other thing that was happening at the time I was there, on top of the general day to day internal tech chaos, was that the whole wider industry was freaking out about the introduction of digital cinema. Every part of the ecosystem was being written and re-written all at the same time.

Was it cool? Yeah it was pretty cool.

I gained a solid foundation in using cool technologies, building infrastructure and workflows, figuring out new ways of collaborating, streamlining existing processes using both off-the-shelf / open source tools, and writing our own when necessary.

2020/11/23

2020/11/22

Why are people chopping off their legs with consent forms?

A recent HN thread was discussing Why Japanese web design is so different. It’s based around an article written in 2013 but both the article and the thread discussion are an interesting read.

I believe that consent forms pose one of the worst usability problems the internet has ever seen, with big companies that have alternative news reading experiences benefitting immensely from the situation. Their products are slick and unbelievably lovely to use in comparison, but fundamentally non-weblike, featuring only big media brands and with no ability to link to articles.

I wondered if the US users knew how bad it had become, I asked about the prevalence of consent forms in Japan closing with:

Probably US users don’t know how bad it is because the US sites only add them to requests originating in the EU.

A canadian user commented that he didn’t think it was only EU users that saw the consent forms because he is seeing a lot of them too.

I don't think that's the case, if only anecdotally, as I have seen a huge uptick in these consent modals in Canada.

Most of these sites needed or wanted them implemented on the cheap. Restricting it to EU customers would require extra work...

If this truely is happening, it makes no sense whatsoever.

Why would a website owner that makes money from people viewing their website risk that by blocking users from viewing their website even when they are not legally required to do so?

Are they living in a strange parallel universe where money doesn’t affect them? (serious question btw)

There’s must be some crucial bit of missing information here, because it really makes no sense.

2020/11/21

I've started a Newsletter!

Earlier in the week I met-up with James Clark from Nomadic Notes for coffee. He’s a long time travel blogger and digital nomad known for his excellent city travel guides. Really interesting guy with lots of insight into the travel blogging scene. It was great to meet him.

Yesterday his newsletter hit my inbox and I noticed that it’s powered by Substack. They’ve recently become very popular for their easy to use newsletter web application. I had coincidentally sent them a job application last week, turns out they use NodeJS. Anyhow I’ve been thinking about starting a newsletter for a while now, and this felt like a good time to do it!

So after deliberating a bit about which day to send it out, I settled on Saturday midday. Post-lunch BBQ time in Australia, tastey lunch time soup time in Asia, croissant and sausage sandwich time in Europe, in the Americas the midnight feast folks are going to love it, and for the others they’ll have something waiting for them to go with their breakfast burritos, cheerios and pancakes when they wake up.

It’s also a good way for me to delineate the start of my weekend. Something I’ve been missing for a while.

The newsletter is a roundup of the best links I’ve posted to my linkblog over the week. The general flavour is javascript / tech / web development but I also mix in other bits and pieces to get a good balance.

Subscribe for free in just a few clicks, the first issue is going out later today.

2020/11/20

My policy on job interviews

UPDATE (2020-12-20) - This blog post has been turned into a separate page, check out the job interviews policy page for the latest policy.

I wanted to describe my policy on job interviews so that I have something to refer to in the future.

Over the years I’ve interacted with many prospects discovered online and done quite a few of these and I have had a number of bad experiences, which ultimately end up wasting my time and money.

I’m just one person and I cannot support people for all the myriad of exotic reasons they want to spend their time doing interviews. It’s exhausting and never ending.

For new accounts I have 1 short call to meet and get a high level overview of the project.

My policy then is quite simple. My hands don’t touch the keyboard until I am getting paid. Also if you want consulting services, it’s possible, but again you need to pay me for that.

For clients that want extended discussions, I suggest the consultancy service package which is delivered via video chat. We can verbally discuss code in any way you would like.

You can send me short code samples to read in advance if you’d like so we have something concrete to discuss.

What I suggest to clients that want to work with me live is to start with a 1-3 day development service package where you can have the time to work directly with me in a live setting.

I am open to long term arrangements and would be happy to transition to a more permanent role, but I could also continue on as a freelancer, whichever suits your needs best.

Check out the service package descriptions and price list for more details.

2020/11/19

Mozilla MDN Docs are going full Jamstack

Earlier in the year MDN Web Docs turned 15 years old. They have been around since the early days of the web and made huge contributions to it’s evolution. More info on their Wikipedia page.

But how big are they?

Let’s look at a few stats:

  • 15 million views per month
  • 3,000 new articles in the last 3 years, 260,000 article edits
  • Grown in double-digit percentages, year over year, every year since 2015
  • Serving more than 15 million web developers on a monthly basis

They are big!

They’re also doing a phenomenal job of organising the web’s developer documentation, adding new site features to make learning easier, while building a vibrant community.

So it’s big news when they decide to re-architect their platform, especially when it happens just after they announced 250 layoffs (1/4 of it's workforce).

I posted on my linkblog a few weeks ago:

MDN Web Docs evolves! - the folks at Mozilla are going Jamstack + GitHub for their new MDN docs content contribution workflows, great writeup of the planned architecture, this is definitely an interesting space to keep an eye on, it will be really cool to see the collaboration workflows they build

Here is a link to the original article on hacks.mozilla.org.

Their writeup does a really good job of explaining the architecture changes they are making.

Essentially it boils down to this:

We are updating the platform to move the content from a MySQL database to being hosted in a GitHub repository (codename: Project Yari)

And the key takeway:

We are replacing the current MDN Wiki platform with a JAMStack approach, which publishes the content managed in a GitHub repo

In the piece they list many reasons including:

  • Making it much easier to add new features to the platform
  • Better contribution workflow, moving from a Wiki model to a pull request (PR) model
  • Better community building by adding discussions, feedback, review and approve steps
  • Simplified frontend architecture improving accessibility

And they go on to describe the planned architecture in a lot of detail with some illustrative diagrams. Jamstack website architectures are officially where new custom content workflows are being built.

I think this is perhaps a sign that there are some big shifts happening in web development, where the architectures that have powered the web for the past 10 years are being rewritten, enabling new ways of collaborating together to make digital things.

Which other large websites will make similar moves?

2020/11/18

2020/11/17

Live Music From Small Boat

Guitarist plays on a small river boat, while a group of people eat an evening meal.

2020/11/16

2020/11/15

2020/11/14

2020/11/13

Robust NodeJS Deployment Architecture

The aim of this post is to succinctly describe an effective and robust architecture for self hosting your NodeJS web applications. I’m going to stay relatively high level, describing the technologies, and components, by the end of it you will have a good idea of what such a system looks like. There is a focus on standard well tested pieces rather than the latest shiny cloud / containerisation offerings. It is well suited for running small to medium size applications.

Node.js web development technologies

Features of the architecture

  • Runs on standard VPS hosts
  • Possibility to scale
  • Secure
  • Easy to maintain
  • Fault tolerant
  • Low cost
  • Backed up and easy to restore
  • Easy machine provisioning
  • Easy to deploy code
  • Support multiple databases

3 main components

  • Load balancer
  • Web and API application servers
  • Datastore

During it’s life cycle, a client web request travels over the internet and eventually arrives at the load balancer where any SSL/TLS connections are terminated, then re-encrypted using self-signed certs and sent to an available application server. That application server performs the tasks it needs to do, persisting information on a shared datastore. Responses are sent directly from the application servers to the client.

The SSL/TLS termination happens on the load balancer because it makes managing the certificates much easier, with only a single place to renew, create, update and backup certificates.

Having a load balancer ensures that you can have several application servers running in parallel, which means you can scale by just adding more application servers, but it also means you can reboot servers without impacting site uptime.

As for the application servers, you can separate out web servers from API servers, but for ease of maintenance you can also just run both on the same machine on different ports, with a reverse proxy on the machine directing the requests to the right application. In this way you have one discrete unit which makes it much easier to add capacity. In the vaste majority of cases this setup is good enough, though could be optimised later.

Having a shared datastore is key to being able to run the application servers in parallel. This is a single machine that has a large storage volume mounted. It runs all the databases which write their data to the storage volume. The datastore can also run on a clustered set of machines for high availability, though this adds quite a lot of complexity, so initially it’s probably best to run one machine with good backups, so if anything goes wrong you can be restored and running with a minimum of downtime.

Technologies

  • Nginx - Load balancer and reverse proxy
  • Redis - Key/value very fast database often used for storing sessions and caching
  • Mongodb - NoSQL database
  • Postgres - SQL database
  • Letsencrypt certbot - for generating and maintaining certificates
  • Linux Ubuntu - Operating system for all 3 components
  • Pm2 - NodeJS process manager, runs the applications, handles logging and a variety of other runtime activities
  • RabbitMQ - Message queue software very important for fault tolerant backend systems
  • Mongodb-queue - Message queue implemented via a NodeJS library backed by MongoDB

Provisioning infrastructure

You can keep things quite simple in this regard, using a Bash script for each of the 3 main components. The script would need to do the following:

  • Install latest OS updates
  • Install necessary software
  • Configure users and groups
  • Write/update software configuration files
  • Start and stop various services

These are some of the important Linux items you would need to know about:

  • sshd - server for ssh connections
  • stunnel - creates secure connections, used on datastore for applications without built in SSL - e.g. Redis
  • ufw / iptables - firewalls
  • PKI and creating self-signed certificates
  • logrotate - manage rotating and backing up application log files
  • cron - schedule the running of maintenance scripts like backups
  • certbot - generate and renew certs
  • rsync - securely synchronize files between machines

It’s likely that your VPS hosting provider has an API and / or command line tools, making it possible to create a provisioning script that creates a VPS server, rsyncs the bash install script to the machine and runs it. So with a minimum of fuss you can provision fresh servers by running a script, so it’s completely repeatable.

It’s worth noting that there are modern tools that use containerisation like Kubernetes, which are very powerful but can get quite complex.

Deploying code

This is another place where a simple bash script can be very effective.

It would need to do the following:

  • Build your application to a deploy directory
  • Backup currently running app
  • Rsync the files to the application servers
  • Restart the application server

There is a lot of variety in this area. Many modern workflows that use CI/CD systems use git to clone your entire application
repository to the server, rather than rsyncing just the built files. Requirements vary a lot from project to project.

The bash script route is great for simplicity, but there are often more manual steps involved, especially if your application has complex configuration. In the early days of a project it’s often good enough.

Backups

Backups are super important. You need to have all the important files backed up and ideally scripts to restore the backups in the event that a component fails and needs to be restored.

Consider backing up:

  • Each deployed application version, along with configuration
  • Log files for databases, firewalls
  • Certificates
  • Contents of all databases
  • Configurations for every 3rd party application you are using

It’s a good idea to use storage from big cloud providers, they are low cost and have good scripting tools.

Security

It’s important to configure your machines securely, set firewalls (local and cloud) appropriately. Always use TLS/SSL for inter machine communication. Follow the security advice from the various pieces of software you install, for example creating different users for specific purposes e.g. application access vs access for backups. Only give the minimum of access rights necessary to perform a given task.

Staging and production environments

Once the application is running in production, you will benefit a lot from having a staging environment. It’s a replica of the production environment where you can try out new code without being worried to break the live system. Never deploy directly to production, always test it out in staging first.

Wrapping up

The infrastructure side of running applications can get quite complex, but there are a lot of advantages to knowing how to construct these setups yourself:

  • Keep costs at a minimum
  • Be in full control of the infrastructure
  • Be able to deploy anywhere

It’s also worth experimenting with integrating serverless technologies for aspects that are very high load, the low cost and high performance might be worth the portability trade-off, but be aware that a move might require rewriting parts of your application should you need to change providers.

This post is part the choosing your web development stack series.

2020/11/11

2020/11/10

2020/11/08

2020/11/07

2020/11/06

2020/11/05

2020/11/04

2020/11/03

2020/11/02

Self-hosted web based tool to get tasks done efficiently

A short while ago I was wanting to have a self-hosted tool to manage sets of tasks in an efficient way, so that tasks actually get done. I wanted it to be simple enough not to require much maintenance, with minimal chances of something breaking because of software upgrades.

I built a simple statically generated web based tool that is based on the agile software development methodology, which uses a “backlog” of “stories” that you complete during time periods called “sprints” (usually 2 weeks long).

Another key idea is that writing things down is a great way to focus on the task at hand and a way to drive your way as you forge your path.

It uses the Eleventy static site generator to render all the pages.

Here is the Github repo and there is a demo site with a bit of example data.

How does it work?

From the repo docs:

Use blog posts to describe your work, what you did, what you are about to do, then create "stories" that you add to the "backlog". Assign stories to "sprints", these last 1 week. Flesh out the stories, implement them, and then move these to "done" when you complete them.

At the end of the week do a retrospective of what you did, and plan (i.e. create and assign stories to the next sprint) for the upcomming week.

Whenever you are a bit unsure of your path, read the above 2 paragraphs. You probably need to write a blog post.

Benefits:

  • Self hosted - run it in the cloud or just locally on you machine
  • Everything is a file, versioned using git
  • Create items written in markdown
  • Easily backup your repo using one of the many git hosting providers
  • Never have a migration problem, it’s all just text files
  • Works offline

This is aimed at personal use rather than for large projects. I think for big projects, especially where you are collaborating with others, it’s probably better to use more comprehensive tools.

2020/11/01

2020/10/31

RecRain With Thunder 1

Mostly rain with a bit of thunder.

Rain With Thunder 2

Mostly rain with a bit of thunder.

2020/10/30

GitHub Actions for custom content workflows

I’ve spent the past few weeks making some updates to the build system of the latest incarnation of my long running linkblog, now a statically generated website. In doing so I had my first chance to try out GitHub Actions and what I discovered is an extremely versatile tool that makes it possible to create a very wide variety of software developer workflows, centering predominantly around coding activities.

I believe another area where Github Actions might be very effective is in modern collaborative workflows for content creators. And this is happening now because of the recent resurgence of building static websites in a way refered to as the Jamstack.

In this post I’m going to describe the pieces of the puzzle, and the types of workflows that are possible, but without a lot of the complicated jargon.

Github is most well known for hosting repositories, places where developers can collaborate on code. It’s really a web interface for the open source git command line tool, which can be used completely separately, but their interface has become very popular over the past few years, and they are a big reason why there has been a renaissance in open source software.

Github Actions is their workflow automation tool, and it has a focus on automating tasks in and around repositories. It makes it possible to trigger custom actions when various repository events happen, such as adding new content, or when users have reviewed content, or when discussions mention a certain keyword, and lots more. It’s really very versatile.

The Jamstack movement is a way to build websites that has become popular recently. It focusses on pre-rendering all the pages of a website so that they are static files, rather than have a server running an application that dynamically renders the pages when they are requested. There is a lot of variations and nuance to what it encompasses but that’s the general idea. A lot of the technology isn’t actually new but it’s the first time this way of thinking about and building sites has had a name, and it’s resulted in some very interesting forward motion in how websites are built.

There are a lot of benefits to Jamstack sites including:

  • Security because there’s no server to hack
  • Speed because all the pages are pre-rendered
  • Being a naturally a good fit for automation

And it's this last point that opens up new possibilites for collaborating.

I’ve been running my linkblog for close to 10 years, there is a lot of content, but it’s possible to render out all the site pages in just a few seconds. I’ve built my own static site generator (SSG) tool in NodeJS because I’ve been able to optimise for that use case specifically, but there are many open source SSG tools out there.

One of the major benefits of creating a content workflow that uses git, is that you gain all the safety that makes it a great code collaboration tool. By using similar, all be it simpler, workflows to developers you can be quite confident that you won’t loose any work. It’s all just text files, and everything is versioned. As well as safety you get a considerable amount of future proofing because in the end it’s all just text files, so much less danger of software update breakages, and since it’s based on standard git tools you can move to another hosting provider relatively easily. Most of the providers have their own workflow tools but there is some interoperability, Bitbucket for instance can run Github workflows, and making platform specific tweaks isn’t that difficult in a lot of cases.

As an example, for my linkblog I have some HTML forms built using serverless cloud functions, that I use to easily add new content to a Github repo throughout the day. I have a scheduled workflow that runs at the end of every day and merges in the new content in a safe way using what is called a ‘Pull Request’, often referred to as a PR. This makes it easy to revert the merge if that’s ever necessary. There is then another workflow that detects newly merged content and triggers a site re-build, and then a deploy to the website hosting provider, and the site is live with the new content.

It’s also possible to use some of the other tools that Github provides as part of the workflow, for example the Issues and Pull-Request pages, to create places where you can discuss new additions with collaborators, as well as setup notifications, and to only publish content that has been approved by a certain number of reviewers.

It’s not strictly text based content either, as part of my updates I added a podcast to my website. The files are hosted outside of Github, but the GitHub action renders all the podcast pages and creates the RSS feed.

If you want to listen to some very experimental audio, the show is in iTunes!

Also if this type of content workflow is interesting to you, feel free to get in touch with me to let me know. I might post more details, tips and tricks about the workflows I am using.

I’m also available for hire on projects you have. Check out my Github profile.

2020/10/29

Man Playing Trumpet 1

Man playing trumpet in the park.

Man Playing Trumpet 2

Man playing trumpet in the park.

Man Playing Trumpet 3

Man playing trumpet in the park.

Man Playing Trumpet 4

Man playing trumpet in the park.

2020/10/28

Raining A Lot

It’s been raining a lot lately, this is what it sounds like.

2020/10/27

Man Riding Bike

Man rides exercise bike in the fitness area of a public park.

2020/10/24

2020/10/23

2020/10/22

Motorbike Sales Guy 1

Field recording of the chant of a motorbike sales guy that drives around with a sort of mobile thrift store on the back of his motorbike.

2020/10/21

Noisey Cafe

Field recording in a cafe in south east asia.

Noisey Cafe 2

Field recording in a noisey cafe in south east asia.

2020/10/20

2020/10/19

Silly Chant Too Early

Recorded in the morning somewhere in South East Asia.

2020/10/18

2020/10/17

2020/10/16

2020/10/15

2020/10/14

2020/10/12

2020/10/11

2020/10/10

2020/10/04

2020/10/03

2020/10/02

2020/09/30

2020/09/29

2020/09/28

2020/09/27

2020/09/26

2020/09/25

2020/09/23

2020/09/22

2020/09/21

2020/09/20

2020/09/18

2020/09/17

2020/09/16

2020/09/15

2020/09/13

2020/09/11

2020/09/10

2020/09/09

2020/09/06

2020/09/05

2020/09/04

2020/09/03

2020/09/02

2020/09/01

2020/08/31

2020/08/30

2020/08/29

2020/08/28

2020/08/27

2020/08/26

2020/08/25

2020/08/24

2020/08/23

2020/08/22

2020/08/21

2020/08/20

2020/08/19

2020/08/18

2020/08/17

2020/08/16

2020/08/15

2020/08/14

2020/08/13

2020/08/12

2020/08/11

2020/08/10

2020/08/09

2020/08/08

2020/08/07

2020/08/06

2020/08/05

2020/08/04

2020/08/03

2020/08/01

2020/07/31

2020/07/30

2020/07/28

2020/07/27

2020/07/26

2020/07/19

2020/07/17

2020/07/15

2020/07/07

2020/07/06

2020/07/03

2020/07/02

2020/07/01

2020/06/30

2020/06/27

2020/06/24

2020/06/22

2020/06/21

2020/06/20

2020/06/19

2020/06/18

2020/06/15

2020/06/14

2020/06/13

2020/06/12

2020/06/05

2020/06/03

2020/06/01

2020/05/30

2020/05/29

2020/05/28

2020/05/27

2020/05/24

2020/05/23

2020/05/22

2020/05/21

2020/05/20

2020/05/19

2020/05/18

2020/05/17

2020/05/16

2020/05/15

2020/05/14

2020/05/11

2020/05/10

2020/05/08

2020/05/07

2020/05/06

2020/05/05

2020/05/01

2020/04/30

2020/04/27

2020/04/26

2020/04/25

2020/04/24

2020/04/23

2020/04/19

2020/04/18

2020/04/17

2020/04/15

2020/04/14

2020/04/11

2020/04/10

2020/04/08

2020/04/07

2020/04/06

2020/04/04

2020/04/03

2020/04/01

2020/03/28

2020/03/27

2020/03/26

2020/03/25

2020/03/24

2020/03/23

2020/03/22

2020/03/21

2020/03/20

2020/03/19

2020/03/18

2020/03/16

2020/03/15

2020/03/14

2020/03/12

2020/03/10

2020/03/09

2020/03/08

2020/03/05

2020/03/04

2020/03/02

2020/02/29

2020/02/28

2020/02/25

2020/02/24

2020/02/23

2020/02/22

2020/02/21

2020/02/19

2020/02/17

2020/02/16

2020/02/15

2020/02/14

2020/02/13

2020/02/12

2020/02/11

2020/02/10

2020/02/09

2020/02/08

2020/02/07

2020/02/06

2020/02/05

2020/02/04

2020/02/03

2020/02/02

Test Post Written Using Github

Just a quick test to see if it's possible to write a blog post directly in Github.

Usually when you push to Github using git, it triggers Jekyll to rebuild the static site, but I don't know if that happens if you save the file directly to the repo without a push. Based on this stack overflow post looks like it probably doesn't but figured I would test it out just in case.

Update 1: It worked! My first attempt didn't work because I forgot the .md file extension, once I got the file name right, the build was automatically triggered. Saving the file now should result in another build which will add this update.

Update 2: The update resulted in a site build too :)

Update 3: However I now see that though there is an edit button for files that already exist, there is no "create new file" button in the mobile version of guthub.com :(

2020/02/01

2020/01/31

2020/01/30

2020/01/29

2020/01/28

2020/01/27

2020/01/26

2020/01/25

2020/01/24

2020/01/23

2020/01/22

2020/01/21

2020/01/20

2020/01/19

List of freelance web development websites and marketplaces

When looking for remote web development opportunities there are broadly two categories - freelancing and remote positions.

Here is a list of websites for finding freelancing opportunities:

Also worth checking out is this article that has lots of details about freelance contracts.

For remote positions check out my list of remote friendly web development job board websites.

List of remote friendly web development job board websites

When looking for remote web development opportunities there are broadly two categories - freelancing and remote positions.

Here is a list of websites for finding remote position opportunities:

List of remote friendly companies:

Also worth checking out is this list of resources for remote workers.

For freelancing check out my list of freelance web development websites and marketplaces.

2020/01/18

2020/01/17

New Linkblog feature: rss, atom and json feeds

The latest linkblog feature is feeds. These are a versions of your linkblog that are easily readable by a computer. This makes it easier for people to read your linkblog but also enables you to do neat things like auto post your links to social media such as Twitter and Facebook. Each time a day finishes where you posted links, the hash link of that day gets added to your feed.

Linkblog rss, atom and json feeds

To access your feed, right click on the icon in the toolbar (it looks a bit like the Wifi symbol rotated by 45 degrees) and select copy link address. This should copy the url of your feed to your computer's clipboard.

If you then paste it into some form of text editor you will see what the feed url looks like. Here are my feeds for my regular linkblog and for my custom domain linkblog:

https://linkblog.io/users/mark/feeds/daily/rss
                        https://links.markjgsmith.com/feeds/daily/rss

The advantge of using your custom domain url is that theorectically if you ever needed to move your linkblog to another provider, then as long as you still own the custom domain, that could be done without the people already using your feed having to update to a new url.

If you try to load these pages in a browser you'll see lots of strange looking text with HTML tags, but you will also notice that your daily posts are within <item> elements. Computers can read these feeds easily. The vaste majority of the time though you never need to read the actual feed, you only really have to copy and paste the urls of the feed.

The most common type of feed is RSS so that's what is used in your navbar. Atom and JSON versions of the feed are also available by replacing 'rss' in the url with 'atom' or 'json'.

A very typical thing people do with feeds is add them to their feed reader, which makes it possible to read many sites from 1 place rather than have to remember to visit all the sites individually. Seen like this, feeds are very similar to the follow feature in many social media sites, they have the advantage of working across many different sites, though they are a bit more complicated.

Feeds also make it possible to do things automatically with your posts, like posting them to sites like Twitter or Facebook. For example I have setup my linkblog so days are posted to my Twitter account @markjgsmith. There are lots of 3rd party online sites that offer rss-to-[insert favorite social media site] type services.

For a more detailed description about feeds check out the You Need Feeds site. Also useful is this list of rss feeds for social media sites which might give you some ideas about what is possible.

If you want to try out running a linkblog then signup for the 14 day free trial, after the trial it's just a few dollars per year.

Setting up as a freelance web developer

There are lots of reasons for doing freelance work, whether it's for what people refer to as a side hustle or whether it's more of a full time freelancing goal, but in all cases it's necessary to have a minimal infrastructure setup. You fine tune this over time and adjust according to the direction of the projects. Broadly speaking the essentials are payments, contracts and some form of marketing.

I got some of the details from this article from an audio on soundcloud called Web dev freelancing high level overview basic setup. The author makes some good points and is worth the listen.

Payments

Freelancers need a way to receive payments. There are a lot of online services that specialise in providing services for freelancers. Freshbooks seems to be a favorite for a lot of web developer freelancers. I hear it mentioned a lot on podcasts. Whichever service you choose, you will need to create an account, connect it to Paypal, and connect it to your checking account. It's a good idea to setup a seperate checking account for your freelancing work than your personal banking account. Then it's a good idea to test out the setup by sending a $1 invoice from Freshbooks, pay and ensure that the money reaches your paypal and then bank account.

Another possibility here is to use a partly self hosted solution. You will need some web development skills, but you could use the Freelancer project (One of my projects) to host a payment site that uses Stripe as the backend payment provider. You can host the code on a public Github repo, so customers can inspect the code if they want.

Joan from Toptal contacted me after reading this post to inform me that they have just released a freelance calculator tool that could be useful to easily figure out your hourly rate and yearly income. Thanks Joan!

Contracts

You will need some basic freelancer contracts that will clearly set out the project goals and responsibilities, and a way to send these contracts back and forth securly. Docusign has been recommended a lot, but there are other similar services available online, easily findable via Google search.

As far as the contract google search 'standard exchange of services contract' or 'web development contract' and use one of these as the basis for your contract. Make sure that the contract contains at least the following:

  • State that you will build website, describe the website, pages, api etc
  • State the price for building the website
  • Client will pay 1/3 before, 1/3 midway, 1/3 at the end
  • After contract finishes, will leave project
  • Hourly rate applicable for maintenance after contract finishes, should be arranged as a seperate engagment

Also worth checking out is this article that has lots of details about freelance contracts.

Marketing

Some way to promote your services, this doesn't need to be too envolved initially.

A basic website with a logo, contact details, short description of the services offered. Something you can add to the footer of your emails, it will be useful for referals too, a way for other to point people towards you. You might want to setup a blog at some stage, but initially a basic 1 page website should be enough.

How to deliver

General advice is to deliver a working website using new accounts for hosting and give these to customer as a deliverable. Hosting your customers website is not advised.

Now you have the basic infrastructure to make proposals, draft contracts, and receive payments.

Related posts:

List of freelance web development websites and marketplaces

List of remote friendly web development job board websites

2020/01/16

2020/01/15

2020/01/14

2020/01/13

2020/01/12

2020/01/11

2020/01/10

2020/01/09

2020/01/08

2020/01/07

2020/01/06

2020/01/05

2020/01/04

2020/01/03

2020/01/02

2020/01/01