Module caches and reusable workflows
2024-04-05 23:16:00 +07:00 by Mark Smith
The last few days have been a bit of a disaster development-wise. I was forced into pressing forward down a difficult path because I was blocked for many days on all other paths. It ended quite badly, and I was unable to deploy to the website and a bunch of stuff broke.
I was able to fix it within a day, but it was kind of scary, because since I'd been forced into pushing directly to main branch on several repos, the only way out was to reset hard the branch, losing several days work. Anyway it's fixed.
All sorts of irony surrounding the whole episode. Here from a note I wrote mid way through:
"The very thing I was worried about happening, which was the reason I was tidying up the code, actually happened when the world blocked me from tidying up the code!"
There were a couple of useful take aways from the exercise.
Firstly I've found a better way to implement the npm module caches used in Github Actions that makes it easier to refresh them when code changes in the plugin modules. That's a big deal because it was a massive source of confusion and wasted build minutes. I've also removed all caches that were used in plugins. The plugins now use simplified local versions of the server components in tests rather than those in the plugins. I guess you could call them mocks. That means there's no dependencies, no need to install plugin repos, so no need for an npm cache. The only repo that still needs an npm cache is the main website because it has to npm install all the plugins.
That should make things much less confusing. Imagine how complicated it was debugging the main website with a cache, when the bug is in a plugin that also has a cache.
Another thing I realised earlier was that the work I did trying to tidy up the plugins by consolidating them into a couple of repos might not have been for nothing. Reason being that I got reusable workflows operational between private repos. Theoretically it means I should be able to consolidate all the plugin workflows into one workflow that they all use to build and deploy. That will be much easier to maintain.
The final thing I realised was that it would be nice to update the cache refresh logic into a seperate reusable workflow. Currently the build deploy backup workflow is dual purpose which is confusing and error prone. Since I know reusable workflows between private repos work, that might be possible. But for it to work the caches need to be useable across workflows in the same repo. The cache refresh workflow would create the new cache, which could then be used by the build and deploy workflow. I'm not sure if that's possible yet.
To be clear having lots of private repos is still a bit of a headache, but all in all if I manage to get all or most of these optimisations working, the simplified cache setup and consolidating the workflows might make the overhead of having seperate repos for each plugin just about manageable.