Fixing an out of memory error
2024-05-02 05:54:00 +07:00 by Mark Smith
Yesterday I ran into a memory error which can be a scary error when you are developing software.
It's scary because it often means the software you are developing has a bug which is causing a memory leak. That's when as the program runs and the memory garbage collector isn't able to clean up some part of the memory you are using. The leak bug tends to be located in some form of loop, so the program runs, but gets slower and slower because each time around the loop the memory of the machine the program is executing on fills up a bit more. Eventually it crashes with a memory error. Typically memory leak bugs are very difficult to find. You need all sorts of special tooling, which you run to profile your code, then you analyse graphs and huge amounts of nonsensical looking data from memory dump files. It's a big ordeal.
The other possibility with memory errors is something less scary. Sometimes you are just pushing the program to reach the system limits, there's no bug, it's just that you have inputed so much data into your running program that the program runs out of memory. There is the possibility that the program could be refactored to use less memory, but there is no memory leak bug hidden somewhere. The program is behaving as expected. In these situations increasing the memory allocation assigned to the program tends to fix the issue, and that's quite straight forward to do.
I fixed the memory error by adding the --max_old_space_size option to the node process. Before doing that though, I verified that running the build using the posts data set from the week before completed without error. That confirmed that the memory error started happening after that point, that I had just added a few too many blog markdown files, which pushed the program to fill up it's memory allocation.
You can up the memory allocation directly on the node process:
node --max_old_space_size=4096 app.jsThe other way to do it if you want a bit more flexibilty and you want to keep your npm scripts tidy, is to set an environment variable called NODE_OPTIONS. When node runs it will pickup any options you put in this variable and apply them:
export NODE_OPTIONS=--max_old_space_size=4096You can add that to your bashrc file so it get set automatically. That's outside of the scope of this article.
Since I'm running the build in a Github action, it's easy to set a new environment variable either at global or step level using the following syntax in the workflow yaml file:
env: NODE_OPTIONS: --max_old_space_size=4096That's it now when you run the static file generator it should hopefully complete without a memory error.