markjgsmith

How to rebuild a project from scratch

2025-10-02 12:52:21 +01:00 by Mark Smith

Rebuilding a project from scratch is one of those things that you never really want to do, and so there is a tendency to avoid it. But sometimes you realise that some or all of the foundations are compromised in some way, and the only way to fix them is a complete project rebuild. That just happened to me, and though it was pretty scary, and a lot of effort, I got through it, and it actually wasn't as bad as I thought it would be once I made the decision to do the full rebuild.

Keep in mind that I was the only person working on this repo, if you are working on a repo with others, definitely discuss with them before undertaking anything like this.

I think this is a situation that will become more common place now that many of us are using powerful AI/LLM tools, because as much as you can gain tremendous ground in build outs, it's very easy for instability to creep in without you noticing. This is something that would have happened anyway without these tools eventually, it's just that you reach that point a lot faster. Anyhow you will likely find most of this is just common sense, but when you are in the thick of it, seeing the woods for the trees isn't always obvious.

So what happened? Well the past month or so I have been building an Oauth 2.0 REST API backend with a React frontend. And I have been using AI/LLMs and devcontainers. Quite a few ups and downs, but I was making some seriously great progress, at least from the perspective of the application code. Both the frontend and backend were super solid. I was on solid ground, or so I thought. While experimenting with running Ollama locally on my laptop I uncovered some problems with my dev environment. The devcontainers I had been using started crashing on startup. I tested various configurations, and that uncovered some misalignment in my typescript and linting configurations, and before I knew it, I was in a giant tangle of errors.

I am big into writing minimal projects. Over the years, I have written many of these, I have a special section dedicated to them in my portfolio. When you get into difficulties it's often a great way to find stability again. There is a real art to it. In this specific case, things were so tangled that it wasn't obvious at the outset what exactly the minimal example should be. I thought initially that what I needed was a minimal example of an ollama based webdev environment, and spent a while doing that. Eventually I realised that ollama was complicating things, and that actually what I needed was a NodeJS Typescript Express React minimal example. A minimal project all built on containers. That was a bit of a voyage in itself, but once this new well structured project was working, the big challenge was to backport it into the original, much more complex, REST API + React project.

I thought initially that I would be able to move over pieces of it bit by bit into the old structure, reshaping as needed. It became obvious that was just never going to happen. I spent hours just pushing small pieces around, testing and seeing everything was still broken, and being totally overwhelmed by the enormous mountain of files, all out of shape, plagued by lots of false starts and bits sticking out all over the place. It was a monstrosity. I knew I now had a known good structure, but I didn't want to start a fresh project because I didn't want to lose the project history. There was a lot of valuable information in there. Here's what you do.

• First of all, backup the main branch, call it something like main-backup. Also backup the entire project into a tar ball.
• Ok you backed everything up. That's good, but also very very important, backup the project git directory (.git). It's a hidden dotdirectory. You will only be able to see those with ls -la.
• I decided to not use AI in Vscode for any of this. It has a tendency to try and take over and lead you down terrible paths. I did use AI but only via web browser. I was using Gemini.
• Create a new feature branch, call it something like refactor-new-project-structure.
• Make sure you are on your new feature branch, and then delete everything! Literally rm -rf * in your project directory. You read that correctly. This is the scary part. Remember you have backups. If you don't have backups, go back to the start of this list and make backups! Remember to delete all the dotfiles and dotdirectories in the project directory too. I just did that one by one, after deleting all the regular non-hidden files. One important exception: Don't delete the .git directory. That's where all your project git history is stored!
• Recreate / copy across your minimal working project to your new feature branch.
• Get that working, test it's all working. There is no reason it won't work since you already got it working in a separate project.
• Now pull bits in piece by piece from the old main branch into your feature branch. It's a bit tedious, but you can do this relatively easily. Take your time. Don't miss anything. List, view and checkout files and directories direct from the main branch using these commands:
  • git ls-tree --full-tree -r main
  • git ls-tree main:some/folder
  • git show main:path/to/some/file.js
  • git checkout main -- some/folder
  • git checkout main -- path/to/some/file.js
• Add and commit each piece with a descriptive title. Keep testing it's still working along the way. Ditch anything you no longer need from the old main branch.
• Eventually you will have recreated your project, and it will all work again!
• You will have a long series of rebuild commits. I had about 30 commits.
• Backup the rebuilt branch, call it something like backup-rebuilt-project.
• At this stage, things were still kind of complex, so I asked AI to help me group these into 6-7 groups of related commits that describe the rebuild process, so I could squash them into a more focussed history. I copied and pasted into the chat the relevant commits listed from running git log.
• The AI came up with a great rebase plan, with the commit titles and the commits all grouped. I just had to copy and paste into the rebase in neovim. It also prepared titles for each of the 6-7 squashed commits, which I copied and pasted into the rebase as it progressed. I rebase squashed the slightly haphazard 30 commits into 6-7 tidy and focussed commits.
• At this stage, make another backup of the rebased branch, just in case!
• Now change to main branch, merge in the squashed branch. In my case there were literally no conflicts, which was a relief but not too surprising.
• Backup the merged main branch into your final branch that you will merge in on Github, call it something like refactor-new-project-structure-final.
• Reset your local main branch to the state of the Github remote main branch.
• Push your final merged branch to Github and create a PR.
• Merge the PR into main. I got the AI to write me a comprehensive PR message that summarized the rebuild.
• Pull latest changes to your local main branch from remote, test it's all working on main branch locally.
• Now that everything is merged into main, and tested, cleanup by deleting all your backups and the feature branch.
• You are done! Do a retrospective on the nightmare you just completed. What did you learn?

You see it actually wasn't that bad, mostly common sense.

In my case the biggest structural change I had made was to move several top level folders which were all npm workspaces under a single packages folder. Git is quite clever and can mostly keep track of where files have moved to. Apparently having a focussed series of commits makes that easier. Aside from that most of the changes were to do with typescript and eslint configurations. The bulk of the code was completely unchanged. The only code that had changed was some of the npm scripts and Dockerfiles/Containerfiles. But there wasn't that much to merge in, just a few things that were in the original files that I wanted to keep, like installing neovim and networking tools for the devcontainer image. I just copy and pasted those across manually, and tested it was all working.

There were a few odds and ends to smooth out, like updating to a more recent development base image, so neovim had all the libraries it needed, and a bit of a refactor of how the backend production image assembles the final production node_modules folder to include the shared workspace. On the whole though, it was a lot smoother than I expected, even if it was kind if scary.

If you are in a similar situation, I know it sure isn't that much fun, but hopefully reading this will help somewhat. Sometimes it just helps to see something similar mapped out and to have the knowledge that it's possible. #