An interesting thing about my recent sprint to get the multi-target users working. I had been working for several days straight, through much confusion and all sorts of head winds, but got to the stage where all the unit and integration tests were passing on all parts of the software apart from core. I had been working with the LLM along side, helping me on many things the whole way through. Now we did run into a lot of issues along the way, but we were always able to work them out one way or another.
So when we got to the final push I gave the LLM the full list of failing tests. And we worked through them one by one, somewhat laboriously but we made good progress. That was until we got to the final test. There was literally 1 test left in all the several days of work, and wouldn't you know it, the LLM suddenly became completely incapable of doing the most basic of things.
It started deleting useful comments in the code that we absolutely needed. This is something that happens quite regularly because I guess there is a prune directive that often kicks in and it‘s always trying to prune and tidy up stuff which can be useful but also a lot of the time is not useful. Normally it suffices to say don‘t do that, but now that we got to the final test, it not only didn‘t stop deleting comments, but started changing the comments that it didn‘t delete. When I pointed this out it started changing whole blocks of code that had zero to do with what we had been fixing. It was extremely dangerous, because this was code that was very difficult to write, and had I not noticed, it could have been disastrous.
It became clear after several back and forths that it had become belligerent and no matter what I would say it would claim to have fixed things apologising while simultaneously making things much worse. I ended up giving up on the LLM and doing it myself.
This is a pattern I have seen a lot, and I wonder where it comes from. Remember LLMs are trained on human data, so is this something they have somehow learned to do from analysing our data? Or is it something more nefarious? The behaviour is night and day, it‘s like somebody has flipped a switch, it somehow knows this is the very end of something important and tries to f-it up.
Side note, I suspect it might be some strange reflection of humanity because I see vaguely related and obliquely related instances of this pattern all the time in the real world.