Post by @harrysintonen@infosec.exchange

2026-08-31 12:14 UTC

I spent nearly six hours tracking down a bug that was caused by not blocking signals correctly. As a result, signal processing could occur in the middle of a critical section. To make matters worse, the crash would only occur much later when the desync from the earlier overlapping execution would result in use-after-free and crash. The crash would not occur every time since it was timing-dependent - the signal would need to hit a relatively small window to trigger the issue. Figuring out the root cause was a lot of manual elimination of potential causes and old-fashioned mental gymnastics. When everything else was ruled out, the only explanation was that two supposedly synchronised calls really were overlapping, despite this being supposedly "impossible". Once this fact was established, I could turn into verifying signal blocking, and the root cause was found rather quickly. Is it weird that I loved spending six hours of my Sunday on stuff like this? #development #programming #problemsolving

Replies (3)

@harrysintonen@infosec.exchange 2026-08-31 12:18

I'm sure that some LLM could have been thrown at the issue, too, and after some prompt massaging, the root cause might have been found, eventually. But what's the fun in that? Would I have felt as accomplished with my witty prompt massaging vs actually using my wetware to work the issue itself?

Open ##4584865
@harrysintonen@infosec.exchange 2026-08-31 12:24

I should also add that I added the very bug myself years before. That kind of dampened by joy of root cause discovery. 😆

Open ##4585029
@Arpie4Math@mathstodon.xyz 2026-08-31 17:09

@harrysintonen@infosec.exchange Most of my career has been as the indispensable engineer in the IT department, making things work. One day I determined that the product engineers were shipping native code libraries for a Java project and had never bothered to use even thread-safe libraries.* They didn’t want to hear it from “the IT guy” so I went through their code, built it with the right libraries and calls and the “intermittent crashing problem”* went away. I cited the official Sun guide to native code libraries so this was a week-long exercise of RTFM to a team short on that deep platform knowledge of how things work. Junior coders masquerading as senior engineers. * Threads and other forms of concurrency can result in two different flows of execution using the same memory so unless steps are taken to mitigate it, an algorithm could assume the zero it stored would still be a zero when it came time to read it and depend on that fact unaware the other flow changed it to one. Java could be written as if it was a single flow of execution but it would, unasked, use just such a second flow of execution to clean up the memory allocated by the main program, so all code needed to be thread-safe. Many historical C libraries had routines which were unsafe because it was impossible to mitigate the risk described above. And “intermittent crashing” is precisely the type of problem that is caused by some unsafe code failing due to circumstances (threads concurrently using the same memory) more likely to happen in long production sessions under high loads that the developer desktop tests. They needed a minimum amount of platform knowledge for this product and their customers were right to complain until an IT manager asked me

Open ##5152987