Pay no attention to the developers behind the curtain.
I’ve relayed this story many times over the years and it remains one of my favorite developer anecdotes.
As is common, the requirements of the business do not always align with the reality of software development. Occasionally, we’re asked to do something really, really stupid for business to succeed. This was very much the case for the company I worked for back in the early 2000s, when I helped develop a web-based front end to a small subset of client data. I’ve touched on similar stories with this company in the past: The environment was very much the Wild West of developer environments.
The web-based front end was very new and we had spent a lot of time writing all of the various components, but it wasn’t robust and it wasn’t solid and it certainly wasn’t production-ready. In fact, it had a lot of bugs that we simply hadn’t ironed out, and we were discovering new bugs on a daily basis.
Despite our best efforts to the contrary, however, we were tasked with bringing in production client data for an upcoming demonstration with said client. The company president and VP were flying to Scotland to present our front end, bugs and all. The demonstration was already set up and we (the three-person developer team charged with the task) had adequately prepared as best we could.
However, just before the demo we discovered a problem. A big problem. The site would crash during a sort operation, and the demonstration of the sort was elemental to the demo. It’s two days before the demo and we have broken code.
We tried, desperately, to find the root of the problem to no avail. We could replicate it as long as we weren’t in the debugger: During debugging, we discovered the problem didn’t exist at all! In fact, the bug didn’t manifest inside the debugger if and only if we stopped at a specific breakpoint! Under any other circumstance, the application would fast-exit and present an ugly Internal Server Error to the client, but if we stopped at one specific breakpoint and simply continued execution, the application worked just fine!
Our president and VP were adamant that the demo go off without a hitch. It was very important to the business. Of course.
So, we had a work around. We could execute the demo as long as we had a developer sitting at a debugger on the server, waiting for the breakpoint to hit. The developers job, then, would be to hit F5 when they saw the breakpoint. This introduced a delay, but due to the slow pipe overseas, the client (probably) wouldn’t notice.
But the demo was overseas and the developer had to be at the office at the server between 1:00 and 2:00 AM (I don’t recall the exact time, but it was late). What if the developer had to step away? Or fell asleep? Obviously the developer needed backup, and a manager present. Obviously.
So there we are, Katie the manager, Chris the developer and me as backup and we’re sitting around a CRT, all of us staring at the Visual Basic 6 IDE (yes, VB6). We get the call that the demo is about to begin and we wait for the breakpoint to hit. We had snacks. We had drinks. We were having a good time.
And we’re waiting.
We had no idea what was happening in Scotland, so we were just… sitting around. Three developers, huddled around a monitor, waiting for a breakpoint to hit.
And suddenly, there it is.
The line highlighted in yellow. We hit F5 and away it goes. And there it is again. And again. Each time, we hit F5 and each time the site functions as expected.
We get a call letting us know the demo is over, and we all return to our respective homes. For all intents and purposes, the demo was a success and the client was none the wiser.
I don’t recall, unfortunately, what we did to fix the bug. The whole component library was pretty complex, and the bug may have involved an external core dependency… but I’m just speculating. It was so long ago that I don’t remember the technical details, just the human story of sitting around a server with a couple of really great people enjoying the absurdity of the whole thing.
It’s the people I’ve learned to appreciate over the years, even if the job is occasionally stupid.
We had one job. It was stupid. But we did it. And I enjoyed every moment of it.
Have you ever had an Oz-like experience?