Pointed Development

It’s Bugs All the Way Down

| Comments

Wherein I learn that some developers don’t know how to play well with others and get upset about it.

First, apologies to Stephen Hawking and Dan Rosenberg for the title.

Second, some back story…

Years ago, I was responsible for a web-based application that consumed a small XML payload, performed some calculations and spit out a small XML answer.

The application consumed and produced very precise coefficients, such as 0.000000000000007.

As the application matured and time progressed, we eventually had to split the calculation model from the validation model in order to handle change-management procedures. Additionally, we were slowly aligning our core platform with a future-state vision, little-by-little moving chucks of code from Platform A to Platform B.

In order to prevent a platform war and to Maintain Focus, I’m going to simply state that I developed on Platform A, while another team within my organization developed on Platform B.

To validate code changes in Dev, we’d run tens of thousands of rows of test data through both the validation model and the calculation model.

Test data was always constant.

Production data was variable.

Uh oh.

A Bug is Born

As I said, the calculator consumed and produced precise decimal values, with very high values for precision and scale (SQL Server DECIMAL(38,20) if I remember correctly).

More often than not, we’d produce low(er)-precision answers, such as 0.00005 and everything hummed along just fine. Every once in a while, though, the stars would align just so and the calculator would produce a higher-precision answer, such as 0.000005. Platform A, the calculator, had no issue with this level of precision, and Platform B, the validator, consumed our high-precision values just fine.

Platform B would occassionally run off the rails, however, refusing to send back a validation answer. The validator would silently consume its exception, returning nothing to the calculator, which in turn would raise an exception.

Our test cases never caught this issue.

After a couple of days of research, I discovered that the serialiazation library used by Platform B wouldn’t properly serialize decimal values, returning something like “5E-6″ rather than “0.000005″ but only for a certain level of precision.

The bug unequivically existed on Platform B due to a buggy library!

When Platforms Collide

How terrible to come in to work one day and find that scores of answers produced by your application are incorrect.

It took me a day or two to track down the bug, and once I found the bug, it took me another day or two to gather enough evidence to present to the validation team that the issue resided on their side.

They refused to fix the issue.

I mean, they flat out refused to ackowledge a bug even existed, even after I presented solid evidence to the contrary[1]! Feel free to read between the lines.

It took another day to implement a fix on my side to get around the bug in Platform B.

I Don’t Care, Fix It

Overall the bug took a week to fix. It involved a simple hack to truncate decimal values at a given precision on the calculator side so that the validation side could consume the data without exception.

It was not a happy fix. It was not where the fix should have been applied at all, but it was the best we could do given the refusal by the other team to upgrade their library, or otherwise implement a workaround (which they easily could have done).

It’s Bugs All The Way Down

We’re developers. We have an obligation to our client to fix their problems. Sometimes the bug actually exists in your code, sometimes it exists in a dependency. But dammit, fix it!

It’s our most important function!

I was floored that my own team members wouldn’t fix their own issue, just because they reported under a different organizational structure. They wouldn’t even admit the issue existed on their side despite chrystal-clear evidence to the contrary.

It’s unacceptable to consider yourself a developer and refuse to troubleshoot, diagnose and fix bugs.


I’m so thankful that for the last three or four years, I’ve worked with great people – exceptional developers, particularly at TrackAbout and IBM – that actually care deeply about the code they’re writing.

And fix bugs!

It’s refreshing.

When you have the right mindset, bugs actually get fixed! And more often than not, they get fixed the right way!

Moral of the story: Fix your bugs… It’s bugs all the way down.


1. JAXB was broken and has since been fixed.