REM State

You just synced your source tree with the repository — and lo and behold, the application fails to start once the build is complete. Naturally, your first response is to do a complete, fresh checkout in a new directory to verify that it is something in the repository that is broken, as opposed to some stray local modifications you had made… but the program still refuses to run after you sync. It worked fine yesterday; what gives?

This is probably one of the easier scenarios to deal with — given that the program worked before, and is now exhibiting erroneous behavior, it is safe to say that the source of this bug was introduced between the current version and the last version that worked. Your version control system (you are using version control, right?) is built for this very task — to let you go back in time to when the program worked, and discern what changes were made that caused it to not work. In the simplest case, you can simply revert a complete checkin to solve the problem. In the worst case, so many checkins have been made (or the diffs are so messy) that the source of the bug is buried in all the cruft. However, it is always possible to revert changes in reverse-chronological order until the bug goes away: that will, at the very least, provide a discrete checkin that introduced the bug.

This technique is extremely powerful, because it will sniff out any type of bug, including threading, dangling pointer, and other non-deterministic bugs. It’s also the go-to if you have automated tests that run on a regular basis, and some of those tests went from passing to failing. Because of this, it is highly recommended that all devs be conscious of the effect that their checkins have. Source should be formatted to keep diffs readable, so that modifications will stand out clearly. An excellent example of this crops up with SQL embedded in other languages, such as PHP. Which diff would you find easier to understand:


AND meta_key=’_series_order’
- AND CAST(meta_value AS UNSIGNED) > $position
+ AND CAST(meta_value AS SIGNED) > $position
AND $wpdb->posts.post_author = ‘$author’
AND $wpdb->posts.post_date < '$now'
AND ($wpdb->posts.post_status=’publish’
OR $wpdb->posts.post_status=’static’)
- ORDER BY CAST(meta_value AS UNSIGNED) ASC
+ ORDER BY CAST(meta_value AS SIGNED) ASC
LIMIT 1″);


Or:


- AND meta_key=’_series_order’ AND CAST(meta_value AS UNSIGNED) > $position AND $wpdb->posts.post_author = ‘$author’ AND $wpdb->posts.post_date < '$now' AND ($wpdb->posts.post_status=’publish’ OR $wpdb->posts.post_status=’static’) ORDER BY CAST(meta_value AS UNSIGNED) ASC LIMIT 1″);
+ AND meta_key=’_series_order’ AND CAST(meta_value AS SIGNED) > $position AND $wpdb->posts.post_author = ‘$author’ AND $wpdb->posts.post_date < '$now' AND ($wpdb->posts.post_status=’publish’ OR $wpdb->posts.post_status=’static’) ORDER BY CAST(meta_value AS SIGNED) ASC LIMIT 1″);


It is pretty easy to see what’s going on in the first case, whereas the changes in the second case are lost in the sea of text. It is hard to spot changes in code like this, so it is important to make sure that code gets broken in logical places. Note also that the former block is formatted such that new statements can be added without affecting the surrounding statements at all — e.g., it is possible to add another AND clause with only the new line showing up in the diff. This greatly improves the signal-to-noise ratio, and is the only non-religious reason I have come across to prefer one bracing style over another.

Given that the source is formatted so that changes stand out, the next big thing is to ensure that each checkin does one (and only one) thing, and that the purpose is clearly stated in the version log. These two concepts are closely tied together: when bug-hunting, the log entries are the first thing you use when trying to filter out which change set is most likely to contain the bug you are looking for. If changesets try to do more than one thing at a time, it becomes more difficult to dissect the diff, and it also increases the likelihood that the log entry will either become too long-winded, or miss mentioning a critical detail about what was changed.

Unfortunately, because a multitude of factors can make this debugging technique infeasible (messy checkins, initial checkins, misleading checkin comments), we can’t always rely on it. Next week, I’ll discuss the developer’s old standby: debugging/tracing.