Airbrake leadership came to me with a grandiose goal in mind: to offer its users an enhanced, intuitive, and efficient dashboard for managing their cloud resources. They were building a solution that would cut down hours for complex tasks of hunting down the catalyst that broke developer code, without compromising on the depth of functionality that was needed to support the best practices within the industry.

Airbrake is made up of two distinct areas:

• set-up side where developers would install tools that track their codebase health

• surface-level side with a list view of errors that occurred in the repository (split up by apps, varying programming languages, environments and users that saw the error).

The list view is akin a detective interface a developer would use to look for clues to find the first error that occurred and propagated across the whole application. In theory going by volume and finding the object called the most means the function evoking it was causing the problem. Elimination of most everything else meant dev eyes were focusing on figuring out what was wrong as opposed to finding where the hole was to begin with.

The problem? Airbrake was seeing a lot of churn and they were bleeding money. So if everything was working as intended and developers were saving time, why were they jumping ship?

Investigation

One of the first changes made to how we did things was introducing exit surveys and hunting down users who have cancelled within the last 6 months to understand their reasons. Following those we couldn’t rule out that a lot of user pain causing low retention occurred during the setup phase. A hastily-installed back-end resulting in mismatched API calls could not only render the service useless but could confuse and waste users time looking for the cause. Indeed, a lot of our users, especially non-technical ones, suggested that setup instructions were difficult, so we made that step easier to overcome with an easy way to email install instructions to code owner.

Having done that, we gathered that the rest of those interviewed felt they were spending more than optimal amounts of time navigating through the interface – scrolling and clicking within the error view itself, rather than analyzing the high-confidence data presented by Airbrake to follow the trail suggested.

Not only were we directing our user into laborious manual flows, the UI made their lives harder:

Error messages truncate, can’t see line numbers, similar or the same errors in different files should not be grouped together

This was both upsetting but made us feel optimistic. If all of the data was there, we just had to present it in an intuitive way that was most usable to the user.

Following the next few weeks we’ve outlined the main problems we had to address:

• It's hard to scan our list of errors at a glance

• It's hard to read summaries of errors continuously occurring

• It is hard for users to see the important data within a single erroneous object

The challenge was to use fewer UI patterns within a single screen in order to minimize distractions within the interface. A lot of visual noise was reduced by hiding non-essential details for unselected errors and focusing the user on the step of the flow they were within.

Errors that could be accompanied by a concise visual summary – graphs and key stats – were, to help users with navigating through what was happening in their code.

Addressing groups of similar error instances, we introduced a tab eliminated more than half of clicks in the interaction eventually speeding up the workflow, and improved side-by-side comparisons.

Non-visual tools like enhanced sorting and the addition of filters (e.g., by text, environment, date) were introduced for exploration support and overall improvement.

By continuing to ask the right questions at the right time of customer lifetime we allowed ourselves to quickly address issues that resulted in customers leaving. We could've gone after releasing dozens of new features but instead prioritized usability and stripped interface down to wha mattered most.

Then after stopping the bleeding we focused on experience: users wanted to track their code health on mobile phones and be part of a strong community.

After we realized that the stickiness of the tool was so high we could only hurt ourselves, we took customer obsession to the next level, increasing our marketing efforts, hosting and sponsoring hackathons and meetups across the world and eventually got acquired by the mastodon of the industry – Rackspace.