Implementing Startup Metrics for Pirates using MixPanel – Part 1: Features

Perhaps the biggest advantage of providing software as a service is you can push out changes immediately – there’s no upgrade process and users get access to the latest features right away. Developing your product, you want to work on these cool new features so you can sell them to your users, the question is what do you work on next and how do you know if what you just released is being used?

The idea

Your first job is to work out what your goal is when adding new features. Are you trying to use them as selling points because users are asking for them or are you trying to get people to stay on your site longer? Are you trying to help users use existing features or are you making subtle tweaks to improve the overall experience?

Once you have decided, you can work out what you need to measure to determine if the goal has been reached. And measuring is the key – there’s no point working on improving something that is never used. That usage data can then help you figure out what to do next.

This is all part of the “Startup Metrics for Pirates” presentation given by Dave McClure:

The idea, put into practice

Both this and last weekend, we pushed out two minor feature releases to our server monitoring product, Server Density, which I’m going to use as examples of how we use metrics.

Auto refreshing dashboard

The whole point of server monitoring is to see what is going on with your server and we store real time data for each of the key metrics. A summary for each server is displayed on a dashboard within Server Density and as of last weekend, this now refreshes automatically every 60 seconds. The idea is that this can be kept open on a second monitor or TV to give you an “at-a-glance” view of your servers. The refresh is implemented using AJAX and JQuery and within that we record every time a refresh is triggered.

Why? We can keep people using the Server Density UI as part of their daily toolset and have it always at the back of their mind. Other people will also see the product being used.

Help popups

Some form fields in Server Density can be improved by providing further advice to the user about what to enter in that field. Yesterday we pushed out an update to include an “on hover” popup to provide some guidance for that field. We record every time a popup is triggered.

Why? To help people use the form and track which fields get the most help “requests” to look at ways of making it easier or more obvious.

Measuring

There are probably many products that can be used to do the tracking (indeed, I believe it is possible with Google Analytics) but we are using MixPanel.

Using some very simple JavaScript calls, we are able to track things within our application and include as much meta information as we want. We include their code to measure page views (so we can see which areas of Server Density are being used most) but also with specific actions.

All that is required after including their JS is a call to their track function with whatever meta information we want:

mpmetrics.track("page-view", {"page" : "dashboard-ajax-refresh"});

This can just be part of the page JS (as it is for the page view tracking) or within your own JS functions (as it is for the dashboard refresh and help popup).

The results

Following the introduction of the auto refresh dashboard, we saw the daily actions in our UI increase from around 260 to almost 12,000.

which corresponded with an increase in the average number of actions per user from around 10 per day to almost 300:

And for the help popup we can get a breakdown to see which fields users are requesting more help with:

Conclusions

Although it’s too early to draw any conclusions from the help popup metrics, we can clearly see that the addition of the refreshing dashboard has increased the amount of interaction each user has with our UI. This means they are making more use of the service.

Without the use of these metrics we would have no idea whether the effort we put into the development was actually having effect.

Part 2 – Funnel & Conversions

MixPanel can also be used to track your signup funnel all the way to conversion. Part 2 will look at how we implemented those metrics, what we are looking at, what we did as a result and how that affected our conversion rate. We are halfway through this process so expect that post near the end of next month.

UPDATE: Part 2 is now available here.