In a recent post I recommended watching the startups for insights into UX. One example of a useful UX technique that came out of the startup space is the cohort analysis.
A cohort analysis is a tool that helps measure user engagement over time. It helps UX designers know whether user engagement is actually getting better over time or is only appearing to improve because of growth.
A cohort is a group of people who share a common characteristic over a period of time. In a user engagement cohort analysis, we group people based on their join date. The people who joined our service in January make up the January cohort, the people who joined in February make up the February cohort, and so on. We then investigate how each cohort stays engaged over time, comparing the cohorts against each other to make sure that the people who joined in February are more engaged than those who joined in January, for example.
One reason why the cohort analysis is valuable is because it helps to separate growth metrics from engagement metrics. This is important because growth can easily mask engagement problems. If you’re successfully adding lots of users to your service, your overall engagement numbers will look positive because those new users are relatively well-engaged, spending lots of time on the site in the beginning. If you only looked at overall engagement numbers then you would think that your service is continuously getting stronger.
In reality, however, it may be that people stop being engaged after a couple of weeks on the service. They might leave for any number of reasons: it’s not useful, the novelty wore off, they added all their friends and now have nothing to do, etc. But the lack of activity of these users is being hidden by the impressive growth numbers of new users…there are enough people being added to the service that the lack of engagement from a small number of folks just doesn’t show up.
This is where the cohort analysis proves valuable. By bucketing people into the month (or week) they started using the service, you can keep track of their engagement over time. You can now make assessments like “the March cohort is engaged better than the February cohort” and the like. If your numbers are flat month over month, which is often the case even if the face of impressive growth, then you have not improved user engagement over that time.
Twitter investor Fred Wilson suggests that every startup uses a cohort analysis:
“I’d encourage everyone doing a web startup to adopt the startup metrics methodology and within that methodology, make sure you are looking at cohorts of users, not just all of your users in the aggregate.”
And this goes for most businesses as well. The cohort analysis can be a valuable tool for UX professionals who need to measure user engagement over time.