TL;DR from Chat GPT
Cohort analysis is a powerful method for understanding user retention. Cohorts are groups of users with similar traits who perform a specific action within a defined time frame. Net 30 day retention is a versatile criterion for cohort analysis. Monthly cohorts can reveal customer retention patterns, and cohort metrics can be aligned with monthly reporting. Consider cohort segmentation, cohort comparison, cohort size and duration, cohort retention curves, and applying cohort analysis to different business metrics for a comprehensive analysis.
Intro
I have accumulated a decade of experience working with cohort analysis in various capacities, including stints at Google, Spotify, and my current role at Ark Kapital. In this post, I will delve into several different ways of defining cohorts, which can have significant implications for the insights you can glean from the analysis and their validity. Please note that this is not an introduction to cohort analysis; for that, you may need to look elsewhere. Consider this as a step two, building upon your understanding of how cohort analysis works.
Cohort analysis involves analysing groups of items, and in this post, we will specifically focus on customers as the cohorts. However, the same logic can be applied to other entities such as active users, suppliers, or any other group of items you wish to analyse.
Cohorts are a valuable tool in customer analysis because they enable us to study groups of customers, which tend to exhibit more consistent behaviour compared to individual customers. For instance, while each customer may behave in a unique way, analysing a large group of customers allows us to focus on the 'average customer' and mitigate the impact of outliers on metrics. Defining cohorts involves making decisions about how to group customers, determining the conditions under which a customer remains in a cohort, and establishing the frequency at which metrics are tracked. These three dimensions offer various options, resulting in different permutations for cohort definition, each with its own advantages and disadvantages.
Cohort Metrics
Cohorts are useful to track groups of users over time, and hence metrics are generally concerned with retention, engagement and monetisation of those users. While you might easily calculate that “5% of your active users made a purchase in a February”, you might be interested in knowing that “2% of users who registered in February, made a purchase already in their first week”, and compare that number to the same metric in January.
Cohort metrics are hence tied to either the share of, or average value of, some action in a given time that is relative to when the user entered the cohort. In the example above, the metric is the share of users making a purchase, and the relative time is the first week after registering.
Common metrics are the share of users who are “still active N months after their first activity”, or “still subscribers N months after buying their first subscription”.
Another set of metrics are cumulative. For instance, you might have a cohort of 1,000 users, and 12 months after the “start of the cohort”, the total amount of goods they’ve purchased amounts to $3,200. You can then say that the average user spends $32 USD in their first 12 months. With some financial modelling added (subtracting cost of sales and cost of capital etc) you can come up with a Customer Lifetime Value this way.
One important thing to bring from this section, is that most — if not all — interesting metrics will have the cohort size as a denominator. The numerator being some metric like number of retained customers, or total sum of money spent. In the “Group Retention Criteria” section below, I’ll discuss an important aspect of when customers are allowed to be included in the numerator side of the equation, whereas the denominator (cohort size) should stay constant.
Defining (and measuring) the group
The most common way to divide users into groups is to define an event that they have in common, and that happened within some time frame. For instance, you might be interested in tracking your users’ retention after signing up to your application. The capture how this behaviour changes over time, you might create “Monthly Registration Cohorts”.
A Monthly Registration Cohort will hence consist of all users who registered to your service in a given month, and your hero metric might be the percentage of those registered users who remain active (retain) into their 6th month post registration.
You could also look at a cohort of “All customers who made their first purchase in a given month, and discovered you via a Facebook Advertising campaign”. With this group, you can look at the cumulative profit generated by the cohort, and compare it with how much money you spent on the advertising campaign(s). That way you’d be tracking your Customer Lifetime Value (LTV) over your Customer Acquisition Cost (CAC), a measure on your return on advertising spend.
Notable here is that as most metrics are averages or shares, the denominator is more or less always the size of the cohort. The denominator is always constant for a cohort, so as the cohort of users move through time, the cohort size is constant. This may come as quite intuitive, but important to keep in mind when reading the next section about Group Retention Criteria.
In my experience, monthly cohorts are the most commonly used, and they are often split by country. E.g. there’s a “US January 2022” cohort and so on. You might also want to split your cohorts based on the product of the first purchase of a customer, the marketing campaign you’ve attributed them to, or whether they use an Android or iOS device.
As cohorts are used to measure things like Lifetime Value, differentiating it per market and campaign can be really powerful when prioritising product improvements or marketing spend.
Cohort Maturity
An interesting aspect of cohort analysis is that of maturity. I’ll just say a few things about it here. What I mean by maturity, btw, is that a cohort is ready to be analysed. For instance, if you’re looking at the customer retention month 2 of the January cohort, the users who came in on the 1st of January can be “measured” already after the 1st of March. However, the users who converted on Jan 31st aren’t ready until end of March. Hence you’ll likely want to exclude month 2 from your analysis of this cohort until the end of March data is available.
An implication of this is also that a long cohort, say a Quarterly cohort, will take an entire quarter to mature. Showing the half-baked “last data point” for those may lead to confusing looking graphs — and you’d need to be careful to ensure that you calculate your metrics on a sub-set of the cohort size — which you usually don’t want to do.
A way to mitigate the quarter maturity, is to look at “rolling 3 month windows” instead of actual quarters. These will mature as new months mature, but will only align with actual calendar quarters a third of the time.
I’ve also seen similar thinking with monthly cohorts, where one has looked at rolling windows of days instead. These won’t align with calendar months, but may be equally valuable to your analysis. In particular, if you don’t sell a monthly product, but a product that lasts for 30 days regardless of when it’s purchased, looking at 30 day rolling cohorts may be a better fit.
When looking at user activity metrics these are often strongly correlated to weekly patterns, and a 28 day rolling window (4 weeks) is often a more fair cohort definition if you want to compare trends over time without having varying month lengths and “distribution of weekdays” skew results.
Anyway, maturity is a good reason to use “reasonably long” cohort time frames, especially when your cohorts are defined by users being exposed to some A/B test or similar. You want to get fairly quick reads of long-term results, but also need the trade-off of certain cohort size for conclusions to be “statistically significant”.
Group Retention Criteria
This is an often overlooked part of defining cohorts, but it is absolutely critical to your results’ usefulness. The retention criteria specifies under what conditions a user is considered “active (or retained) in your cohort”. This is best explained with a few examples.
Let’s say you’re a subscription business, and have a “Conversion Cohort” from January 2022, in other words, it contains all customers who converted into paying users in that month.
Now you want to follow how well these customers retain their subscription over time. In the first data-point, 0 months after first purchase, 100% of customers are retained. After 1 month, something like 40 % may have churned, and hence only 60% retain. After 2 months you might have 55% retain etc.
So far so good. Now, what happens if a customer churn month 1, but then comes back again month 2?
This is when you’ll apply a group retention criterion. There’s a bunch of ways to do this, and I’ll introduce the two extremes and a balanced version below — followed by a discussion of the three.
Gross Retention
Given the example above, Gross Retention cohorts present a “really pessimistic” view of your customers retention. Under this definition, the customer that churn month 1 and come back month 2 is no longer counted as retained. Hence when you measure retention in a Gross cohort, you’re really measuring “The percentage of customers who are still paying customers, and who have consistently been paying customers since the start of the cohort”. (You can of course replace “paying” with “active” or “engaged” depending on what behaviour you track).
Under this definition, your retention rate can only go down as time passes, customers who churn are not allowed back in.
A consequence of using Gross Retention cohorts is that you want to align the definition of your cohorts.
Since a churned customer is not let back in, it usually makes sense to allow them to enter into another cohort when they come back. In short, the customer from before that converted in January then churn in February won’t count as retained month 3 in the January cohort. Instead, when they come back, they’ll be included as “retained month 0” in the March cohort.
Given this, the same customer can be a part of multiple cohorts. In other words, they are a part of the “denominator” (cohort size) in several cohorts at once, however they only count for “the numerator” (retained customer) in one cohort at the time.
Net Retention
Under the net retention criterion, the customer that churns in February, and comes back in March, will be considered “retained” again.
Two side effects of this is
A customer will only be a member of one cohort, the one of their first conversion
Since churned customers are allowed back into the “numerator” of the metrics, retention metrics is not limited to go down over time.
What you’re really measuring in Net retention cohorts is “The share of customers who made their first ever purchase in a month, who are still paying customers at some point later in time, regardless of whether they churned in the period inbetween”.
Net retention is often the easiest cohort to calculate in modern data tooling, it comes with the benefit that the total sum of your cohort sizes is also the number of customers’ you’ve ever had.
Another advantage is that it aligns with “first time customers”, which is often used when calculating Customer Acquisition Cost (CAC), in other words, we only attribute a cost to a customer the first time we acquire them. Hence, the cumulative revenue in a net retention cohort can often be directly compared to CAC.
A big disadvantage however is that the cohort is often used to measure some behaviour that was influenced by the customer experience when the customer entered the cohort. For instance, the onboarding and potential trial experience of new customers. If a customer churns immediately, and then comes back two years later, it’s a stretch to say that the “come-back” was a result of the onboarding of the first conversion. It’s more likely that the “come-back” was driven by a new push to acquire customers, and hence the customer should likely be counted for the CAC of the month two years later.
To handle this, you can consider using the Gross retention criterion, and ensure that your CAC is based not on first-time customers, but any conversion and re-conversion. That way, your customer intake will “go up” (as you can acquire the same physical person as customer multiple times), driving down your CAC, but also driving down your Lifetime Value (LTV) (as, in order to align it with CAC, you should now base it on the Gross retention criterion, where customer churn is permanent, and hence cumulative value per-each-converison, is lower)
Net Nd Retention
Net Nd Retention, where the N is intended to be replaced by a number, is a version of cohorts that mixes the ideas of Net and Gross.
A customer that churns out of the cohort is allowed to be “stay” in the numerator of cohort metrics if they come back within N days. You can think of N as a grace period, often the period between “soft churn” and “hard churn”. At Google we used to call this “churn” and “lapse”, where a churned customer who didn’t come back within a month was considered lapsed.
A common example of this is when credit cards expire and/or decline. Your customer may churn for a few days while fixing their card details, but then come back again. In the Net retention cohort, this is fine, the customer is allowed back in — but in Gross, the customer will be “churned forever”.
The idea here is that churned users may come back by themselves, and churn might have been involuntary. It wouldn’t make sense to permanently churn these customers from the cohort, in particular if you were to then consider their re-subscription another conversion that counts towards the CAC.
On the other hand, if a customer lapse, e.g. stay churned for “longer than N days”, it’s quite likely that them coming back again is the result of some marketing initiative, and it makes sense to include them in another cohort and count them towards CAC again.
Note that the N days will have an impact on cohort maturity, as we don’t know for users who churned in the last N days whether they’ll come back. For this reason, you might want to split your retention metrics into a third bucket leading to the last month having a retention of e.g. “60% retained, 10% pending, 30% lapsed”.
Finding N
N is the time between churn and lapse, the time it takes an involuntary churner to come back. You might have an obvious N if you look at a graph with “time since churn” on the x-axis, and “% resubscribed” on y. If you see a clear inflection point, or elbow, or similar in the graph, that might be a good point in time.
From my experience, N will fall somewhere between 14 and 30 days. A longer window makes it hard to report data as some customers are “still pending possible re-subscription”, a shorter window won’t give people enough time to realise they’ve churned, identify why and fix the problem.
My favourite cohort definition and retention criterion is the “Monthly Net 30 day Conversion cohort”. In this cohort, acquired customers stay in the numerator of the cohort metrics from conversion up until they churn for longer than 30 days (lapse). When a lapsed user converts again, they’ll be counted towards the numerator in a new cohort.
Cohort Cadence
So now you have your cohort definition, both for the grouping and retention criterion. Now at what points in time do you check in with the customers?
If your cohort is daily, for instance a “Daily Gross Conversion Cohort”, you'll group all customers that converted on the same date into a cohort. You can then follow up on a daily basis, so essentially you track “how many customers are still retained 1 day later, 2 days later, …, 90 days later” etc.
If you do this for a year of cohorts (365 days) split into say 20 countries where you track each country cohort, you’ll end up with 7,300 cohorts. Keeping data on each cohort day 1, 2, 3 … will quickly blow up your data size. Tracking these daily cohorts “forever” in a company that’s been around for about 5 years, could mean that the average cohort is about 2 years old. That’d bean you have 5.3 million records of data. Unless you have a massive business going, each cohort will be tiny, and you can’t rely on the property that users in groups behave consistently, so in this case, your metrics will likely fluctuate all over the place, and insights will be hard to draw.
Usually, you want high resolution in the beginning, for instance you might track users registering to your product closely for their first three weeks. Knowing that a customer were still active 5 days after registering etc. is quite an important insight.
After a few weeks however, it’s not that interesting whether a user is active on day 37 or 38. At that point, you’re probably more interested in knowing whether they’re a weekly active user at any point in their 5th week after registration.
If that’s true, you might want a “Weekly Net Registration Cohort” (as registration is usually only done once per user, Net cohorts make the most sense IMO). Having cohort metrics on a weekly resolution might make sense for the first few months, but then the exact week is no longer interesting either. That’s when the work-horse monthly cohorts come in. In my experience, monthly cohorts that you track per month after start are the most balanced.
Hence you’d track “Monthly net registration Cohorts per month after registration”, sometimes called “month-month” registration cohorts. In fact, monthly cohorts are so commonly used that I’ll spend the next section talking a bit more about how you can define months to reach different results.
Defining month deltas
When working with “month-month” cohorts, you group users who shared some trait in one month, and then track their behaviour for months following that. A naive way of doing this would be to say “a month is approximately 30 days, so let’s follow up on 30 day intervals”. This particular way of doing things is quite crude, and I won’t delve further into it here.
There are two “less naive” ways of doing this, and I’ll again use the customer conversion cohort example from before.
Even Month
When following a cohort per even month, you’d give each customer a “conversion date”, and then check in with them on that date for subsequent months. For instance, a customer who converted on January 12th, will be checked in with again on the February 12th. If they’re still a customer, their 1 month retention will be 1, else 0.
When you group this data together for the entire cohort, you’ll likely want to get rid of the “12th” part of the dates, and that works great! There’s only one thing to keep in mind. If you look at the January cohort, and their “month 1 retention”, that retention is measured across a whole month, so you can’t use this data to say anything about things like “How many customers did we have on the last of February”.
The even month should be the way you measure retention rates and lifetime values, and set KPIs around those. It’s a fair way of ensuring that every customer in a cohort has the same “distance of time” between month 0 and 1, and in particular on monthly auto-renewing products, this makes the retention rates look intuitive and easy to reason about.
The only tricky part of calculating this is for customers who convert on e.g. the 31st of January. Their 1 month retention should be measured on the 31st of February, which for obvious reasons is hard to measure. Hence for these customers you have to measure on Feb 28th (or 29th). A common mistake people make for these customers is to measure their month 0 on Jan 31, month 1 on Feb 28th and month 2 on Mar 28th. That’s not how it should be done, but rather in March, you’d be back at the 31st.
Calendar Month
When checking in on your cohort on even calendar months, what you do is to say that the starting point of the cohort metrics is at midnight between two calendar months. In other words, if a customer make their first purchase on Jan 12th, we check their month 0 retention at midnight between Jan 31st and Feb 1st. In most datasets I’ve worked, this is equivalent to the data-point with a date-dimension of Jan 31st.
The calendar month cohorts are usually inconvenient to use. For instance, the customer from Jan 12th might have already churned by the 31st, and hence the retention rate in the cohort won’t start at 100%. You’ll also treat customers who converted late in the month differently from those who convert early — as the early ones get more time to churn before they’re measured.
Because of this, you should not use calendar month check-ins with cohorts that you use to calculate things like lifetime values or “Month N retention rates”.
So why even bother with calendar month cohorts? At Spotify we called these “forecast cohorts”, because they possess one very practical property. You can answer the question “How many customers, who converted in December 2013, are active subscribers on November 30th of 2014 (midnight between Nov and Dec)”. This property means that you can sum up the retained subscribers for all your cohorts, and know how many subscribers you had on a given date.
So if you take all historical cohorts, filter them on whatever time-delta takes them to November 2014, you can see the number of subscribers there. Why is this good? Because cohorts tend to behave predictably, and are hence fairly easy to extrapolate. So with some fairly simple extrapolation techniques, you can get a really good forecast of how your customer base will develop into the future.
Final Summary & Thoughts
Measuring cohorts is a powerful way of understanding user journeys. There are many ways to define cohorts, but they usually are a combination of:
Users who share some trait (e.g. all Android users)
Who did something (e.g. converted to a paid plan)
In some time-frame (e.g. in the same calendar month)
The way you define the time-frame should also align with your group retention criteria, and that should align with your metrics and benchmarks.
The most versatile criteria I’ve used is the “Net 30 day retention” which means that users “stay in the numerator” of the cohort, as long as they don’t lapse (stay churned for >30 days). If they lapse and come back, they become a part of a new cohort. Calculating user lifetime-value (cumulative profit over cohort size, possibly as a net present value) for these users is a pretty good metric, and if you compare it to your customer acquisition cost, you should use the same definition of “acquisition”, e.g. the customer is acquired either the first time they convert, or if they re-convert after being lapsed.
Checking in on monthly cohorts on even intervals of months after conversion will give you a really good understanding of customer retention. Churn usually happens at exactly that point — on the billing date of the customer and even number of “monthly products” after their first purchase.
If you want your cohort metrics to align with your monthly reported numbers, you can check in on calendar month intervals instead, meaning that you disregard what date the customer actually converted. This cohort definition is nice to use to align those metrics, and in particular it is nice if you forecast the cohorts future performance — that you can translate that cohort forecast into your regular monthly metrics.