Cohort analysis - 4 ways to analyze your product retention rate

What's retention rate and what are the best ways to measure it? 
What’s a negative churn, and what can you learn from it about the performance of your cohorts and the sustainability of your business? ?How does it all relate to unit economics? Answers below: 

We all know the importance of retention for the long-term success of our products.
Retention is the key to creating a sustainable business. It shows the long-term engagement of your most loyal users - and that’s a strong sign of a product/market fit. It influences how much revenue will each cohort produce over time and the lifetime value of each user.

Higher retention = more recurring paying users.

Retention is the key for creating a sustainable business. 

And while tweaking your onboarding process and funnels may drive immediate improvements in conversion rates (and produce instant gratification) - retention is a long term process, it often requires some heavy lifting, deeper analysis, but usually makes a bigger long term business impact. 

In this post, I would like to share 4 useful variations of retention reports that can become very handy when analyzing the performance of a product across time.

So let's begin:

4 practical methods to analyze your product's retention rate and learn about the sustainability of your business.

1. The basics: 

Common retention reports use a certain time unit (i.e. days, weeks, months) and base the cohorts on the installation/activation date (if it’s a mobile app) or the registration date (if it’s a web-based product). They measure the number of users that originated on a certain cohort and keep using the product across time.

The report shows how many users were "born"(started) each day/week/month and what happened to them across the next time units (how many churned and how many continued to use the product).

The average mobile app loses 58% of the users in the first 30 days (that's 42% retention rate) and loses 75% of the users in the first 90 days (that's 25% retention rate). These vanilla numbers are based on user app visits. Here's the report: what is good retention.

Although counting repeated visits can help you understand the engagement level of the users across time (just like DAU/MAU etc.) - for many products and businesses this is not enough and the retention needs to be based on a meaningful activity that is expected to be repeated by the users (for instance: making a purchase, assigning a task or completing a session).

Here’s a made-up example of a retention report that uses the definitions of a marketplace startup that measures monthly transactions.

This retention report analyzes how many paying users (users who ordered services through the app) in each cohort, continue to use the app and order more services in the following months.
The numbers are made up of course, but I tried to place them in a way that will tell a story of a real business:

The chart above presents the retention based on users who completed their bookings every month within a period of 16 months.

• For example, in November 2020, the product had 146 new paying users (month 0), 102 of them ordered in the next month (month 1), 88 continued to use the app in the next month (month 2), and so on. 
• Out of the first 100 paying users of the first cohort (January 2020), 24 users made an order in month 15. This means that the retention for this cohort, after 15 months was 24%.

We use months because a typical paying customer in this specific business places an order once a month. 

The cohorts are based on the registration dates, but if your product suffers from “late conversion” (where the first significant action happens long after the download/sign-up date) you may want to set your cohorts based on the first activity date and not the sign-up date. 

Every entrepreneur (or a product leader) usually knows the business well enough to decide on those specifics.

I’m a visual kind of guy, so I usually prefer charts over tables:





In the chart above, you can see the retention based on the number of paying users in each cohort across the months.




Now, personally speaking, I’m able to digest and analyze all of the 15 cohorts much more in a just few seconds... it’s my superpower (not)… but if you don’t have such superpowers yourself (I assume you don't) - it might be a good idea to select a few cohorts and make it easier on the eyes.

Here’s a cleaned-up version:

Of course, every cohort has its’ own story:

• Some cohorts start with a higher number of users due to certain marketing activities or product optimization
• Some cohorts are influenced by holidays or seasonality
• Some cohorts are just better than others because of external market forces

Because every cohort has a different starting point - using actual numbers doesn’t make it easy to identify trends and patterns, so the next thing to do is to convert the numbers into percentages:

Once the values are normalized into percentages - it’s easier to identify trends, compare different cohorts, calculate averages, and more.

Here's the average retention rate:




Things to look for:

• If your number of monthly paying users (by cohort) keeps dropping - you have a leakage - that’s a bad sign (although in some businesses it might still be OK! It all depends on how hard it is to bring those users onboard - but more on that later) 
• If the number of monthly paying users (by cohort) stabilizes at a very low point - there’s a need to dig further into the numbers: maybe there isn’t enough product/market fit, maybe the original cohort is not focused enough and you're bringing the wrong users in, maybe the product has some bugs that cause users to drop. 
• If the number of monthly paying users (by cohort) stabilizes at a high point - you rock! But don’t start celebrating yet - there are some more numbers to check. 

Another good exercise is to compare time ranges and see if the retention is improving across time.

Here’s a comparison between the first and second half of the year:

As you can see from the above, it seems like the second half of 2020 is performing much better than the first half. It might be the result of some product improvements or marketing activities - but even without exploring why it happened - the bottom line is that the product is experiencing a positive trend.

2. Switching from users to revenue

There are a few problems with measuring retention based on the number of paying (or active) users each month:

1. It doesn't measure the level of engagement of the cohorts (number of sessions, deals, volumes, revenue)
2. It doesn’t measure the frequency of use (some will use the app once a month, some will use it every day - that's why you also need to measure DAU/MAU) 
3. It’s impossible to know if the same users are using the product every month or sporadically (i.e. once every 3-4 months)

In order to further explore the behavior of the cohorts - I typically measure 2 additional retention reports:

1. Retention rate based meaningful actions (number of sessions, orders, posts, games, etc.) every cohort produces in each month  
2. Retention rate based on the revenue each cohort produces in each month

I always used the following retention trio: users, orders, and revenue. The 3 of them together give a good picture of the cohorts' behavior, the frequency of use, the engagement level, etc. 

For the sake of keeping this post simple - let’s dive straight into the revenue option and see what can be learned from it.

Here’s the full retention table measuring the revenue each cohort produces across months:

And here’s the average retention based on those numbers:

As you can see from the above, in month 15 (that’s 15 months since sign-up) - the cohort produces 38% of the revenue it did in the first month. Earlier we saw that when we measured the number of users - we found that 24% of them used the product in month 15. Here, the number is quite higher: 38%.

What does it mean? 

It means that 24% of the cohort’s original users (paying users to be accurate) produces 38% of its’ first-month revenue. This is a good indication that the retained users are becoming more engaged as time passes: they are either making more transactions or may their transactions are getting bigger - both are a good thing. 

Let’s repeat the time range comparison to see if we can identify a positive trend:




Yep, as you can see from the above, as the months pass, the revenue is actually going up.

Beautiful trend.

Of course, I know that already, given that I fabricated those numbers... 🤟

It means that although many users churn, the remaining users are spending more money than before and the ARPPU (Average revenue per paying user) is increasing. It's an important sign of engagement: the retained users are becoming more engaged over time.

The Smiley retention curve

Shameless bragging anecdote ahead:
Analyzing our real cohorts at Missbeez, we often found an interesting trend where the cohort’s monthly revenue decreased (in the first few months) and then slowly went up until it surpassed the original revenue it created in month 0. We call it the smiley retention rate, which is professionally called a Net Negative Churn.

It’s quite unusual, but it can happen:




OK, enough bragging, let’s move on…

Check out the best of the mobile spoon or subscribe to my occasional newsletter and become 23% more awesome than average!

3. From revenue to cohort-based revenue

Here’s another cool way to look at your cohorts and understand the impact of retention on your product’s performance (did I just use the word cool for describing a report?).
This cool layered graph shows the monthly revenue broken down by the different cohorts.

It’s one of the best ways to demonstrate how retention supports the growth and the long term sustainability of the business:

Every layer in the above graph represents a cohort. It’s the same data I used before but the representation makes it easy to understand: the thickness of each layer over time represents the retention rate of the cohort (measured in revenue).

Higher retention leads to thicker layers - and that leads to a bulk of layers supporting your monthly revenue stream, just like MRR works in subscription-based products.

In the made-up example above, the revenue grows by around 10%-12% each month (MoM) although the number of new users each month grows by just 3%-4%. The rest of the growth is supported by the revenue generated by the repeat customers (thanks to a fairly high retention rate).

As compassion - let’s see what might happen if the retention rate is very low, and those layers keep vanishing across the months:




As you can see from the above example: many layers last for only 5-6 months and then they disappear.

In order to maintain some sort of growth, this example assumes that the company spends much more money on binging new users onboard than the previous one, they don’t last for long, so the budget constantly needs to grow while repeat revenue stays small. The result is that while the revenue in the previous example reached $60K without spending a lot on user acquisition - this example reaches $30K while spending much more on user acquisition.

One graph - so many beautiful insights.

Many startups increase user acquisition budgets to demonstrate growth: as a result, the revenue graph goes up, but experienced VCs can immediately identify the problems by analyzing retention rates and using this cohort-based report or by measuring the unit economics (CAC vs. LTV) so I don't recommend on using this "trick". 


In order to know where’s the exact sweet spot, and when does each cohort become profitable - we need to move on to the next (and last) report for today: the cohort analysis report (drums drums drums). Details below:

4. The cohort analysis 

So you want to know if a certain cohort is profitable or not.
You want to know how long does it take for each cohort to pay back the CAC (cost of acquisition) and understand your product's unit economics.  

In order to do so, we will have to tweak things up in our revenue-based retention report, add the CAC, and basically compare it with the LTV of each cohort.

Let’s dive in:

There are a few steps required in order to create this report:

• First, insert a column (right before the first-month column) named CAC and enter the total cost of acquisition for each cohort. 
• Then, replace the values in each cell from gross-revenue (that’s what we used in the previous retention table) to net-revenue. In my made-up example below I multiply all the numbers with 0.3 because I assume that 30% of each transaction is kept as the commission for the deal.  

So the modified table should look something like this:

And now for the last step:

• Duplicate your table and change the cells values from simple values to running totals, so instead of showing: {-5,000 | 1,500 | 825 | 600} - that first row should now present: {-5000 | -3500 | -2675 | -1625}. 

Here’s how it should look like:

The idea is simple: each row represents a cohort: it starts with the CAC (how much money did we spend to “buy” the cohort). Then, the cohort starts to generate revenue, and in each month that passes the cohort’s debt is decreasing (thanks to the net-revenue it generates) until it pays back the CAC, breaks even, and becomes profitable. Now, keep in mind this doesn't mean the company is profitable (most startups are not), but it does mean the unit economics is positive, and a healthy business can be built with it.

Using conditional formatting makes it easy to spot those break-even points in time.

If that’s not a beautiful sight I don’t know what a beautiful sight is. 

As you can see, with the first few cohorts it took 7 months to break-even. In January 2021 (12 months later) it took only 3 months to break-even.

This makes sense of course, as we saw earlier that the later cohorts have better retention-rates and are generating more revenue, but the beauty with this full cohort analysis report is that it visualizes the value of each cohort in a very powerful way.

At the end of the day, we want our users to remain engaged across time. The retention rate is crucial, but it’s more important that the retained users will continue to create a revenue stream. That stream is the basis of their LTV (lifetime value) and it needs to be compared to the cost we paid to bring them onboard.

Let’s take a look at a few cohorts presented in a graph:

The break-even moment happens when a cohort breaks-through that sexy zero-point and rises up. Any net revenue that follows that point is a profit generated by this cohort. 

It’s like a phoenix arising from the ashes!

(forgive me… I’m getting over-excited around numbers) 

 

While reaching profitability in each cohort is something investors will definitely look for - most VCs will also check how fast it happens. It’s much riskier to invest in a business where cohorts become profitable after 6-12 months (too many things can happen in such a long period, the startup might even die before those cohorts reach profitability and any hiccup might lead to additional losses).

If your business doesn’t sell long term technological dreams and needs to demonstrate positive unit economics - that model needs to not only be well structured and stable - it also needs to show how it achieves cohorts profitability quite fast.


What happens if the phoenix never arises?





Well, that sucks. 

As you can see from the above - Cohort A starts with a fairly low CAC but the retention seems so low that the net revenue is barely growing. As a result, the poor phoenix never rises from the ashes…

Cohort B, on the other hand, does show some decent net revenue growth (at least in the first few months) but the CAC is so high, it's going to take forever to reach that desirable break-even point, if at all…

When reaching such situations, there are a few things to do:

1. Make sure no one, never ever sees this horrible report.
2. Focus on the CAC (more relevant for cohort B) - maybe your user acquisition strategy is wrong, maybe the wrong users are brought into the system. Will you ever be able to bring those users in half of the price? If the answer is yes then future cohorts might become profitable. 
3. Explore the reasons for the poor retention rate (more relevant for cohort A) - are you lacking a product/market fit? Is there a pricing problem? Product limitations? Without improving your retention rates, it will be very hard to build a sustainable business with these numbers. 

Summary: 

This post is about retention. As you can see, there are many ways to explore the performance of the cohorts. Almost all of them are directly driven by retention rates.

High retention rates can be translated into a solid, repeated monthly revenue, that supports both the stability of the business and its’ growth, and that’s even before improving other aspects of the product such as conversion rates, functionality, brand awareness, network effect, etc.

If you found this post interesting I would highly recommend that you read my thoughts on early adopters here: the good, the bad, and the ugly truth about early adopters. The post talks about the risks of focusing too much on the early adopters when polishing the business model and unit economics.

Other posts you may like:

• How to prepare your data for user segmentation - tips for early-stage startups
• 10 shades of MVP (or: how to develop a product without developing a product...)
• 11 lessons learned while trying to become a data-driven company
• How to maintain your product momentum when you’re out of development budget

Check out the best of the mobile spoon or subscribe to my occasional newsletter and become 23% more awesome than average! (Don't worry about spam, I'm too busy for that...)