[notes] Lean Analytics
Lean Analytics: Use Data to Build a Better Startup Faster (2013) by Alistair Croll and Benjamin Yoskovitz
Buy from Amazon
NOTE: Bewarned, these notes are un-edited, un-revised, and un-styled. I plan on cleaning them up eventually, but until then, I apologize.
This book gets right down to the nitty gritty. So I don’t think I’m going to take notes on everything. For the stuff that relates to specific business models, I’ll just note that it exists, BUT IT’S ALL REALLY GOOD STUFF so I should remember that it exists for when I need it.
Part 1: Stop Lying to Yourself
1. We’re All Liars
“In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out.”
2. How to Keep Score
What makes a good metric?
- a good metric is comparative
- a good metric is understandable
- a good metric is a ratio or a rate
- ratios are easier to act on
- ratios are inherently comparative
- ratios are also good for comparing factors that are somehow opposed, or for which there’s an inherent tension
- a good metric changes the way you behave
- “draw a line in the sand”
Qualitative versus quantitative metrics
Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less insight.
“Quantitative data abhors emotion; qualitative data marinates in it.”
Vanity versus actionable metrics
Vanity metrics might make you feel good, but they don’t change how you act. Actionable metrics change your behavior by helping you pick a course of action.
Exploratory versus reporting metrics
Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations.
Leading versus lagging metrics
Leading metrics give you a predictive understanding of the future; lagging metrics explain the past. Leading metrics are better because you still have time to act on them—the horse hasn’t left the barn yet.
“Lagging and leading metrics can both be actionable, but leading indicators show you what will happen, reducing your cycle time and making you leaner.”
Correlated versus causal metrics
If two metrics change together, they’re correlated, but if one metric causes another metric to change, they’re causal.
“If you find a causal relationship between something you want and something you can control, then you can change the future.”
Key performance indicators (KPIs) => specific metrics that drive the business
Segments, Cohorts, A/B testing, and Multivariate Analysis
- A segment is simply a group that shares some common characteristic.
- Each group of users is a cohort—participants in an experiment across their lifecycle. You can compare cohorts against one another to see if, on the whole, key metrics are getting better over time.
- Cohort experiments that compare groups like the one in Table 2-2 are called longitudinal studies, since the data is collected along the natural lifespan of a customer group. By contrast, studies in which different groups of test subjects are given different experiences at the same time are called cross-sectional studies.
“Much of Lean Analytics is about finding a meaningful metric, then running experiments to improve it until that metric is good enough for you to move to the next problem or the next stage of your business.”
3. Deciding What to Do with Your Life
Bud’s diagram shows three overlapping rings: what you like to do, what you’re good at, and what you can be paid to do. For each intersection between rings, he suggests a course of action:
• If you want to do something and are good at it, but can’t be paid to do it, learn to monetize.
• If you’re good at something and can be paid to do it, but don’t like doing it, learn to say no.
• If you like to do something and can be paid to do it, but aren’t very good at it, learn to do it well.
4. Data-Driven Verus Data-Informed
“Data is a powerful thing. It can be addictive, making you overanalyze everything. But much of what we actually do is unconscious, based on past experience and pragmatism. And with good reason: relying on wisdom and experience, rather than rigid analysis, helps us get through our day. After all, you don’t run A/B testing before deciding what pants to put on in the morning; if you did, you’d never get out the door.”
“Humans do inspiration; machines do validation.”
“Machine-only optimization suffers from similar limitations as evolution. If you’re optimizing for local maxima, you might be missing a bigger, more important opportunity. It’s your job to be the intelligent designer to data’s evolution.”
10 common pitfalls when analyzing data:
- Assuming the data is clean
- Not normalizing
- Excluding outliers
- Including outliers
- Ignoring seasonality
- Ignoring size when reporting growth
- Data vomit
- Metrics that cry wolf
- Not mashing up your data with data from other sources
- Not looking at the bigger picture
Part 2: Finding the Right Metric For Right Now
5. Analytics Frameworks
Dave McClure’s Pirate Metrics
“McClure categorizes the metrics a startup needs to watch into acquisition, activation, retention, revenue, and referral—AARRR.”
Eric Ries’s Engines of growth
- Sticky Engine
- Virality Engine
- Paid Engine
Ash Maurya’s Lean Canvas
Sean Ellis’s Startup growth Pyramid
- Product/Market Fit (decide what you sell to whom, and prove it)
- Stack the Odds (find a defensible unfair advantage)
- Scale Growth (new markets, products, and channels)
The Lean Analytics Stages and Gates
“We believe most startups go through these stages [Empathy, Stickiness, Virality, Revenue, and Scale], and in order to move from one to the next they need to achieve certain goals with respect to the metrics they’re tracking.”
6. The Discipline of One Metric that Matters
- OMTM (One Metric That Matters) => at any given time, the one metric you should care about above all else
- It answers the most important question you have.
- It forces you to draw a line in the sand
- It focuses the entire company
- It inspires a culture of experimentation
“Capture everything, but focus on what’s important.”
“Optimizing your OMTM not only squeezes that metric so you get the most out of it, but it also reveals the next place you need to focus your efforts, which often happens at an inflection point for your business.”
7. What Business Are You In?
“Sergio Zyman, Coca-Cola’s CMO, said marketing is about selling more stuff to more people more often for more money more efficiently.”
- More stuff means adding products or services
- More people means adding users, ideally through virality or word of mouth, but also through paid advertising
- More often means stickiness (so people come back), reduced churn (so they don’t leave), and repeated use (so they use it more frequently)
- More money means upselling and maximizing the price users will pay
- More efficiently means reducing the cost of delivering and supporting your service
“Not all customers are good. Don’t fall victim to customer counting. Instead, optimize for good customers and segment your activities based on the kinds of customer those activities attract.”
The Business Model Flipbook
- acquisition channel => how people find out about you
- selling tactic => how you convince visitors to become users or users to become customers
- revenue source => simply how you make money
- product type => what value your business offers in return for the revenue
- delivery model => how you get your product to the customer
8. Model One: E-commerce
“It’s vital to know if you’re focused on loyalty or acquisition. This drives your whole marketing strategy and many of the features you build.”
“While conversion rates, repeat purchases, and transaction sizes are important, the ultimate metric is the product of the three of them: revenue per customer.”
9. Model Two: Software as a Service (SaaS)
“In SaaS, churn is everything. If you can build a group of loyal users faster than they erode, you’ll thrive.”
“Many people equate SaaS models with subscription, but you can monetize on-demand software in many other ways, sometimes to great effect.”
10. Model Three: Free Mobile App
“Most of the money comes from a small number of users; these should be segmented and analyzed as a distinct group.”
11. Model Four: Media Site
“It’s hard to strike a balance between having good content and enough ads to pay the bills.”
12. Model Five: User-Generated Content
“Visitor engagement is everything in UGC. You track visitors’ involvement in an ‘engagement funnel.’”
13. Model Six: Two-Sided Marketplaces
“Early on, the big challenge is solving the “chicken and egg” problem of finding enough buyers and sellers. It’s usually good to focus on the people who have money to spend first.”
“Since sellers are inventory, you need to track the growth of that inventory and how well it fits what buyers are looking for.”
14. What Stage Are You At?
- Empathy => you need to get inside your target market’s head and be sure you’re solving a problem people care about in a way someone will pay for
- Stickiness => you need to find out if you can build a solution to the problem you’ve discovered
- Virality => you need to get your product in the hands of visitors who are motivated to try you, because they have an implied endorsement from an existing user
- Revenue => you need to focus on maximizing and optimizing revenue
- Scale => you need to acquire more customers from new verticals and geographies
15. Stage One: Empathy
“Right now, your job isn’t to prove you’re smart, or that you’ve found a solution. Your job is to get inside someone else’s head. That means discovering and validating a problem and then finding out
whether your proposed solution to that problem is likely to work.”
Finding a Problem to Fix (or, How to Validate a Problem)
- The problem is painful enough
- Enough people care
- “Solving a problem for one person is called consulting.”
- “Marketers want audiences that are homogeneous within and heterogeneous between.”
- homogeneous within => members of the segment have things in common to which you can appeal
- heterogeneous between => you can segment and target each market segment in a focused manner with a tailored message
- They’re already trying to solve it
- “The current solution, whatever it is, will be your biggest competitor at first, because it’s the path of least resistance for people.”
- “Note that in some cases, your market won’t know it has a problem… In this case, rather than just testing for a problem people know they have, you’re also interested in what it takes to make them aware of the problem.”
“Initially you’ll use qualitative metrics to measure whether or not the problem you’ve identified is worth pursuing. You start this process by conducting problem interviews with prospective customers.”
- Briefly set the stage for how the interview works
- Test the customer segment by collecting demographics
- Set the problem context by telling a story
- Test the problem by asking the subject to rank them in order of importance
- Test the solution by going through each problem and asking the subject how they solve it today
- Ask for the subject to agree to do a solution interview with you when you’re ready with something to show, or to refer other people for you to interview
How to Avoid Leading the Witness
- Don’t Tip Your Hand
- Biased wording
- “You can get around this by asking people the opposite of what you’re hoping they’ll say.”
- Preconceptions => if the subject knows things about you, he’ll likely go along with them.
- Make the Questions Real
- “The more concrete you can make the question, the more real the answer.”
- “One other trick to overcome a subject’s desire to please an interviewer is to ask her how her friends would act.”
- Keep Digging
- “A great trick for customer development interviews is to ask ‘why?’ three times.”
- Look for Other Clues
Convergent and Divergent Problem Interviews
“Problem validation can happen in two distinct stages.”
- convergent approach => directed, focused, and intended to quantify the urgency and prevalence of the problems so you can compare the many issues you’ve identified
- divergent approach => much more speculative, intended to broaden your search for something useful you might go build
“A convergent problem interview gives you a clear course of action at the risk of focusing too narrowly on the problems that you think matter, rather than freeing interviewees to identify other problems that may be more important to them.”
How Do I Know If the Problem Is Really Painful Enough?
- Did the interviewee successfully rank the problems you presented?
- Is the interviewee actively trying to solve the problems, or has he done so in the past?
- Was the interviewee engaged and focused throughout the interview?
- Did the interviewee agree to a follow-up meeting/interview?
- Did the interviewee offer to refer others to you for interviews?
- Did the interviewee offer to pay you immediately for the solution?
Getting Answers at Scale
- expand the scope of your efforts and move into doing some quantitative analysis
- forces you to formalize your discussions, moving from subjective to objective
- tests whether you can command the attention—at scale—that you’ll need to thrive
- gives you quantitative information you can analyze and segment, which can reveal patterns you won’t get from individual groups
- respondents may become your beta users and the base of your community
“Don’t just ask questions. Know how the answers to the questions will change your behavior.”
Build It Before you Build It (or, How to Validate the Solution)
“Once again, this starts with interviewing customers (what Lean Startup describes as solution interviews) to get the qualitative feedback and confidence necessary to build a minimum viable product.”
“This is the reverse Field of Dreams moment: if they come, you will build it.”
Deciding What Goes into the MVP
“It’s important to contrast an MVP with a smoke-test approach… With a smoke-test page, you’re testing the risk that the message isn’t compelling enough to get signups. With the MVP, you’re testing the risk that the product won’t solve a need that people want solved in a way that will make them permanently change their behavior.”
Measuring the MVP
“The key is to identify the riskiest parts of your business and de-risk them through a constant cycle of testing and learning. Metrics is how you measure and learn whether the risk has been overcome.”
16. Stage Two: Stickiness
“The focus now is squarely on retention and engagement.”
- Are people using the product as you expected?
- Are people getting enough value out of it?
“Your goal is to prove that you’ve solved a problem in a way that keeps people coming back.”
Iterating the MVP
“Iterations are evolutionary; pivots are revolutionary.”
Seven Questions to Ask Yourself Before Building a Feature
- Why Will It Make Things Better?
- Can You Measure the Effect of the Feature?
- How Long Will the Feature Take to Build?
- Will the Feature Over-complicate Things?
- How Much Risk Is There in This New Feature?
- How Innovative Is the New Feature?
- What Do Users Say They Want?
How to Handle User Feedback
- Plan tests ahead of time, and know what you want to learn before you get started
- Group feedback from similar personas
- Review results quickly as you collect data
The Minimum Viable Vision
“Even though you’re building the minimal product, your vision should still be big enough to inspire customers, employees, and investors— and there has to be a credible way to get from the current proof to the future vision.”
“A minimum viable vision (MVV) is one that captivates. It scales. It has potential. It’s audacious and compelling.”
17. Stage Three: Virality
- virality => the spread of a message from existing, “infected” users to new users
The Three Ways Things Spread
- Inherent virality => happens naturally as users interact with your product
- Artificial virality => incentivized and less genuine, often built into a reward system
- Word-of-mouth virality => independent of product or service
Metrics for the Viral Phase
“If every user successfully invites more than one other user, your growth is almost assured.”
“In addition to viral coefficient, you care about viral cycle time.”
- growth hacking => data-driven guerrilla marketing
- the key to the growth hacking process is the early metric (leading indicator)
- “correlation predicts tomorrow”
- “causality hacks the future”
18. Stage Four: Revenue
“The goal in the Revenue stage is to turn your focus from proving your idea is right to proving you can make money in a scalable, consistent, self- sustaining way. Think of this as the piñata phase, where you beat on your business model in different ways until candy pours out.”
Metrics for the Revenue Stage
“You’re moving from proving you have the right product to proving you have a real business. As a result, your metrics shift from usage patterns to business ratios.”
Customer Lifetime Value > Customer Acquisition Cost
“The core equation for the Revenue stage is the money a customer brings in minus the cost of acquiring that customer. This is the return on acquisition investment that drives your growth.”
“Think of a business as a machine that converts money into greater sums of money. The ratio of money in to money out, as well as the maximum amount of money you can put in, dictates the value of the business.”
“If things aren’t working, it may be easier to pivot your initial product to a new market rather than starting from scratch.”
The Breakeven Lines in the Sand
- EBITDA Breakeven => earnings before income tax, depreciation, and amortization Hibernation Breakeven => If you reduced the company to its minimum could you survive?
“While your goal is to grow, you should also keep an eye on breakeven, because once you can pay your own bills you can survive indefinitely.”
19. Stage Five: Scale
“When you’re scaling, you know your product and your market. Your metrics are now focused on the health of your ecosystem, and your ability to enter new markets.”
The Hole in the Middle
- => the challenge facing firms that are too big to adopt a niche strategy efficiently, but too small to compete on cost or scale
- need to differentiate themselves to survive the midsize gap, and then achieve scale and efficiency
“You need to understand if you’re focused on efficiency or differentiation. Trying to do both as a way of scaling is difficult. If you’re efficiency-focused, you’re trying to reduce costs; if you’re differentiation-focused, you’re increasing margins.”
Metrics for the Scale Stage
“You’ll look at compensation, API traffic, channel relationships, and competitors at this stage—whereas before, these were distractions.”
The Three-Threes Model
“As you grow, you’ll need to have more than one metric at a time. Set up a hierarchy of metrics that keeps the strategy, the tactics, and the implementation aligned with a consistent set of goals.”
20. Model + Stage Drives the Metric You Track
Part 3: Lines In The Sand
21. Am I Good Enough?
Number of Engaged Visitors
- Fred Wilson’s 30/10/10 rule:
- 30% of registered users will a use a service at least once a month
- 10% of registered users will use the service every day
- the maximum number of concurrent users will be 10% of the number of daily users
Cost of Customer Acquisition
- “a good rule of thumb is that your acquisition cost should be less than a third of the total value a customer brings you over their lifetime”
22. E-commerce: Lines in the Sand
23. SaaS: Lines in the Sand
“Try to get down to 5% churn a month before looking at other things to optimize. If churn is higher than that, chances are you’re not sticky enough.”
24. Free Mobile App: Lines in the Sand
Average Revenue Per Paying user
- Whales: 10% of payers, ARPPU of $20
- Dolphins: 40% of payers, ARPPU of $5
- Minnows: 50% of payers, ARPPU of $1
25. Media Site: Lines in the Sand
26. User-Generated Content: Lines in the Sand
“You’ll have a very good indicator of stickiness when site visitors are spending 17 minutes a day on your site.”
27. Two-Sided Marketplaces: Lines in the Sand
28. What to Do When you Don’t Have a Baseline
“For many metrics, there’s simply no ‘normal.’”
“Nearly any optimization effort has diminishing returns.”
Part 4: Putting Lean Analytics to Work
29. Selling into Enterprise Markets
“In most cases, enterprise sales involve bigger-ticket items, sold to fewer customers.”
“Zero Overhead Principle: no feature may add training costs to the user.”
“…most B2B-focused startups consist of two people: a domain expert and a disruption expert.”
- domain expert knows the industry and the problem domain
- disruption expert knows the technology that will produce a change on which the startup can capitalize
30. Lean from Within: Intrapreneurs
Daniel C. McCallum
- enterprising general superintendent of the railroad era
- first management scientist, introducing controls, structure, and regulations in order to reduce risk and increase predictability at scale
- but intrapreneurs’ job is to take risks, and to uncover the non-obvious and the unpredictable
“In their book Confronting Reality (Crown Business), Larry Bossidy and Ram Charan list the six habits of highly unrealistic leaders: filtered information, selective hearing, wishful thinking, fear, emotional over- investment, and unrealistic expectations from capital markets.”
14 Rules for Lean Intrapreneurs
- “Build a small, agile team of high performers who aren’t risk-averse, and who lean toward action.”
- “Use tools that can handle rapid change.”
- “Agree on goals and success criteria before starting the project.”
- “Limit access to the team by outsiders as much as possible. Don’t poison the team with naysayers, and don’t leak half-finished ideas to the company before they’re properly tested.”
- “Reward performance based on results”
Changing—or Innovating to Resist Change?
“Critics of Microsoft’s reactions complain that the company isn’t changing; rather, it’s managing to stay the same and exert its dominance, avoiding or delaying market change. ‘I realized that Microsoft had not turned at all,’ said Dave Winer in 1999. ‘What’s actually been happening is that Microsoft is exerting tremendous energy to stay right where it is.’”
Skip the Business Case, Do the Analytics
“Traditional product managers build profit- and-loss analyses to try to justify their plans: they create a convincing business case, and once someone believes it, they get funding to proceed. But a Lean mindset reverses this: you sell the business model—not the plan—without a lot of prediction, and then rely heavily on analytics to decide whether to kill the product or double-down on it.”
“This analyze-after rather than predict-before model is possible because many of the costs of innovation can be pushed later in the product development cycle. Just-in-time manufacturing, on-demand printing, services that replace upfront investment with pay-by-the-drink capacity, CAD/CAM design, and mercenary contractors all mean that you don’t have to invest heavily up front (and therefore don’t have to argue a business case at the outset). Rather, you can ask for a modest budget, build analytics into the product, and launch sooner for less money. You can then use the data and customer feedback you get, which is vanishingly cheap to collect given today’s technology, to plead your case based on actual evidence.”
Scale and the Handoff
“Most of the time, the DNA of a disruptive organization isn’t well suited to “boring” management and growth, so you’ll need to hand off the product to the rest of the organization and find the next thing to disrupt.”
31. Conclusion: Beyond Startups
Don’t Eliminate Your Gut
“Go forth and ask good questions.”