[notes] Lean Analytics

Lean Analytics: Use Data to Build a Better Startup Faster (2013) by Alistair Croll and Benjamin Yoskovitz #

Buy from Amazon

NOTE: Bewarned, these notes are un-edited, un-revised, and un-styled. I plan on cleaning them up eventually, but until then, I apologize.

This book gets right down to the nitty gritty. So I don’t think I’m going to take notes on everything. For the stuff that relates to specific business models, I’ll just note that it exists, BUT IT’S ALL REALLY GOOD STUFF so I should remember that it exists for when I need it.

Part 1: Stop Lying to Yourself #

1. We’re All Liars #

“In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out.”

2. How to Keep Score #

What makes a good metric? #

Qualitative versus quantitative metrics #

Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less insight.

“Quantitative data abhors emotion; qualitative data marinates in it.”

Vanity versus actionable metrics #

Vanity metrics might make you feel good, but they don’t change how you act. Actionable metrics change your behavior by helping you pick a course of action.

Exploratory versus reporting metrics #

Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations.

Leading versus lagging metrics #

Leading metrics give you a predictive understanding of the future; lagging metrics explain the past. Leading metrics are better because you still have time to act on them—the horse hasn’t left the barn yet.

“Lagging and leading metrics can both be actionable, but leading indicators show you what will happen, reducing your cycle time and making you leaner.”

Correlated versus causal metrics #

If two metrics change together, they’re correlated, but if one metric causes another metric to change, they’re causal.

“If you find a causal relationship between something you want and something you can control, then you can change the future.”

Key performance indicators (KPIs) => specific metrics that drive the business

Segments, Cohorts, A/B testing, and Multivariate Analysis #

“Much of Lean Analytics is about finding a meaningful metric, then running experiments to improve it until that metric is good enough for you to move to the next problem or the next stage of your business.”

3. Deciding What to Do with Your Life #

Bud’s diagram shows three overlapping rings: what you like to do, what you’re good at, and what you can be paid to do. For each intersection between rings, he suggests a course of action:
• If you want to do something and are good at it, but can’t be paid to do it, learn to monetize.
• If you’re good at something and can be paid to do it, but don’t like doing it, learn to say no.
• If you like to do something and can be paid to do it, but aren’t very good at it, learn to do it well.

4. Data-Driven Verus Data-Informed #

“Data is a powerful thing. It can be addictive, making you overanalyze everything. But much of what we actually do is unconscious, based on past experience and pragmatism. And with good reason: relying on wisdom and experience, rather than rigid analysis, helps us get through our day. After all, you don’t run A/B testing before deciding what pants to put on in the morning; if you did, you’d never get out the door.”

“Humans do inspiration; machines do validation.”

“Machine-only optimization suffers from similar limitations as evolution. If you’re optimizing for local maxima, you might be missing a bigger, more important opportunity. It’s your job to be the intelligent designer to data’s evolution.”

10 common pitfalls when analyzing data: #

  1. Assuming the data is clean
  2. Not normalizing
  3. Excluding outliers
  4. Including outliers
  5. Ignoring seasonality
  6. Ignoring size when reporting growth
  7. Data vomit
  8. Metrics that cry wolf
  9. Not mashing up your data with data from other sources
  10. Not looking at the bigger picture

Part 2: Finding the Right Metric For Right Now #

5. Analytics Frameworks #

Dave McClure’s Pirate Metrics #

“McClure categorizes the metrics a startup needs to watch into acquisition, activation, retention, revenue, and referral—AARRR.”

Eric Ries’s Engines of growth #

Ash Maurya’s Lean Canvas #

Sean Ellis’s Startup growth Pyramid #

  1. Product/Market Fit (decide what you sell to whom, and prove it)
  2. Stack the Odds (find a defensible unfair advantage)
  3. Scale Growth (new markets, products, and channels)

The Lean Analytics Stages and Gates #

“We believe most startups go through these stages [Empathy, Stickiness, Virality, Revenue, and Scale], and in order to move from one to the next they need to achieve certain goals with respect to the metrics they’re tracking.”

6. The Discipline of One Metric that Matters #

“Capture everything, but focus on what’s important.”

“Optimizing your OMTM not only squeezes that metric so you get the most out of it, but it also reveals the next place you need to focus your efforts, which often happens at an inflection point for your business.”

7. What Business Are You In? #

“Sergio Zyman, Coca-Cola’s CMO, said marketing is about selling more stuff to more people more often for more money more efficiently.”

“Not all customers are good. Don’t fall victim to customer counting. Instead, optimize for good customers and segment your activities based on the kinds of customer those activities attract.”

The Business Model Flipbook #

8. Model One: E-commerce #

“It’s vital to know if you’re focused on loyalty or acquisition. This drives your whole marketing strategy and many of the features you build.”

“While conversion rates, repeat purchases, and transaction sizes are important, the ultimate metric is the product of the three of them: revenue per customer.”

9. Model Two: Software as a Service (SaaS) #

“In SaaS, churn is everything. If you can build a group of loyal users faster than they erode, you’ll thrive.”

“Many people equate SaaS models with subscription, but you can monetize on-demand software in many other ways, sometimes to great effect.”

10. Model Three: Free Mobile App #

“Most of the money comes from a small number of users; these should be segmented and analyzed as a distinct group.”

11. Model Four: Media Site #

“It’s hard to strike a balance between having good content and enough ads to pay the bills.”

12. Model Five: User-Generated Content #

“Visitor engagement is everything in UGC. You track visitors’ involvement in an ‘engagement funnel.’”

13. Model Six: Two-Sided Marketplaces #

“Early on, the big challenge is solving the “chicken and egg” problem of finding enough buyers and sellers. It’s usually good to focus on the people who have money to spend first.”

“Since sellers are inventory, you need to track the growth of that inventory and how well it fits what buyers are looking for.”

14. What Stage Are You At? #

  1. Empathy => you need to get inside your target market’s head and be sure you’re solving a problem people care about in a way someone will pay for
  2. Stickiness => you need to find out if you can build a solution to the problem you’ve discovered
  3. Virality => you need to get your product in the hands of visitors who are motivated to try you, because they have an implied endorsement from an existing user
  4. Revenue => you need to focus on maximizing and optimizing revenue
  5. Scale => you need to acquire more customers from new verticals and geographies

15. Stage One: Empathy #

“Right now, your job isn’t to prove you’re smart, or that you’ve found a solution. Your job is to get inside someone else’s head. That means discovering and validating a problem and then finding out
whether your proposed solution to that problem is likely to work.”

Finding a Problem to Fix (or, How to Validate a Problem) #

“Initially you’ll use qualitative metrics to measure whether or not the problem you’ve identified is worth pursuing. You start this process by conducting problem interviews with prospective customers.”

  1. Briefly set the stage for how the interview works
  2. Test the customer segment by collecting demographics
  3. Set the problem context by telling a story
  4. Test the problem by asking the subject to rank them in order of importance
  5. Test the solution by going through each problem and asking the subject how they solve it today
  6. Ask for the subject to agree to do a solution interview with you when you’re ready with something to show, or to refer other people for you to interview

How to Avoid Leading the Witness #

  1. Don’t Tip Your Hand
    • Biased wording
    • “You can get around this by asking people the opposite of what you’re hoping they’ll say.”
    • Preconceptions => if the subject knows things about you, he’ll likely go along with them.
  2. Make the Questions Real
    • “The more concrete you can make the question, the more real the answer.”
    • “One other trick to overcome a subject’s desire to please an interviewer is to ask her how her friends would act.”
  3. Keep Digging
    • “A great trick for customer development interviews is to ask ‘why?’ three times.”
  4. Look for Other Clues

Convergent and Divergent Problem Interviews #

“Problem validation can happen in two distinct stages.”

“A convergent problem interview gives you a clear course of action at the risk of focusing too narrowly on the problems that you think matter, rather than freeing interviewees to identify other problems that may be more important to them.”

How Do I Know If the Problem Is Really Painful Enough? #

  1. Did the interviewee successfully rank the problems you presented?
  2. Is the interviewee actively trying to solve the problems, or has he done so in the past?
  3. Was the interviewee engaged and focused throughout the interview?
  4. Did the interviewee agree to a follow-up meeting/interview?
  5. Did the interviewee offer to refer others to you for interviews?
  6. Did the interviewee offer to pay you immediately for the solution?

Getting Answers at Scale #

“Don’t just ask questions. Know how the answers to the questions will change your behavior.”

Build It Before you Build It (or, How to Validate the Solution) #

“Once again, this starts with interviewing customers (what Lean Startup describes as solution interviews) to get the qualitative feedback and confidence necessary to build a minimum viable product.”

“This is the reverse Field of Dreams moment: if they come, you will build it.”

Deciding What Goes into the MVP #

“It’s important to contrast an MVP with a smoke-test approach… With a smoke-test page, you’re testing the risk that the message isn’t compelling enough to get signups. With the MVP, you’re testing the risk that the product won’t solve a need that people want solved in a way that will make them permanently change their behavior.”

Measuring the MVP #

“The key is to identify the riskiest parts of your business and de-risk them through a constant cycle of testing and learning. Metrics is how you measure and learn whether the risk has been overcome.”

16. Stage Two: Stickiness #

“The focus now is squarely on retention and engagement.”

  1. Are people using the product as you expected?
  2. Are people getting enough value out of it?

“Your goal is to prove that you’ve solved a problem in a way that keeps people coming back.”

Iterating the MVP #

“Iterations are evolutionary; pivots are revolutionary.”

Seven Questions to Ask Yourself Before Building a Feature #

  1. Why Will It Make Things Better?
  2. Can You Measure the Effect of the Feature?
  3. How Long Will the Feature Take to Build?
  4. Will the Feature Over-complicate Things?
  5. How Much Risk Is There in This New Feature?
  6. How Innovative Is the New Feature?
  7. What Do Users Say They Want?

How to Handle User Feedback #

  1. Plan tests ahead of time, and know what you want to learn before you get started
  2. Group feedback from similar personas
  3. Review results quickly as you collect data

The Minimum Viable Vision #

“Even though you’re building the minimal product, your vision should still be big enough to inspire customers, employees, and investors— and there has to be a credible way to get from the current proof to the future vision.”

“A minimum viable vision (MVV) is one that captivates. It scales. It has potential. It’s audacious and compelling.”

17. Stage Three: Virality #

The Three Ways Things Spread #

  1. Inherent virality => happens naturally as users interact with your product
  2. Artificial virality => incentivized and less genuine, often built into a reward system
  3. Word-of-mouth virality => independent of product or service

Metrics for the Viral Phase #

“If every user successfully invites more than one other user, your growth is almost assured.”

“In addition to viral coefficient, you care about viral cycle time.”

Growth Hacking #

18. Stage Four: Revenue #

“The goal in the Revenue stage is to turn your focus from proving your idea is right to proving you can make money in a scalable, consistent, self- sustaining way. Think of this as the piñata phase, where you beat on your business model in different ways until candy pours out.”

Metrics for the Revenue Stage #

“You’re moving from proving you have the right product to proving you have a real business. As a result, your metrics shift from usage patterns to business ratios.”

Customer Lifetime Value > Customer Acquisition Cost #

“The core equation for the Revenue stage is the money a customer brings in minus the cost of acquiring that customer. This is the return on acquisition investment that drives your growth.”

“Think of a business as a machine that converts money into greater sums of money. The ratio of money in to money out, as well as the maximum amount of money you can put in, dictates the value of the business.”

Market/Product Fit #

“If things aren’t working, it may be easier to pivot your initial product to a new market rather than starting from scratch.”

The Breakeven Lines in the Sand #

“While your goal is to grow, you should also keep an eye on breakeven, because once you can pay your own bills you can survive indefinitely.”

19. Stage Five: Scale #

“When you’re scaling, you know your product and your market. Your metrics are now focused on the health of your ecosystem, and your ability to enter new markets.”

The Hole in the Middle #

“You need to understand if you’re focused on efficiency or differentiation. Trying to do both as a way of scaling is difficult. If you’re efficiency-focused, you’re trying to reduce costs; if you’re differentiation-focused, you’re increasing margins.”

Metrics for the Scale Stage #

“You’ll look at compensation, API traffic, channel relationships, and competitors at this stage—whereas before, these were distractions.”

The Three-Threes Model #

“As you grow, you’ll need to have more than one metric at a time. Set up a hierarchy of metrics that keeps the strategy, the tactics, and the implementation aligned with a consistent set of goals.”

20. Model + Stage Drives the Metric You Track #

Part 3: Lines In The Sand #

21. Am I Good Enough? #

Number of Engaged Visitors #

Cost of Customer Acquisition #

22. E-commerce: Lines in the Sand #

23. SaaS: Lines in the Sand #

“Try to get down to 5% churn a month before looking at other things to optimize. If churn is higher than that, chances are you’re not sticky enough.”

24. Free Mobile App: Lines in the Sand #

Average Revenue Per Paying user #

25. Media Site: Lines in the Sand #

26. User-Generated Content: Lines in the Sand #

“You’ll have a very good indicator of stickiness when site visitors are spending 17 minutes a day on your site.”

27. Two-Sided Marketplaces: Lines in the Sand #

28. What to Do When you Don’t Have a Baseline #

“For many metrics, there’s simply no ‘normal.’”

“Nearly any optimization effort has diminishing returns.”

Part 4: Putting Lean Analytics to Work #

29. Selling into Enterprise Markets #

“In most cases, enterprise sales involve bigger-ticket items, sold to fewer customers.”

“Zero Overhead Principle: no feature may add training costs to the user.”

“…most B2B-focused startups consist of two people: a domain expert and a disruption expert.”

30. Lean from Within: Intrapreneurs #

Daniel C. McCallum #

“In their book Confronting Reality (Crown Business), Larry Bossidy and Ram Charan list the six habits of highly unrealistic leaders: filtered information, selective hearing, wishful thinking, fear, emotional over- investment, and unrealistic expectations from capital markets.”

14 Rules for Lean Intrapreneurs #

Changing—or Innovating to Resist Change? #

“Critics of Microsoft’s reactions complain that the company isn’t changing; rather, it’s managing to stay the same and exert its dominance, avoiding or delaying market change. ‘I realized that Microsoft had not turned at all,’ said Dave Winer in 1999. ‘What’s actually been happening is that Microsoft is exerting tremendous energy to stay right where it is.’”

Skip the Business Case, Do the Analytics #

[!!! interesting]

“Traditional product managers build profit- and-loss analyses to try to justify their plans: they create a convincing business case, and once someone believes it, they get funding to proceed. But a Lean mindset reverses this: you sell the business model—not the plan—without a lot of prediction, and then rely heavily on analytics to decide whether to kill the product or double-down on it.”

“This analyze-after rather than predict-before model is possible because many of the costs of innovation can be pushed later in the product development cycle. Just-in-time manufacturing, on-demand printing, services that replace upfront investment with pay-by-the-drink capacity, CAD/CAM design, and mercenary contractors all mean that you don’t have to invest heavily up front (and therefore don’t have to argue a business case at the outset). Rather, you can ask for a modest budget, build analytics into the product, and launch sooner for less money. You can then use the data and customer feedback you get, which is vanishingly cheap to collect given today’s technology, to plead your case based on actual evidence.”

Scale and the Handoff #

“Most of the time, the DNA of a disruptive organization isn’t well suited to “boring” management and growth, so you’ll need to hand off the product to the rest of the organization and find the next thing to disrupt.”

31. Conclusion: Beyond Startups #

Don’t Eliminate Your Gut #

“Go forth and ask good questions.”


Now read this

Hackathons, or Now I’m Standing on the Overpass and Screaming at the Cars, “Hey, I Wanna Get Better!”

I think we should take four months off next Fall to run the hackathon circuit. Why? # We all have experience with product design, but our experience has been with sustaining innovation. We’ve been given existing products and told to... Continue →