the Sha Ellis test such a seemingly simple idea that has had such a profound impact on the startup world the question is how would you feel if you could no longer use this product once you got a high enough percentage of users saying they'd be very disappointed most of those products did pretty well if you felt too low those products tended to suffer say someone is listening and they're like okay man I'm getting like 10% I don't know what to do what do you find often works just ignore the people who say they'd be somewhat disappointed they're telling you it's a nice to have if you start paying attention to what your somewhat disappointed users are telling you and then you start tweaking onboarding and product based on their feedback maybe you're going to dilute it for your musthave users moving retention often is really hard but I guess it sounds like there's often something you can do it's usually much more a function of onboarding to the right user experience than it is about the kind of the Tactical things that people try to do to improve retention what are like three or four things that you think people should definitely try to help improve activation in my experience today my guest is sha Ellis Shawn is one of the earliest and most influential thinkers and operators in the world of growth he coined the term growth hacking invented the ice prioritization framework was one of the earliest people to use fremium as a growth strategy and maybe most famously developed the Sha Ellis test to help you understand if you have product Market fit which a large percentage of Founders use today and profoundly impacted the way startups are built over the course of his career Shawn was head of growth at Dropbox and Eventbrite helped companies like Microsoft and newbank refine their growth strategy was on the founding team of log me in which eventually sold for over $4 billion and he's the author of one of the most popular growth books of all time called hacking growth in our conversation we dive deep into two topics one how to know if you've got product Market fit and what to do if you don't and two how to figure out how to grow once you found product Market fit if you're in the early stages of a new product wrangling with product Market fit or trying to figure out how to jump start or further accelerate growth for your product this episode is for you if you enjoy this podcast don't forget to subscribe and follow it in your favorite podcasting app or YouTube it's the best way to avoid missing future episodes and it helps the podcast tremendously with that I bring you Shan Alice Sean thank you so much for being here and welcome to the podcast thanks Lenny I'm I'm super excited to be on with you there's so much that want to talk about there's so many directions we can go but to keep it focused I want to spend time on two areas I want to talk about how to know if you have product Market fit and what to do once you have product Market fit in terms of figuring out how to grow yeah and I know these things are very linked I know you spent a lot of time on these things how does this feel sounds perfect yeah let's do it okay okay amazing let's talk about first of all the Shan Ellis test slash something people call sometimes the product Market fit test such a seemingly simple idea that has had such a profound impact on the startup world I've never actually seen you talk about the history of this thing how you came up with these questions how you came up 40% the whole journey of this thing so let's talk about this but first of all can you just tell people what is the shanell's test for folks that aren't exactly familiar with this it's a simple question that helps you figure out you know does anyone consider your product a must have or you know ideally who and how many people consider it but but ultimately it's trying to figure out you know do is your product a must have which could be equated to to having product Market fit and so the question is how would you feel if you could no longer use this product and I give them the choice very disappointed somewhat disappointed or uh even not disappointed or not applicable I've already stopped using the product and what I'm trying to find are are those people who say I would be very disappointed if I could no longer use this product and that's that's a really powerful vein to dig into when you when you discover that you actually have some people who would who would give a crap if your product disappears this episode is brought to you by gamma an entirely new way to present your ideas powered by AI if you hate designing slides and Dread that feeling of staring at a blank slide gamma is here to help just upload your PRD and turn it into a beautiful ready to present presentation in seconds gamma works with all types of form from Google Docs PDFs to PowerPoint you can even drop in a link to your favorite Lenny's newsletter post and turn it into a presentation for your team gamma has become one of the fastest growing AI web products in the world adding 20 million new users just this past year and is setting its sights on becoming the modern alternative to PowerPoint whether you have design skills or not gamma can save you hours of time synthesizing your ideas and shaping your content visit gamma. apppp and use promo code Lenny to get a free month of gamma Pro that's G.A let me tell you about command bar if you're like me and most users I've built product for you probably find those little in product popups really annoying want to take a tour check out this new feature and these pop-ups are becoming less and less effective since most users don't read what they say they just want to close them as soon as possible but every product Builder knows that users need help to learn the ins and outs of your product we use so many products every day and we can't possibly know the ins and outs of everyone command bar is an AI powered toolkit for product growth marketing and customer teams to help users get the most out of your product without annoying them they use AI to get closer to user intent so they have search and chat products that let users describe what they're trying to do in their own words and then see personalized results like customer walkthroughs or actions and they do pop-ups too but their nudges are based on in product behaviors like confusion or intent classification which makes them much less annoying and much more impactful this works for web apps mobile apps and websites and they work with industry-leading companies like Gusto freshworks hashy Corp and launch Darkly over 15 million end users have interacted with command bar to try out commandbar you can sign up at commandbar docomond per month for any plan that's commandbar and the idea is that if you 40 40% or more of people say they'd be very disappointed if they can no longer use the product you essentially have product Market fit I would say it's a leading indicator of product Market fit the the the lagging indicator is do they actually keep using it so probably retention cohorts are are more more accurate but the problem is like if you you know like your time at Airbnb what how long do you have to look at a retention cohort before you know that you've actually long-term retain someone and so with this question you can you can kind of find out day one you don't actually even need to be you don't need a a good analytic system in place to be able to see if if product Market fit exists and so um yeah the the 40% the 40% was not something I originally had in there I was also originally I was I was trying to have just a filter so that I was not treating all feedback from customers the same but I was I was trying to find feedback back from from customers who actually really cared about the product and then uh was over time as I at the time I was as working for a couple of ycb companies and so those companies were all pretty pretty connected and and so I would share share the question with a lot of other startups in in Silicon Valley and so over time I started to see there was a pattern that you know once you got a high enough percentage of users saying they'd be very disappointed most of those very disappointed without the product most those products did pretty well and then if you if you fell too low those products tended to suffer okay there's two things I want to definitely follow up on here the first is such an important point that you made at the beginning when I introduced this test is that you described it as a leading indicator of product Market fit and actually retention people actually using your product the product actually being used by the market is the actual ultimate test right so the idea here is this is a good way to get a sense of before you actually have data are we headed in a good direction could you just more about that like how to when to use this and when it's most useful in best yeah I mean so for me in particular when I come into a company my my my goal is to help them grow and so I don't want to put myself in a situation where I'm gonna fail because no one actually cares about the product and so it it can really be asked at at a company of any stage it's it's helpful to understand who your musthave users are but essentially once once you have a an even an MVP of like very first MVP on the product you can still get some some useful feedback about the product if if if it's resonating with anyone so I actually had a a company where um I I had committed to work with them it was right after I I left Dropbox and I I committed to working with these guys for six months to help them grow I ran the question and it came back at only 7% of users saying they'd be very disappointed with without the product and so I'm like I have six months to help them grow and they're only at 7% right now who it might take six months to get the 40% am I doing them a disservice by being in a growth role and and you know being on payroll during during this period of time but fortunately with the signal and the information we got from the initial survey we were able to get them at 40% in two weeks wow what did you do there just as a case study yeah yeah so the company called called Lookout it's a mobile security company and you know now most of the things in Lookout were are are like built into iPhones and Androids but at that time the product had uh had everything from like back up my data to find my lost phone to protecting your phone from uh you know with with a firewall and Antivirus and so when we ran this initial survey I dug into the 7% who said they'd be very disappointed without the product and found that um most of that 7% were focused on the antivirus functionality so they were like they know they need to protect their their computer from viruses smartphones were becoming more like uh computers so it just made a lot of sense for them to that they need to protect their their phone and interesting at the time I think there was only like one kind of phone virus that had ever even happened but it was a pretty easy mental leap for people and so now we knew okay it's antivirus that people really valued and so step one was just uh reposition the product on antivirus so that that kind of creates a filter so anyone who now is coming in to sign up for the product who doesn't care about antivirus is not going to convert and those who are excited about antivirus are going to convert we already know from the initial survey that that people value that after they convert so by setting the right expectations around it up front you're going to bring people in with the right expectations but then the second thing that we did was we streamlined onboarding so that the first thing that they did after signing up for the product was to set up the antivirus and then get a message You're Now protected from from viruses and so it's really the combination of those two things it's set the right expectations and then speed to value and so the next cohort of people that we surveyed were at 40% saying they'd be very disappointed without the products that was literally took two weeks to make those changes six months later it was 60% on on the on the score and then uh I think they hit the the billion dollar valuation four or five years later on uh you know ultimately being one of the one of the you know early unicorns and interestingly they you know as all of those things were built into Mobile phones now they they've completely changed the business but they continue to do really well um but they've continued to iterate the business I think that having having that kind of finger on the pulse early in the business was was important to to build the muscle in the business to be really uh responsive as the market changed Sean this is already amazing there's just like a fractal of topics I want to explore from this very short conversation already so the first is just follow this thread of basically you're sharing kind of a growth strategy that I imagine you execute is look for the percentage of people that would be very disappointed if your product went away see who they are see what they're excited about and lean into that both positioning Wise onboarding Wise and probably also cut out stuff from your product that they don't care about yeah and and I was coming at it from a a marketing perspective initially you know over time I positioned my more myself more in in a growth role with product and and marketing as areas I could influence but as a marketer I I probably didn't have a lot of influence on a on a uh engineering founded company to say let's cut cut out stuff so uh it made more sense to say let's just sequence the onboarding so that we're we're highlighting this and and onboarding to this that was a little easier to sell and just hearing that you can move this score so quickly without even changing the product substantially uh I imagine would surprise a lot of people when you think about like moving retention often is really hard and maybe we talk about that but I guess it sounds like there's often something you can do that's not very hard that might significantly shift uh this product Market fit test right and then that ultimately like yeah moving retention is really hard but it's it's usually much more a function of onboarding to the right user experience than it is about the kind of the Tactical things that people try to do to improve retention okay I want to put a pin in that and come back to that because that's a really important topic I'm going to come back to say someone runs this survey and they get 40% what should they have in their mind of like this is what this is telling me because I think a lot of people are like I got product Market fit I got this let's go go go what's the best way to think about what this tells you yeah I mean it it does it tells you something really important which is you know you you you haven't created something that uh people don't care about so that's that's an important Insight but until you deeply understand that product Market fit you you kind of don't have the tools to be able to grow the business so that's that's really the next step is to to dig in and figure out know who considers it a must have how are they using the product what did they what did they use before what problem are they solving um one of my one of my favorite questions is uh so I I I tend to have a lot of questions that I that I build off of that I'm using that filter of you know trying to drill into the the users who say they'd be very disappointed without the product and one of my favorite questions is what is the primary benefit that you get and then um I use that initially as an open-ended question to to kind of crowdsource different benefits people are getting but then then I run another survey where I turn it into a multiple choice question forc them to pick one of four kind of distinctive benefit statements and then the question that follows on that next survey is why is that benefit important to you and then I start to get really good context so that I actually came up with this question when I was working with uh an early YC company called um zne and uh which is Inbox spelt backwards and I when I when I ran that question basically the people who said they'd be very disappointed without the product were focused on zne helps me find things faster in my email so it's it's great to know okay that's the benefit but when I asked why is that benefit important to you they said I'm drowning an email like I just I kept seeing that statement as as a written statement and so when I then was trying to to to figure out how to acquire customers when I tested drowning an email question mark that set such a good hook that was the context that people were living in that they were really responsive to the message of find things faster with zne and then a description of what zne is so like I I think when you can when you can really dig into the context of why that must have benefit is important to people you you start to you start to get the ingredients to build that flywheel that leads to long-term sustainable growth so what I'm hearing is whether you have 40% whether you have 60% or even 7% the actual best use of this tool is look at that percentage of Highly disappointed and see what they're looking for what they're excited start drilling in start peeling back that onion and just deeply understand them and and uh make sure that you're that you're you know ultimately your road your product road map is doubling down on the things that are important to your must have customers your onboarding is bringing new people to the right experience your messaging is setting the right expectations your your acquisition campaigns are targeting people who actually have the need and so it's it's all about getting the right people to the right experience and then even your engagement loop it's about just reinforcing what uh you know how to get people to experience that benefit more often awesome and the 40% threshold so what you shared is you basically emerged from just looking at tons of startups doing the survey and finding a pattern How firm is that 40% like how big of a deal is it 39 versus 41 I don't think it's that firm you know to me I think the real power is having some kind of some kind of Target for the team to be shooting for that basically says we're not going to aggressively start to grow until we hit this Target and I think that as just a focusing um piece is really really important because I think one of the one of the biggest challenges in in an early stage startup is half the people feel like you know we're we're years from from having this product ready to grow and half the people are like what are we waiting for where if you can actually get people on the same page of what is what what does product Market fit look like for our business and and it's at that point that that we're g yeah and and I before I ever heard the term product Market fit I I remember the conversations back at at log me in in the mid 2000s of of kind of like when do we step on the gas what is what is the combination of factors that need to be in place before we start pouring fuel on the on the early fire and so yeah I think that kind of nail it and scale it it's probably been a a term that's been around for DEC ades now but you know it's it's all kind of pointing to that same concept of product Market fit how often do you have you seen false positives with this test or someone gets 40% and something is not not right they they're actually far from it or is it generally pretty accurate if if you're having people say that they'd be very disappointed without your product that um that's that's a really good sign what I can tell you is that not not just a not necessarily a false positive but like what is driving people to say they'd be very disappointed one of one of my favorite books is uh hooked by nearer all and he talks about in the in the kind of Engagement Loop that your last step is investment and so I ran the survey on a business that I thought was a fairly commoditized business and um I wanted part of it I wanted to see could I could I use the same goto Market approach on a later Stage Company and and and use it to accelerate growth and so this is a a business called um webs.com and uh they they eventually got acquired by Vista Print but they'd been they've been pretty flat for the year before I went in there and then I I started to kind of use this approach to to try to dial in uh their their their growth engine and I ran the survey thinking you know you you've had products like Wix and Weeble that have have come on to the market since this you know more Legacy website Building Product has been around and I personally think they're a little easier to use they're a little better and so I didn't have high hop when I ran the survey but it came back with one of the highest scores I'd ever seen and it was like like 90% of the people saying they'd be very disappointed if they could no longer use the product never seen that and I was like how could that be possible this this product is is kind of a commoditized category I wouldn't even say it's one of the best and and then when I want I dug into it again it comes back to that Nuri all hooked model is that the investment people have made in building that website they've put so much into you know that they they know exactly how to make the changes and in the kind of the the CMS kind of site of things they have have spent a lot of time just making it beautiful and so so ultimately um it was something that that that was why they were saying they'd be very disappointed I mean but but for you know kind of fast forward and uh on when I initially went in still doing these things helped the business resume growth and and and have significant growth over the next 12 months after we did these things so still the signal we got from why people would be very disappointed uh without the product was was important and speed the value and kind of all all of the all the other things I think about in go to market for an early stage product still were relevant but just the I think they're they were a little stronger on the uh you know percentage he'd be very disappointing even even Eventbrite when I was there when we ran it was probably the second highest I'd ever seen but with event organizers if they've already set their event up on that platform and they've sent it out and and to to their list and and all those people are coming in and they're managing their event again they've they've invested a a lot in the platform so so sort of switching costs I think I think can can factor in there so it's it's a a function of both switching costs and utility of the product so that that's a question I wanted to ask is what's what's your guidance on when to ask this question what I'm hearing is if you ask it very far along the journey when they're very invested you'll get a much higher score is there advice on how to the timing on the best way the best time to ask this question of your users what I recommend is a random sample of people who've uh really used your product so they they they've gone in they didn't just sign up but they went in and and and hopefully like hit that activation moment they've used it twice at you know two plus times and uh and and they've they've ideally used it say with in the last week or two weeks so that it's you know they haven't churned yet so if it's a random sample of those people that's that's kind of the ideal time to ask it got it so basically it's people that have activated whatever that means to you and have been using it for a couple weeks not not people like landing on your homepage not people just signing up right not people months later not people who've seen like a demo of your product but it's it's people who actually have experienced the product but it's okay if they you if if you're hitting people who've used it months later but like in that Lookout example that I gave if I'm if I'm testing people's uh perception of the product after I made updates to the onboarding I'm GNA only want to I'm gonna only want to survey people who went through the new onboarding yeah experimental group yeah okay so I asked people on Twitter what to ask you a lot of people had a lot of awesome questions I'm gonna ask I'm G to sprinkle in a couple of these questions throughout the chat one came in from Shas Doshi uh popular guest of the one of the ones that I listened to recently amazing I think it's the second most popular episode behind bries um okay so he had a question of just what are the limitations of the score when does it break down when should you not use it if ever is there anything of just like here's when it's not going to work for I think one off products would probably you know like how would you feel if you could no longer watch the movie you just watch like I wouldn't care um even like when I run a workshop I don't run I don't run this this as part of my survey after I do a workshop because um like how would you feel if you could no longer attend the workshop you just attend it doesn't make make sense so I'll ask an NPS uh question as my as my filtering question so that I I'm I'm looking at you know focusing on feedback of people who love it also then through a separate lens looking at the people maybe who who would be my detractors so I think I think one-off products are probably not not good products to to run the question on there may be other other uh places as well that I'm not thinking of right now but that but it sounds like not many what I'm hearing is it's generally widely applicable yeah I think it is like at least from from my perspective like I uh it's been really useful for me anyway awesome okay and then the followup question from stos is and I kind of asked this but I'm curious if there's anything more here if just have you seen any instances of startups over relying on the score prematurely declaring product Market fit when in reality they haven't reached it yet and just they nether caveats of like cool I got 40% is there anything else you should know like okay but maybe check this one thing yeah I mean I think to me it's kind of like what really is the definition of product Market fit is the definition that people who get through my crappy onboarding and actually experience the product love it and if I'm able to retain those people that means I have product Market fit or is fixing that crappy onboarding part of getting to product Market fit as well I think that's Up For Debate so I you know to me the hardest I wouldn't obsess on onboarding if if I know those who who kind of get get through the challenge of getting started with a product still don't like the product then feels like it's a a core product issue or you know wrong people using it in the wrong way issue but you know once once you have that then then ultimately it doesn't mean that you're you're ready to grow like when when I focus on growth then customer acquisition is almost the last step like once I once I validate that it's must have for those early users then I'm thinking about okay how how do I optimize speed to Value how do I make sure that people have the right prompts to come back and use the product at the right time so it's kind of more of that engagement Loop how do I how how do I get my existing users to bring in more users if if there's something that makes sense on that end even how do I how do I optimize my Revenue model and um once all of those things are working well then I'll obsess on the customer acquisition side but um like customer acquisition is so hard that if you if you're not really efficient at converting and retaining and monetizing people you're gonna really struggle on the customer acquisition side yeah cool and we'll talk about uh customer acquisition growth sure um another question I want to ask and a couple listeners asked is the 40% so I had a Jag from newbank on the podcast I think you may have worked with them and they use 50% as their threshold because uh apparently Brazilians are very nice and they just I think he said yeah yeah so uh I guess the question is do you find instances where you should increase that percentage and slash in B2B is anything different uh do you change the percentage in B2B any advice there just like when you adjust the threshold yeah I hadn't really thought thought too much on that um again for for me you know generally I'm like I'm I'm trying to just figure out is this a product that can grow so like when I'm when I'm using it I'm I'm trying to just uh you know so if I got a 37% am I going to be like oh no this would be impossible um you know or if I had a a 70% does that mean I guarantee like say oh yeah I want to jump in and work with this company um like it's it's more nuanced than that like like obviously um if if it's a 70% but I have have no idea how I grow the business that I'm I'm going to be stuck there so I but I do think he he brought up a really good point that that culturally some people are going to be more um more optimistic or pessimistic interestingly when I came up with a question I I used to I used to just use kind of a normal satisfaction question I was when I was working at zne I I would I'm I'm just like an intensely curious person anyway so I'm just trying to dig in and understand the customers and so I've always done lots of surveying but at at zne I was gonna ask it as like use my filter as a satisfaction question so you know how satisfied are you with this I'm I'm very satisfied I'm somewhat satisfied and I our our main customers were actually Senior Management and so I thought you know senior Management's Never Satisfied I'm GNA get always this like super lukewarm thing how can I how can I change this question to be more uh you give give me a more kind of real answer from these guys well if I if I flip it and say how would you feel if you could no longer use this product I'll probably get a more honest answer back from them and and of course they're very disappointed if they can't get what they want and so initially it was just for the for the case of of zne but then I went to to Dropbox right after zne and and like oh I'll try the question again and the insights I got back were really useful and so each company I went to I I kept using the question I'm like this works way better than your typical satisfaction question but but initially it was was more about thinking just you know Senior Management to to get a more honest answer out of them so that's the origin story right there yeah wow that's senior managers are just very harsh and they they're not they don't need anything yeah and you have to flip it that is so interesting just like that question is such a good reminder of how hard it is to build anything people really would be disappointed not to have like that's why this works so well people are like ah I don't need this and app cares right like that's the core of this is just that is hard I me especially when I first moved to Silicon Valley so first like 15 years of my career were not in Silicon Valley and so that was as an Eastern Europe and then New York and then Boston you move to Silicon Valley and you have you have people who get really excited about technology for technology sake and so you know just something being cool is like isn't it cool that we can actually do this you know drives a lot of people and so you know to to me I'm I'm I'm very like practical if if if it's not something that that is really bringing value to people then then the likelihood that that product's successful long term is going to be pretty low and so even interestingly at at Dropbox I through the six months I was there I would ask I'd ask one question like like multiple times a month um to I broke kind of the the early beta users into into a bunch of different lists and I'd ask which best describes you I like to be among the first to try cool new technology or I only try things that I think will be useful for me and over the six months it flipped from 90% being people who try things that they want to try cool new technology to six months later it was people who only you know only are going to try something that they feel like is is useful but what's kind of cool is just because what motivates you to try something is like you're an early adopter and you want to try something cool if you're G to keep using it it's because it's giving you some utility and so I could I can still use those early adopters to uh help me figure out where is the value inside the product awesome so actually two questions along those lines how durable do you find this percentage being say you hit 4% how often does that fade and go away versus stay there or go higher yeah I haven't seen it really like fade back down but it's um but I've seen companies Fail You know despite having it and I think a lot of times then it's it's uh you know it's it becomes like an execution challenge once you once you have product Market fit you know not everyone's going to be a good executor but before that like I think getting to product Market fit obviously there's there's a lot of methodology for for for doing it today that that might make it a bit easier for people but I still think it's fairly random and pretty dang hard and so ultimately like the risk factor of creating something that people care about is is really difficult so if you can get to the point where you have 40% of the people are using it saying they'd be very disappointed and and you have a reasonable sample size let's say you know you got 10 people and four of them said they'd be very disappointed without it you're still going to get something useful from those four but I I wouldn't say that's a sample size that you can you can really like go to market on so yeah what's a good sample size you look for just like okay this is actually good data I want to rely on that's really funny like I so so so much of the stuff I kind of like self-learned but I I I basically at one point said I need at least 30 responses and and I just thought I just kind of randomly made up a number and then and then I had people telling me yeah 30 is kind of the minimum that you want on stuff like okay and even when I first cre the survey I remember showing it to the to the uh co-founder of SlideShare and she was like her PhD was in you know survey related stuff like cognitive psychology but she basically said it was it was really about surveying and she's like this methodology is amazing how' you come up with this and so having some of that validation around these things helped but you know a lot of it was just again driven by my own curiosity and um and also just knowing that the failure is such a likely outcome that you know trying to trying to reverse engineer that failure and the and the number one reason for failure would be that people don't actually care about the product and so when I find that that's that's a that's a really good sign that it's we're now down to an execution Challenge and there's this obvious element of you may have product Market fit with people but that group might end up being very small and the business you build around it could actually be cool but it's not going to be a massive business is there anything there you can share just like it's hard to know the size opportunity even though you know some people really really like it yeah I talked about I I I go to a uh multiple choice after initially use um open-ended to open-ended questions to sort of crowdsource the different use cases but then I try to force people in a bucket and then I can run filters on each of those buckets and I'll be like oh people who use it this way are like 60% likely to be very disappointed without the product but people who use it this way are 35% likely to be very disappointed but way more people use it the 35% way and so then then having to like you know do do you want that like intensely loyal group or the or the much broader much broader group that's that's maybe a bit less but almost there I think that becomes a bit of a strategic um a strategic conversation of of like you know do we do we want to have a better chance of surviving going after a a Niche that that we know we can serve well or have we have we raised so much money that we have to go after like a really big market and and that that's not that one's not gonna be longterm but maybe maybe then you're you're like okay once I once I have Traction in that market I can I can start to try to appeal to some other markets so but I think that's where kind of some strategic decisions come in do you have a euristic of which you often recommend or is it very dependent on the situation I prefer kind of a a more passionate customer base that's and and work from there just because I think your biggest competition when you're when you're really innovating is is just like being irrelevant and and so if you're if you're like deeply relevant to anyone I think that gives you a much better chance of long-term success awesome that's a really good Insight okay two more questions along this line and then I want to talk about growth strategy uh one very tactical question is there is there a tool you recommend for doing this sort of survey like you recommend inline in the product an email something else I've used a lot of different uh tools I actually had a survey business that I sold to private Equity years ago and and um that was a on on product you know in it was it's called qualaroo that's like kind of yeah inflow survey tool I don't think you know I I think just like using Survey Monkey with uh with emailed surveys works fine and for me it's a lot more of like uh what do I what what's like Pleasant for the customer to fill out and then how what's gonna give me something where I can I can work really easily with the data so at bounce for example um they they had already intercom in place that had just introduced surveying but it was kind of a really crappy customer experience on on at least at that time that that's that's been almost a year now actually a little over a year and um so I'm I'm really sensitive to like is it a good survey experience for the consumer itself and then um but yeah I I I don't think I'm I'm stuck to any one platform there such an important topic yeah just like again to remind people why this is so important one of the most common questions Founders ask is do I have product Market fit have I built something people want like that's just an endless series of I don't know how do I know when do I know and this is telling you in a really interesting way so your advice is this is a leading indicator you don't actually know until people actually start using it and whether they retain and continue using it is there just like a advice on the shift you make from relying on the survey to actually looking at retention cohorts is it just once you have enough data once you have a couple cohorts then start looking at that forget about the survey yeah what I would say is um uh like but retention cohorts don't don't give you any of the qualitative insights into the why so that's why I would continue to do the survey so initially I would say if the survey comes back and it's you know it it shows whatever your target number is if you want to be like new bank it be a 50% or you you know I I spent a few years two of the companies I launched we launch in Hungary and I would say it's kind of the opposite end of the spectrum of of uh Brazil maybe maybe more pessimistic than the than the average kind of culture and so maybe maybe 30% is good enough there but but that ultimately whatever your target is that you you you have the signal that says okay we we have enough value here let's start working on growing the business but while you're working on growing the business I would be paying attention to those retention cohorts and if if you're churning out all the customers who uh who who were saying that they'd be very disappointed without the product then okay let's let's retrench and and rethink do do we really have product Market here and what what do we need to do to to to get it if we don't awesome uh and speaking of new bank if anyone wants to see how a company has actually operationalized this in the way they operate that there's an episode that we'll link to in the show notes where every new product at new bank they build yeah before they launch it they wait for a 50% uh threshold for people to say 50% of people would be disappointed if this product did not exist as they're developing it and only then do they launch it publicly yeah I think they even do it down to the feature level wow so if you if you think about it like you know how would you feel if you could no longer use this feature starts to give you again the signal is that is that feature a must have feature and if it's not maybe maybe we shouldn't have it and so um yeah I was I was super excited when I when I saw how they were using the the and they were doing it before I engag with them oh wow but uh awesome they were doing it I think from pretty early on in the business the reason they can do this is they have a lot of users they have millions and millions of users so they can ask some small percentage of people this question because people hearing this might be like oh my God how many times am I going to be asked this question I'm using this feature but they have a lot of users so it's easier yeah yeah okay last question I promise along these lines say someone is listening and they're like okay man I'm getting like 10% I'm getting 15% I don't know what to do to increase my product Market fit you should just like a strategy of just dig into the people that are very disappointed and see what they have to say but any other advice slash what do you find often Works in helping people move from say 10% to 40% yeah so one of the things that's kind of cool about almost like open sourcing the survey approach is uh is again watching like how how newbank has uh has evolved their usage but um one of the other companies that I think yeah used it in an interesting way is superum and they I would I would say that they basically ended up probably putting a lot more momentum behind the question than than it had even before they they posted something about how they did it on first round capitals blog and what I have always said and again it's me coming at it from a from probably initially a marketing background which is like I'm I'm kind of taking the product as a as a fixed as a fixed thing and how how do I actually figure out how to how to Market and grow this product and product changes are going to take a long time and and so kind what are the variables that I can control with a marketing background but so one of the things I've always said is you know just ignore the somewhat disappoint the people who say they'd be somewhat disappointed they're telling you it's a nice to have like they're they're as good as gone so just ignore those guys and then but what super hum did was and and the reason that yeah put one piece in the middle there before I say what super human did the reason that I say ignore those guys is that if you start paying attention to what your somewhat disappointed users are telling you and then you start tweaking onboarding and product based on their feedback maybe you're going to dilute it for your musthave users and that ultimately it becomes kind of good for everyone but not great for anyone and so that was my my fear of like trying to trying to read too much into the users who say they'd be somewhat disappointed but the the Superhuman guys actually found I think a good way around that where they said okay what is the benefit that my musthave users are focused on and then of the users who say they'd be somewhat disappointed so the the nice to have users of those users who are also focused on that benefit what what do they need in the product for it then to become a must have for them and so they're staying true to that kind of core benefit but they're trying to essentially take those on the fence users and moving them up and so I think their their way of approaching that addressed what my concern was which is are we going to are we going to kind of break it for the must have users that's an awesome Insight by the way did Rahul and the team there just do this on their own or were you involved in any way in this super apprach I mean that's the same thing like I said I I wasn't initially involved with newbank I wasn't involved with them and that that's the benefit of you know we wrote about it in our book in 2017 and so I think that I got it out there but I I actually teamed up with the kiss metrics team in uh 200 maybe 12 and and and essentially published this survey on survey. where we just made it freely available for people and a really easy template to to to prepare and send out and the how-to guide on it it was all just yeah free I think kissm was kind of using it as uh as maybe Legion and and for me I just wanted a way to to kind of put something out for the community and um and so it's been out there for a long time so it's not surprising that um that that different companies have found different unique ways to use it that's awesome I think that post is one of the most popular in first round really had an impact on a lot of people yeah so just to repeat the uh approach you recommend for when you're digging into I wrote this down when you're talking for how to dig into what benefit people are finding your advice is it's basically a follow-up survey to the extremely disappointed people asking them what is the primary benefit you get it's a open text initially then once you get a collection you do it sounds like another survey as multiple choice here's like to a different group of people to be clear different group got it awesome and then it's like which of these four or five benefits is is what you're getting out of this product and then the question is why is this benefit important to you yeah and then you'll see um we have uh like eventually the the survey. got closed down but the essentially the template that I I typically used was then moved to pmfs survey.com and so you'll see some other question questions that I have on there as well like what would you use instead if this product were no longer available and that's one of the interesting things as you start to see people say they'd be somewhat disappointed usually they're focused on a commodity use case and they they know an easy alternative to switch to so to be a must have it needs to be both uh valuable and unique okay anything else on this topic of the Shan Alis test product Market fit test before we move on to growth strategy advice no I think that's yeah that's I think we did an hour on that one topic which I love because I feel like this is such a powerful tool that I think people sort of know and have used but I think there's a lot of opportunity to use it more effectively and all the stuff you point out about like it's not just get this you have this threshold cool let's move let's grow it's like this is just this is how you figure out how to make it better and better and grow faster and faster and it's actually a good Segway talking about growth so even though you coin the term growth hacking you spend most of your time on the opposite essentially which is helping companies figure out sustainable growth strategies not just a bunch of hacks to grow for a little bit and then disappear and from what I've seen it's all rooted in this idea of product Market fit and what helps you find product Market fit and I imagine many of the stuff we've talked about yeah just one one quick uh interjection there is that when I coined growth hacking I I did not think of it as a bunch of oneoff hacks that what what I thought of it was much more about what what is the way to ultimately Drive sustainable growth but it's uh overtime kind maybe more uh interpreted the way you described it but just just to jump in and say that that's a really good clarification so how did you actually initially frame it when you yeah I just I just said it's it's about you know looking at every single thing that you're doing and uh scrutinizing its impact on growth in the business and um particularly I think most marketers when I first moved to Silicon Valley most CEOs who were asking me to to help their companies they were saying we need help with awareness building and I'm getting introductions from Top BCS and so so much of of I think uh the way people were pushing growth was sort of like textbook marketing you know marketing textbook how to how to approach it and and startups just don't have the luxury to do all of those things and so you got to really focused on how do I acquire customers to an experience that's going to make them want to keep using this product and so I yeah maybe maybe I picked the wrong term in calling it growth hacking but I I think it at least open the conversation to to getting more people thinking about maybe maybe we should be thinking about growth in a different way than as it's traditionally taught in marketing courses in school is there another term you think you should have used do you always think back I should have called it this is there anything that you've had in your mind I don't I I I think um I think sometimes having something that's a little divisive is uh is almost better because it um it's too easy to just go completely unnoticed and and so yeah but I I was trying to put a a name on not just how I was approaching growth but seeing you know Facebook obviously had a very different approach to growth than than most companies LinkedIn Twitter there there was a handful of companies that were approaching it in the same way I had previously been approaching it and I just thought we need this thing needs a name and so sat down with a couple of friends came up with a name and it stuck and um but but yeah obviously from from day one it was pretty pretty divisive with different groups that's a fun story thanks for sharing that okay so talking about about growth and helping companies figure out how to grow so say you go to a company they're getting 40 say 2% 42% on the Shan Alis test and they're like okay cool let's start thinking about growth what's your first piece of advice to them to start when they're thinking about growth and then just broadly how do you approach helping them figure out how to grow ultimately it's it's about trying to get as many of the right people to that same state that we just talked about with the with the must have users so trying to get as many people to experience product in a way where they'd be very disappointed if they could no longer use the product and so that's not just acquisition which is how most companies think about you know initially it was awareness then maybe the more developed way was like oh let's at least focus on profitable acquisition but in my experience uh you know the the the hardest part is is is really sits inside the the product team so what how do you shape that first user experience so they actually use it in the right way and it's not so difficult that they give up and that ultimately like we we understand what makes it a must have product and then and then what we're trying to do is is build a yeah sounds kind of theoretical here but I I can go into the details on on how but build a flywheel around that must have values so step one would be would be understand it step two for me is is then figure out a metric that essentially captures units of that value being delivered and so when I think about a Northstar metric that's that's what I'm thinking about is is like some something that reflects how many people are coming in and experiencing that that product Market fit experience whatever that is and it's not just me telling them here's what your Northstar metric should be it's it's that ultimately ultimately the team needs to decide that together and then and then really just diagramming what are what are all of the different ways that we can grow that that Northstar metric so that's that's where you start to actually build I I call it like a value delivery engine but it's you know what is our what is our onboarding look like what's that that aha moment that activation what what does the engagement Loop look like is there any referral like trying to capture it as it is today and then from there thinking about where are the biggest opportunities for improvement so those High leverage opportunities and then ultimately you know starting to run experiments against those opportunities generally I I think I touched on it a little bit earlier but generally the sequence that I like to do is start with activation because that's that one's just so critical and it's easy to get lost in between uh especially for an early product the product team's so focused on the road map like we're two features away from not even needing not even needing marketing anymore this things going to take off and then and then the marketing team so focus on like bringing new people in with the the how do get those new people to a great first experience kind of falls through the cracks a lot of times so a lot of focus on on activation and then and then engagement and referral and getting getting the revenue model right and then once once each of those pieces are are working well then starting to really obsess on on the Channel Side one thing that I'll say like when I go in and and directly and involved with a company on the acquisition side I am thinking about my hypothesis on on the acquisition pretty early on because if I go into it I have no idea how we'll acquire those customers I'm not I'm not real confident I'll figure it out when I'm there so I want to have like two or three things that seem pretty viable as as ways to uh profitably acquire customers and knowing that once I get deep into it I'll probably come up with one or two more and I've got like five one of them's likely to work but I don't want to like just be under the pressure of having to come up with that once I come in if I if I don't at least see an angle from uh from from that before I get involved with a company what I'm hearing is when you come into a company and they're asking Sean how do we how do we figure out how to grow this thing you actually Focus first on activation onboarding and we're going to talk about all these things then after that like basically these are priority order for you yeah then it's flywheel engagement referral stuff to see if there's a way to drive that then Revenue how do we make money with this and how do we make sure we're doing this profitably and then only then just you start to go big on acquisition top of funnel growth yeah I may need to do some acquisition stuff before just to bring enough flow through but I'm not I'm not obsessing on like how scalable is this it's just like yeah let's get let's get enough people coming through that we can we can start to uh take take the slack out and part of it comes down to that the acquisition site is so competitive now that if you're not really efficient at converting and ret and monetizing customers you just you can't find scalable profitable customer acquisition channels this is fascinating because I think a lot of people are probably do the opposite start driving a bunch of growth to a product then we'll fix onboarding then we'll figure out how we're making money and referrals comes along there so I think this is really important for people to hear so again the reason you invest first and focus a lot on onboarding slash getting people activated is because that is very correlated to retention and this must have customer this like I'll be very disappointed customer yeah and that's and they're you're they're at highest risk risk of losing them at that point they they're probably a little skeptical about a promise that you put out there but they're intrigued enough to want to use it but until until you get them to that must have experience until until you kind of get them to that aha moment they they're they're at high risk of of being lost and so a lot of people focus on oh I better get their email address or their phone number but then then you're essentially having to reacquire them at that point so so to me it's just that if you can collapse that that time to Value I I give you a couple of like incredible examples of like what when we F so at logme in when we initially tried to grow the business I was stuck at being able to spend you know I couldn't spend more than $10,000 per month profitably trying to grow the business and uh then I dug into the data and I and I saw that 95% of the people signing up for log me in so um log me in free at the time free remote access for your computer and uh so you install software and you can control it from any other computer so 95% of the people signing up never once did a remote control session and so not surprisingly then I I had to get my kind of monetization off the 5% who did that was really limiting my ability to to find channels that worked and so our credit our CEO with this that I I shared the data with him and he basically told the product team we are putting a complete freeze on the product development road map so every single person from product engineering design and then also said to me stop trying to find new channels the the three of us on the marketing side are all going to focus on improving the signup to usage rate and so in three months we improved the signup to usage rate by a thousand per. so we went from only 5% of people using the product to 50% I went back tried the exact same channels that previously only scaled to to $10,000 a month now they scaled to a million dollars a month with a three-month payback on marketing dollars invested 80% of new users were coming in through word of mouth so there was this just like major inflection point by just focusing on activation this episode is brought to you by merge product leaders yes like you cringe when they hear the word integ ation they're not fun for you to scope build launch or maintain and Integrations probably aren't what led you to product work in the first place lucky for you the folks at merge are obsessed with Integrations their single API helps SAS companies launch over 200 product Integrations in weeks not quarters think of merge like plaid but for everything B2B SS organizations like ramp data and electric use merge to access their customers accounting data to reconcile bill payments file storage data to create searchable databases in their product or hris data to autoprovision and deprovision access for their customers employees and yes if you need AI ready data for your SAS product then merge is the fastest way to get it so want to solve your organization's integration dilemma once and for all book and attend a meeting at merge. dv/ Lenny and receive a $50 Amazon gift card that's merge. dv/ Lenny what do you find often Works in helping increase activation I know there there's a million things that people do and but I guess what what are like three or four things that you think people should definitely try to help improve activation and their onboarding conversion one of my favorite quotes is a quote from a guy ketering it was early like 100 years ago at GM running Innovation and and he says a problem well stated is a problem half solved and so I think a lot of it comes down to not the things you try but how you deep understand the problem that's preventing someone from getting using your product effectively and so yeah I I'll just give you one example we we had one channel like kind of after we made a lot of these changes and had already D driven a ton of improvement in uh in the log me in onboarding we found a demand generation channel that was like really cheap and the the economics looked great but at the just the download step we had a 90% drop off rate and so we AB tested a bunch of different things there to try to to try to improve that conversion rate and then finally um we we you know 10 plus tests not not able to improve it finally someone said well these people are registering why don't we just ask them why they signed up and didn't download the software and so we didn't want to do it intoo kind of a creepy way so we just said you know made it look like a note coming from customer service we had this this channel was sending 200,000 people a day so 20,000 20,000 people were converting to registering so we had um essentially yeah 20,000 people we could email and then uh 18,000 of them who didn't download and so we just asked hey notice you haven't had a chance to use the product yet it looked like it was coming from customer support what happened and the answer we got back and not a formal survey was oh I just I just this seemed too good to be true I didn't believe this was free I mentioned to you we were one of the first uh free prum SAS products out there and so people were skeptical when especially in a demand gen Channel where they hadn't sort of seen a radio or a TV advertisement from our competitor who was a premium only product these are people who are kind of discovering the category for the first time they were getting there and so our next test once we once we articulated what the problem was our next test gave us a 300% Improvement in the download rate which was a we gave them a choice download a trial of the paid version or download the free version put big graphical check mark next to the free version but when they saw we had a business model and a trial of a paid version the the free version was credible and so that that essentially made that channel work for us so I think again it's that combination of qualitative research looking at how others did it we we had this um Theory our previous company had been a game company that didn't require a download so initially we had this theory that maybe maybe just downloadable software can't be in the Millions of of new customers a month and so we're we're being unrealistic here but then we're like are there any counter examples to that and like no the instant Messengers are downloadable and they they have hundreds of millions of of customers so let's study their download and install process and see if we have any ideas that we could borrow from that so again some inspiration tried some of those things but it was a combination of just trying a bunch of different stuff that that ultimately uh led to there there was I would say there was one big gain it was was a bunch of small gains awesome okay so a few things for people to try if they're like hey how do I improve my activation rate how do I improve my conversion rate is just like drill further into what is stopping people from progressing like ask them why did you bounce here why did what did you think this was going to be why didn't you end up using this look for inspiration from other products I think people probably already know that you talked about earlier this idea of the positioning having a big impact of just like figuring out they want antivirus software let's make that very clear hey we've got the best antivirus software that's what we're here for so there's probably just like messaging that you find works a lot of times right I mean your two big levers on on driving a conversion are uh increase desire reduce friction and so yeah you definitely want to increase the right desire and then and so sometimes it can also just be reminding people along the way of what benefits you're going to get course in the case of log me in it was was probably the most complicated funnel I've ever seen because you you couldn't even get to the aha moment while you're sitting in front of the computer you had you had to actually go to a different computer and to use the service to remote control the computer you're in front of so it's not surprising that there was a like so many steps where we could lose people but we just weren't that intentional about designing each of those steps initially and it wasn't until we kind of thought through what why would we lose someone at this step and studying the data which which steps were we losing the most people at then deeply trying to contextualize why are we losing them there coming up with a a set of tests that we want to run and then having a good way of deciding which one to to test first and and ultimately uh you know ultimately focusing the tests on the areas where we're losing the most people the other element of this is coming up with an activation metric and aligning on like here's what we consider someone activated I know this is very dependent on the product but any advice or heuristic for how to help people decide this is our activated user I tend to start qualitatively so just like when do I think they've had a good enough experience with the product to to to really like like know it and so like in the case of log me in it was pretty easy if they didn't do a remote control session they they didn't use the prod there was no value along the way there and so and then at least try to see if there's a correlation to long-term retention of doing that causation is you you need to do some experimentation to to to prove causation at the very least I want to see that correlation but if if I start with you know two or three ideas of what it might be and then and then go and and study the data that that that can help you focus but again I don't think there's like necessarily one exact right answer of what is what is that aha moment there there might be two or three different things I think it's that intentionality about picking something that's experience-based and saying what is a likely experience that someone's going to get a a good enough Taste of this product I and then I I do see some companies that are like well the the activation moment should be they've used it a hundred times like there that that's G to correlate to long-term retention but you it's just not very actionable it's so far down the user experience so ideally if there's a way that I could get them there in in the the first session in the first day that's great and so it's sort of something that's value that can be experienced super early but I actually give you an example from the first company I worked on was a game company where I actually flipped it and basically instead of making a traditional funnel where they could play our games after they signed up I made our games the advertisements so basically we syndicated our games to 40,000 websites they started gameplay experience on the other website then they would get a message that they now have a qualifying score and uh if they if they register they they'll be in the drawing for the weekly cash prize and then and then we could pull them into multiplayer games on the site and so it was it's kind of the the strategy that YouTube used to grow but it was like two years before YouTube introduced the the approach yeah it feels like you basically created Zinga is what I'm hearing there okay so let's move further down the funnel so we've talked about activation onboarding the next phase that you focus on is basically some people call this growth Loops growth engines flywheels basically it's like the thing that helps your business grow and something I I'm curious if this resonates I found there's basically four ways to grow and usually one of these engines is responsible for almost all of your growth so what I've seen is basically it's you're going to go through sales you're going to grow through SEO you're going to go through verality word of mouth or paid growth does that resonate does that feel right yeah and I wouldn't say it's necessarily one of the other like I I think um I think uh bounce is a really interesting example where SEO was super important for for bounce and or is super important for bounce um so people who are essentially saying luggage storage Paris luggage store you know all most people when they're trying to find a place to store their luggage they're they're starting with Google but at the same time a huge percentage of the people who who use bounce are dragging their their bag down a down a street over you know over cobblestones in Paris and then they then they pass a sign that says store your bag here for five dollar a day and it's like oh Nob brainer and and so 10,000 Partners around the world means that there's there's a lot of people kind of in the right situation on the demand gen side so yeah one one would be I actually think of kind of like um I'm not sure how it would map to this but like demand generation versus demand harvesting and so you know one of those examples would be a demand uh generation example like when you see the signs when you're passing it's like high context right right place and then obviously the demand harvesting would be anyone who's who's Googling and so that there's paid they do paid search and and organic search there interesting uh I don't see that like that sign approach work often but I definitely have seen it work like Yelp I think grew in a lot of ways of just like little Yelp stickers and all the restaurants door Dash I think probably gws through that yeah I think every every business could be a little bit different but um for bounce it makes sense that that would be a really really good uhow opportunity for them how do you help a business figure out which area to bet on like whether they should go paid whether they should go SEO whether they should hire sales that's sales probably an easier one of like probably B2B you're probably going to have to be a sales steam I guess just to help them pick like here's where you have a big opportunity again it kind of comes down like as I'm as I'm going into it I'm I'm thinking what what are the realistic customer acquisition angles for this business and I want to have you know ideally two or three that I'm I'm coming into it with but it's gonna you know obviously like like Dropbox is a classic one of like oh man this this product user get user is going to be just like a a classic there there's you know file share built into it folder collaboration there's there's so many pieces of it that that cross from one user to the next but but interestingly it was fairly similar to log me in in some senses as kind of solving to two business that are solving similar problems in in in different ways where log me in we grew almost entirely off of paid search and and part of it again is that uh for for us we had we had a um competitor that was spending you know tens of millions of dollars a month creating the category with a premium only product um through radio and TV advertising go to my PC they were just creating all this lat and demand and so it just made sense for us to to disrupt them with a premium service and and to insert ourselves in in the flow of someone like what was that thing that I heard about on the TV commercial and now they go and they Google it and you know same thing but free so we weren't really pushing for differentiation but just really you know trying to harvest that so I couldn't do that at Dropbox like no one was looking for Dropbox when I went there and so you know we we we tried a little bit with with search to see can we can we make it work on you know cloud storage or backup or kind of going to some of these like traditional category like cloud storage wasn't even a traditional category at that point but backup was and it was just like the it was fairly expensive um to to be and there was just not that much demand there that way and so it just made more sense to to focus on the user get user Loops at at Dropbox so I think I think basically for for each business it's um it's just thinking about what's what's unique for that that business that um is is going to open up Channel opportunities and everyone's going to be a little bit I think jaded from whatever the last thing that worked really well they're going to think they can they can apply in the next business but um after enough times myself I've just you know you I tend to get the most inspiration by just talking to to customers and and finding out how how did they find it how do they typically find something like that and and that starts to give me some ideas as well I think that last point is really powerful and I'm just writing it down you said so essentially one of your tactics is talking to users asking them how did you find this product and how do you normally find products like this was that the second question I I think that's it's like such a it's like similar to your Shan house test it's such a simple question but it's so powerful because how else will people find your product it's they go to a place to find stuff like this yeah and like they I Google I search Google for folder sharing like like there's so much there that I think to skip over I think the reason that you don't actually kind of hear people taking the obvious route there A lot of times is because um and I I I used to be in the same thing that people tend to be either over indexed on qualitative or over indexed on on quantitative so so kind of like you know analytics I'm going to get all my answers from from testing and analytics or I'm going to get all my ANS from traditional customer research and I I was very much in that initial camp for the first five years of my career I'm just you know measure everything and test the heck out of things and and find stuff that works but I had a a VC who um was our lead VC at at log me in who just said when was the last time you talked to a customer just like pushed me to survey and talk to customers all the time and and um at first I was like yeah gave the smartass answer I don't care what they say I care what they do and that's that's what you he's like no you got to talk to him then then just to appease him I would I would try to have a conversation every day because he was in our office a lot and so I could say hey yeah I talked to a customer today when when he would ask me and then but I started finding that my experiments were so much better the more I talked to customers and and you know eventually I I became very much like the blend of of qualitative and quantitative research leads to much better tests that is another Amazing Story and insight it's so interesting that people sometimes think of you as growth hacker Guy experiments data when most of the advice you've been sharing so far is very qualitative driven very survey driven talking to customers driven yeah and it's it's just really hard to run good experiments when you can't deeply contextualize what's going on I love this by the way I I don't know if I knew this so you helped develop the Dropbox referral program I was there at the time I basically when I um even when I first started talking with with Drew before I came in I was like I think the way we're going to grow this business is by leveraging the really passionate customer base and that's what we need to double down on and uh and we we had tried a a similar kind of referral program at zne and my friend who actually started uh who started ring um Jamie simonoff had previously uh had a company called phone tag like way way before ring and he had actually done a lot of the testing on you know kind of double-sided referral programs and and the the having having incentive on both sides and he he found that that worked the best and so I between what we had tested at zne and those conversations with him I hadn't actually seen PayPal yet at that point what they were doing but yeah that that was kind of like yeah it it seems like a referral program where we have incentives on both sides is the best way to go interestingly you know six months before I was at Dropbox I was at log me in and um I really thought about having incentivized referrals that log me in but 80% of our new users were coming in through word of mouth and you know I had a 100 million devices connected in on our on our system and I I was just so afraid of breaking this growth engine by adding an incentive that I I I didn't want to risk it but at Dropbox it was so early that that yeah I I I would still say like no experiment is one person it it happened to be when I was there I had some insights that I brought in but but ultimately the guy who built it um was actually an intern named Albert KNE and uh he he end up dropping I think out of MIT to to stay with the Dropbox for for a few years after that but um but yeah we he was kind of my right-hand guy to uh collaborating on growth day-to-day wow I would say drawbox as referral program and the PayPal referral program as you mentioned are the two most legendary studied copied out and unfortunately like I think that what they don't realize is that before the referral program dropbo had amazing referral rate you know like they're they're kind of companies that are trying to copy it are are like why isn't anyone talking about our product let's add a referral program with incentives like um to to me I think it's it's a great accelerate when it's already working but it it can't it can't fix it if if people don't want to don't want to talk about your product that's an awesome point and something I was just going to ask about and just kind of coming back to this topic of growing engagement growing referrals as a as a growth mechanism what do you look for to tell you that there's an opportunity there and I'll just answer it partly is I've seen exactly what you just said which is you need to already have strong word of mouth growth because referrals kind of sits on that and gives you a little more incentive to share so maybe do you agree with that not agreeing that any other advice on helping figure out is there some kind of loop here that we can build well one thing I will say is like fremium when we first started with it as I said like we we were one of the first with it um the so it it took me a while figure out exactly how fremium worked but to me uh fremium to to so having a free and a premium version of of your product to really work in any business it it needs to be that your free product is so good that people naturally have Word of Mouth around that product and then to be economically viable you have to have a a premium product that's that's better enough and differentiated enough that people are gonna upgrade to the the premium product but I think a lot of times people are so worried about the second part that they make the free version not very good and then they're surprised when when word of mouth isn't very strong there so I think you you have to essentially have two distinct products that are are great on their own so that that would be the one piece and and so but then obviously companies that have any kind of uh collaborative layer to them are going to benefit from or are going to be more likely to work well with referral and then I think on the engagement side a lot of it comes down to just the nature of the product like you Airbnb you're not going to use it every day like unless you're unless you're you know like a vagrant or something maybe but and then you won't have money to pay for it but the um so so there there's kind of a a natural usage cycle to products and and you want to be able to to maximize against that cycle and I think that's where I was saying coming back to the the the hooked model I think is is a really good way to help to have a framework to think about how do I improve engagement um one good counter example to that though of the kind of natural the natural um frequency of using a product is is Facebook when they change their Northstar metric from monthly active users to daily active users I think again just just having you know what gets measured gets managed like um once once Facebook was on a daily active user goal the team suddenly had a lot more incentive to think about how how do I bring people back every day and use this product where when it was monthly active users they they kind of only got credit for that person for for using once in that month and even if they used 10 times they didn't get 10 times a credit it was just like a oh that's cool too but they they weren't sort of measured on that and so I think it was sort of a random decision for Mark Zuckerberg to move from a uh monthly active to a daily active because they hit one billion monthly active users and they're like okay let's go for one billion daily active users but it it had a a really big impact on making that product way more addictive to the point where obviously they ended up in Congress for uh you know or get a lot of push back I'm not sure they got went Congress for that but they got a lot of push back for having a product that's maybe too addictive and the same thing carrying into Instagram and some of the other um meta products or basically anything that is is highly engaging um so so I I I do think you know the right incentives can actually help a team to to to focus on it and um and then but there's going to be sort of a natural usage cycle to any product as well I'm glad you mentioned northst metrics I actually have a post will link to the show notes where I collected the Northstar metrics of 30 different companies to give you some inspiration I know this is a deep topic of its own but just when someone is trying to pick their own Northstar metric which I 1,00% agree informs so much about how your company operates it basically focuses everyone's incentives to let's drive this thing and that changes so much of what you're building uh any just like bullet point piece of advice for helping you pick your Northstar metric I start with uh the value that's uncovered through the the chanell's test so with a company I'll say okay this is this is what the must-have value is according to our most passionate customers and we want to think about a metric that reflects us delivering that value and then I'll then I'll give them kind of a framework of ways to to think about a Northstar metric but I think it's really important for it to be a a Time capped group conversation and if if you give a team 30 days they'll take 30 days if you give them six months they'll take six months but so I think generally a team can come up with a pretty good Northstar metric after 30 minutes if uh if they have like the right kind of raw ingredients and and a checklist of what's important in a Northstar metric like something that it's not a ratio something that can be up and to the right over time so you can keep managing it and and uh you know feeling good it should it should correlate to revenue growth but not necessarily like re Revenue shouldn't be the shouldn't be the Northstar metric but you know as you as you grow value across your customer base you should be able to grow Revenue at the at at the same rate and so there's there's some other things but I think that would be the most important is that it's uh something that could be up and to the right over time and reflects value that you're delivering to customers awesome and I was gonna ask about Revenue in your opinion there and so your devices don't make revenue or Northstar metric no I like even Amazon um and again like this is just what I know of Amazon's as being but um monthly purchase um but you know someone else might say Amazon no Amazon's is you know gmv or something but like I think monthly purchases is is great because it it like um maps to Value uh that people are getting from Amazon and so um you know even if I spend say,000 on a TV set with with Amazon versus you know $3 on a you know $10 on an electric toothbrush um Amazon from the consumer persp perspective delivered the same value I needed something Amazon helped me find that thing and so units of value from the customer perspective I think is more important than uh than you know overall Revenue but clearly clearly you know with Amazon focusing on driving more monthly purchases at least on their on their store side of the business that has helped them become one of most valuable companies in the world so I I think you know focusing on value is is all revenue should be up a byproduct of doing things right it shouldn't it shouldn't kind of guide your day-to-day actions to make this even more concrete for people are there some nordstar Metric examples you could share that you've seen that are good like say from Eventbrite or Dropbox or any companies you've worked with and I'll share one real quick as you're thinking about it uh at Airbnb was or AR STAR metric was Knight's book and so it similar to Amazon it's not like the money airb made from bookings but it's like nights booked and it it was really and basically every experiment ran is like is this increasing nights is decreasing and so that's that's like a really good Marketplace one Uber obviously you know weekly rides I I'm always surprised with the with the Airbnb that it's there's not a kind of time piece on it like like the weekly rides that you have with Uber but maybe maybe it's because such an infrequent use case on travel that it doesn't make sense to to focus on yeah yeah why why is the time frame important to you why have you why do you encourage that um just you know daily active users you saw the difference between monthly active users and daily active users could could change Behavior a lot of Facebook um it it uh gives you like a quantifiable way if you're just kind of taking an aggregate number over time it always looks like it's going up so it's an engagement element how often are they engaging yeah okay any others any others real quick yeah I mean I didn't really think about Northstar metrics when I was at Dropbox and Eventbrite you know like the the term itself but I was thinking about what is what is a valuable experience with Dropbox and how do I get people to have that more time but like I don't even know what they go with today but maybe like you know files files in drop off files access might be better than just files you know hosted um and then probably for event right again I would say like uh weekly tickets or some something like you could say weekly events but then you have events that don't sell any tickets where weekly tickets would be more likely to reflect events are going to be happy if they're selling tickets and um yeah okay Sean we've gone through so much stuff I I have I'm trying to limit how many more questions we get through just so that we don't we're going long we're going long which is amazing I think there's so much value here that we're collecting for folks so let me just ask maybe a couple more quick questions one is actually from Andrew Chen who uh is currently partner at a16z he was wrote about growth for the longest time I think he helped popularize growth hacking for better worse with this article and it being the future of VP what is it growth hacking is the new VPO marketing right is a title so he actually had a question for you that he shared with me his question is growth strategies have changed a lot over the past decade what is the biggest difference now versus when you first started working on growth when I first started just being data driven on customer acquisition was enough to win and you know being test and data driven on customer acquisition position all the other companies were were like CPM focused and and you know to so like we we could we could do really well just just with lots of testing and and some creativity and how it all worked but um that over time as as now I would say most marketers are very most online marketers are very like data and test driven they they know they need to lots of testing and so to be competitive today you actually have to be able to be super efficient at all parts of the business so you know again like how how how you convert retain monetize and and that's when it gets hard getting getting a marketing team to be to to be data and test driven is pretty easy once you start getting into activation and referral and engagement and retention now you're talking about the overlap between marketing product it's B2B bringing sales in there customer success and those teams are not used to working together and so it's it's really hard to to to drive the collaboration that's needed to to have an effective testing program across the entire growth engine and and that's pretty much any business that's been successful with it implemented it super early in the business and so very very few later stage companies have have been able to make much progress in in replicating that type of approach it's just gotten harder basically things are just getting harder it's gotten harder but I but I but I I think it's um I think it's possible it's what I obsess about all the time is how how how do you how do you get cross functional teams working together on growth now and it's still a huge Advantage when you can pull it off okay totally unrelated question going in completely different direction as we close out or chat so you uh came up with ice the very popular way of prioritizing work uh which is crazy I did not know that until I started prepping for this conversation what's your thoughts on Rice the uh intercom uh version of ice where R stands for reach I believe uh thoughts so I think I think it's an unnecessary addition but maybe it's I'm just being protective of my original idea that that the eye in ice is impact and it's essentially saying best case scenario how much impact could we get from this and reach is a is a super important part of impact and so like I think it's already factored in the eye in ice and so I think if there's anything that I would be accused of it would be being oversimplifying things and um and there's I'm not saying them but there's there's there's a lot of people who approach things with there's got to be a more complex way to approach this and and that's just not me and so yeah I you know more testing is better like that's no it doesn't just work like that I mean better test tests are better than bad tests but just if you have to hold yourself accountable to anything more testing would be better and so I think I one just quick note on ice is that um you know in order to be able to effectively run a high velocity testing program you need to be able to Source ideas from across the company and that's why I came up with ice that if you if you're having people submit ideas and you can't tell them why their idea was not chosen they're just going to get upset and and you're gonna waste a lot of time but if you have a systematic way of being able to compare ideas it's it's more likely that people be able to get it and they'll be able to come up with better ideas I love the way you think Sean I have a post on pration where I basically just make the same argument that there's all these fancy complicated ways to prioritize in the end it's just impact confidence and effort and it really works and rarely is more work necessary on the other hand I do also have a guest post called d by these two guys called detail rice which actually I think is a really good point where sometimes it's worth spending like 30 minutes per idea to just really estimate how long will it take to to avoid doing things that are just going to not work and very unlikely work basically doing this reach piece uh and spending the time do right and I think there's a lot of good value there yeah and that's um what I think is going to be really interesting is that uh over time I think AI is going to is going to to actually change our ability to model out potential outcomes on experiments and start to whether whether it's a more informed way of doing ice or replaces ice that that ultimately you know Pro probability of outcomes is is something that AI will be pretty good at well amazing seg wait to the final question the actually final question is I wanted to ask you about any ways you've been using AI or ways you think AI will impact the work you're doing or other folks are doing and maybe just answered it but oh you tell me no I'll uh I'll touch on a couple um one is that uh like probably the funnest way that I'm using it today like obviously I've done it for coming up with experiment ideas but the the the funnest way I personally use it is I get a lot of people asking me for advice and I don't have very much time to answer you know with thoughtful answers to people and so almost every question that I get I I go to chat GPT and say how would sea Ellis answer this and gives me an initial draft to to like make a couple of tweaks and and um definitely allows me to answer a lot more so it helps to have a a book that's indexed in there and lots of writing that is so funny and is that the question that's the simp as simple as the prompt is how would sea Ellis answer yeah because I know a lot of times that'll say sea Ellis author of hacking growth D believes that you know and then and then it'll like pull that part out in the answer oh my God it's like you're one step away from a Chrome extension or or something that just automatically plugs that into your yeah yeah exactly I I can even start to have my um you know personal system maybe start to answer some of those questions as me um but I I'm a little bit afraid to send something without without reviewing it first because sometimes there's sometimes there's stuff that's pretty different from how I would answer it but longer term I actually think you know as I said I think the cross functional challenge to growth is is a thing that holds a lot of companies back from being able to implement this a bit later mostly like you know product team don't want to be get direction from marketing teams don't want to get D marketing teams don't want to get direction from product teams and and you know maybe a growth layer can can help to do these things but um I I find that you know like if if AI is essentially saying you're underperforming in this area of your business you should drive some experiments in this area it's just it's a lot harder to kind of let ego get in the way when when it's kind of dispassionate recommendations from a system and so I actually think I think the ability to come up with with great experiments is going to keep keep growing with AI and you know identifying opportunities and then obviously like the analytical AI side of things is going to be really exciting in terms of uh being a I do find with most companies once once we get a real high velocity of experiments going the bottleneck ends up happening more on the analysis side and and I think AI will help a lot with that as well super cool these are awesome examples okay Sean is there anything else you wanted to share our leave listeners with before we get to our very exciting light Grand which we'll go through real fast because we've gone very long and I want to let you go yeah as I as I've gone through and done a lot of uh workshops and programs with companies I keep coming back to this advice that I that I heard from uh guy OLG yakubov which is um it it often comes down to asking the right question at the right time in in how you figure things out and he's a he's a for former data scientist from from meta and so you know where he basically boils data science down to learning how to ask the right questions and so um I actually have a a course with him called go practice. where that's that's really the big benefit of the course is to learn how to ask the right questions and yeah you learn how to querium and amplitude but but more importantly uh to being able to ask the right question I think it's it's kind of cool to hear that from a a data scientist from meta that the importance of that but every time I'm going through exercises in my in my work workshops it almost always comes down to people who who aren't able to come up with the right or or a good answer for a challenging in a business it's because they're not asking the obvious question and and soon as they have like why aren't users downloading the software let's let's let's just ask them that question like that would be one example from my my workshop you know who considers the product the must have that part of getting you know to uh figuring out the must must have kind of the benefit that that then allows you to to to hone in on product Market fit and so yeah right questions right time I think is a is a really important uh way to think about uh growth and even getting to product Market fit I love this advice because I think it gives us a glimpse into how your brain has developed these really simple seemingly simple ideas that end up being really powerful and it feels like the advice is just think about the question you need to ask because that'll get you just something that a lot of people just kind of under underthink or don't there like think think maybe too simple like yeah they just jumped right into the solution side of things where they're where they're not really trying to understand what's going on yeah amazing okay well with that Sean we've reached our very exciting lightning round are you ready I am all right our first question is what are two or three books you've recommended most to other people increasingly I I'm recommending a book called presenting to win that's been around forever but um it really helped me with my my presenting and so of course when I'm out traveling I'm often sharing the stage with other speakers and uh and yeah I'd like to recommend that one to them i' I've already talked about um Muriel's hooked I recommend that always and we'll stick with two that that that's good too within within presenting to win is there one tip that you sticks with you of like here's something that really helped me be a better presenter ultimately like confidence in presenting comes down to having very well organized information that you're going to present and when you organize it correctly you are much more likely to deliver it with confidence and so he basically says if I had a presentation to do and I had an hour to present I'd spend 55 minutes creating the right presentation and then five minutes practicing it and uh but yeah there's a lot more to it but that wow yeah amazing okay we'll link to that book in the show notes do you have a favorite recent movie or TV show really enjoyed yeah so I've been binging the Olympics I love that just you know watching people who like work their ass off for years and then maybe have 30 seconds to do the thing that they worked hard for so Olympics have been awesome and then um and then the movie um I actually just saw Blackberry I don't know if you've seen that oh the uh like the story of the Blackberry yeah I mean obviously like we all kind of know the story but it was uh so really I mean it's it's a classic example of like product Market fit and then not actually it's probably even a counter example to the dangers of the uh how would you feel if you could no longer use this product pretty sure most people would have said on BlackBerry it's the keyboard and you know until until iPhone came along you know the keyboard was super important and then suddenly it wasn't but yeah it's also also uh interesting on like Egos and and other things that like everybody's good and friendly in the beginning and then uh and then egos take over and things get a lot harder later on that was actually a really good movie uh there's also an amazing movie called Tetris for some reason I think of these two together about the story of Tetris and there's like some it's like a similar parallels to those two movies awesome I'll have to see that one next question do you have a favorite product you've recently discovered that you really love I forget the name of it but it's it's I think oh it's called pack gear hanging suitcase and I uh basically like I've done almost 100,000 miles in travel this year and um I have another schedule for next week and I I love it because it basically has all my clothes folded in this like little little insert that goes into my suitcase and then I just pull it out and hang it up and um just makes makes travel way easier it's called the pack gear suitcase uh pack gear hanging suitcase organizer so cool gonna check that out uh two more questions do you have a favorite life motto that you often come back to that you find useful in work or in life maybe share with friends and family sometimes focus on on reput and learning over earnings um has just served me super well that um and I'll give you an example I had I had two companies when I was doing a lot of this early interim stuff you know 10 plus years ago and um I had two of them where I talked to the the the founders afterwards and I could tell they weren't like that stoked on my contributions and I offered a full refund to both of them with a thought that like I have this reputation that that's I I randomly pulled the number and said my reputation is worth $5 million why would I possibly you know mortgage that reputation for $20,000 and so you know one of them I I gave the the the check back to them and he he was happy to take it and then uh but he had said oh you can make it up to me you could you know like I you don't have to give me the check just make it up to me by continuing to help me for an unlimited amount of time going forward I was like take the check and then the other one said no no like I'm I'm actually really happy with what you did we're we're fine but the two VCS who had made those introductions were the first two to give me term sheets when I went out to raise money for my company and the uh pre-money ultimately ended up being valued at more than double what I had put my personal reputation at so I yeah I think the yeah unfortunately the company didn't do that well itself because of the elusive product Market fit challenges but yeah the the learning there of you know just F focus on learning and reputation reputation opened the door to more and more learning and as I got more learning the reputation grew and so um yeah there's a really good Corel there with customer support like if someone just hates your product and wants a refund just give it give them over ref fun and let them move on versus being upset yeah absolutely I love that final question you mentioned to me when we before we started recording that you were maybe indirectly responsible for Tik tok's success uh maybe share that story yeah I mean I I don't want to overstate it but I uh yeah my trip around the world that I did um uh three months ago I think I wrapped it up I met with the original founding growth team at Tik Tok they're based in Singapore and they had I can't remember what the previous product was called but they they started with the previous product and then when Tik Tok came they they they were in place to be the initial growth team for for Tik Tok and they basically said all the early stuff we did to grow Tik Tok was based on your writing so that was that was before the book came out so it's a lot of just blogging that I had done but um but was really really cool to get that feedback that um that yeah I I I've always said I I have some really good wins I have a lot of unicorns that I I helped but none of the really really big guys and then uh to hear that it it felt really good to know that I I played some kind of role in Tik Tok of course almost the same week they told me that that was uh you know Congress having Tik Tok band conversations so it it was it was good and at the same time knowing that maybe if they hadn't read my stuff Congress wouldn't be wasting their time on Tik Tok bands oh man Bittersweet I hope they don't pull you into some hearings Sean this was incredible this was everything I was hoping it' be I feel like we collected so much wisdom here for folks to help them figure out product Market fit find product Market fit iterate grow their products so happy we did this two final questions where can folks find stuff that you're up to if they want to learn more and maybe work with you in various ways and how can listeners be useful to you awesome yeah um so Sean ellis. is the website where I kind of link to all the things that I'm doing and so that would be one place where you know and there's contact forms on there if anyone wants to reach out obviously LinkedIn people can can contact me there uh and then I did mention go practice so go practice. um really cool way to learn growth through a simulated environment of being able to to try to grow products so check out uh check out go practice and um if you go to Shaun ellis. me when this comes out I'll I'll put a special offer on there for Lenny's listeners so you can save some money and there's also a llm AI kind of uh I wasn't directly involved on that one but there's yeah there's there's some other really cool stuff that Oleg and the team are doing I've been data driven product management and uh and the uh user growth programs are the ones that I helped with awesome and then for folks if they're wondering do you do advising how do you help with comp how do you work with companies in case they're like hey I need Sean yeah I mean so the The Sweet Spot for me on companies that I go Hands-On with are ideally pretty early uh just after they get to product Market fit and now you know how to measure it so like if you're kind of pre-scale but you're you're seeing that 40% or even if you're a bit earlier than that we can start talking earlier but to me that's my favorite time to get in there build it right from the beginning it's so hard to retroactively do these things and uh and I'll I'll go in for 3 to six months and and I'm I'm Allin full-time one of the team trying to trying to really help build Traction in the business I do I do one of those every every maybe year or uh maybe every year or two because I I purposely burn myself out and then have fun doing more uh lecturing and workshops and stuff awesome well you might get a flood of requests uh after this comes out hope you're ready uh Sean thank you so much for being here awesome thank you Lenny I really appreciate you having me on bye everyone thank you so much for listening if you found this valuable you can subscribe to the show on Apple podcast Spotify or your favorite podcast app also please consider giving us a rating or leaving a review as that really helps other listeners find the podcast you can find all past episodes or learn more about the show at Lenny podcast.com see you in the next episode