Generative research != confirming your priors

One of the fun parts of starting a new job is getting to ask “Why” a thousand times a day.¹  

Why is this our strategy? Why is this our target customer? Why choose this problem? Why do we believe this market is big enough? Why do we believe our solution is differentiated?

Lean Startup Canvas is a handy tool to remind ourselves of the kinds of questions to ask.

If there are questions about revenue model, market size, or similar “what needs to be true for this to be big enough, and how confident are we that those assumptions can become true one day”, I like to build a spreadsheet model that lays out all of those assumptions, and then interrogate each number to see which we think are likely to be true and which we need to de-risk.²

If there are questions about what customer we want to target, what Jobs To Be Done we want them to hire us for, what would make us a differentiated solution, I like to turn to generative research.

This starts with recruiting 6-10 people who 1) have no prior relationship with me or anyone at the company, 2) are likely to represent the kind of customer we would focus on.

Then, we should interview them with open ended questions that stay within the context of our focus area. And don’t sleep on the power of ethnographic research, diary studies, card sorting exercises, etc. Let them take us to where they want to go, what’s on their mind, what feelings do they have.

What we’re hunting for are flare ups of emotion that can reveal a worthy pain point. These are subtle — an edge to their voice that lasts half a sentence, a big sigh or rolling of the eyes, a shaking of the head. We’re fully present during these interviews, because these moments are easy to miss.

When we see those moments, we pounce. “Can you say more about that?” “How does that make you feel?” “How do you deal with that today?”

These are the nuggets of gold that can lead us to a problem worthy of Product Market Fit, if we can validate that the problem is 1) wide spread enough, 2) solvable in a differentiated way, and 3) can be monetized sufficiently.

What we absolutely must not do is pretend we’re doing generative research but actually we’re user testing solutions we already have in mind. Evaluative user testing is a great and under-utilized tool, but it’s not the same thing.

If we go looking for proof that we are right, we’re pretty likely to find it because human brains are great at confirmation bias.

Don’t: “What if I presented you with this solution, that could do x, y, z. Would you use it?”

Do: “You mentioned [problem hypothesis]. Tell me about the last time [problem hypothesis] happened. What was it like? What did you do? How did it make you feel?”

We’ll get to the solution design soon enough, but until we have confidence and clarity about what problem we want to solve, any solutions we do build are hard to evaluate.³

The Workshop

This is a newsletter-only section where I share a half-baked idea in hopes that y’all who are smarter than me can work it out with me.

Back in 2016, when I was running Product and Engineering at Homepolish — a managed marketplace for interior design services — we lost product market fit.

Our customer problem was “I want the benefits of an interior designer, but I don’t have the $100k+ budget that boutique interior design firms want to take me on as a client.”

Our solution was to hire earlier-in-their-career designers, who were willing to work for $30-$90/hr to help build up their portfolios and client base, and matching them clients who had $5k-$50k budgets. Just like the boutique firms, we’d come into your home, sit with you, show you pictures of things, and stage your home at the end so that you got that “HGTV moment”.

And then Havenly and a slew of copycats came out. For $150 bucks total, you could get a Pinterest board of ideas, a floor plan, and 2 zoom calls. It took out the entire bottom of the market from under us. Because what customers actually wanted from us was “I want the reassurance and confidence that I would get from working with an interior designer, but for as little money as possible.”

This keeps coming back to me when I think about other times when people are having to spend a lot of money doing something they’ve never done before.

Wedding planning is the one I’ve thought about the most, but Nikhil Krishnan’s post yesterday on the rise of concierge medicine got me thinking, too. Healthcare obviously requires real doctors with real training and not some crappy metaphorical Pinterest board + AI chatbot, and yet, what is peace of mind worth to the parent of a sick kid? Food for thought.

1  Sometimes the answer comes with data, and sometimes it’s “based on conviction”. Hopefully, the “conviction” is coupled with an intent to validate it. Otherwise, you’re in for a rough time, or you work for a genius.

2  We’ve talked about this before, but for the new people here: open up a spreadsheet, and build a simplified funnel that’s probably 3-6 rows deep and roughly approximates your business. For example, (traffic) x (% who sign up) x (% who find what they are looking for) x ($ per conversion) = revenue. Then put in your assumptions for what this could look like in a mature, fully optimized state, say 3 years from now. If the revenue number at the bottom isn’t big enough to meet company goals, start changing the numbers above until it is big enough, and then start asking yourself if the numbers within each row seem easy to attain, possible to attain, or risky to attain. Then start designing minimum viable experiments to de-risk these assumptions.

3  Like, how can we know if the solution we built is any good, if we don’t know what problem we’re trying to solve? This is where the “build me a faster horse” quote from Ford makes its entrance.