LLMs turn product development upside down

Today begins my 7th week as CPO at ClosedLoop.ai, and I wanted to share an early insight about building an LLM-based product.

I’m going to keep this generic as we haven’t launched yet, and I’m sharing these in a personal capacity, and not on behalf of the company.

In the history of software development, we always started from a blank slate, and created a set of features that became a product business¹ . 0 → 1 → 100.

Conversational AI products start with infinite capabilities, of varying and sometimes unpredictable quality, and we have to tame it and sculpt it down to a set of features that then become a product business. ∞ → 1 → 100.

The question becomes, do you want your users to be able to type in anything and get a response? What does that mean for your marketing and value proposition? What does that mean for your trust & safety? How do you build customer journey maps and conversion funnels for infinite use cases?

The answer you might reach for is “we’ll very narrowly define ourselves around a specific user and use case. For everything else, we’ll refuse to answer the user’s prompt.” Basically, trying to force the technology into the traditional 0 → 1 → 100 product development approach.

That might be the right answer. I’m sure it is for some.

Or, you might reach for the opposite answer: “Infinite use cases means infinite business opportunity.” Which means competing with Big Tech on their turf.² Good luck with that.

I’m not going to give away what I think the answer is, other than that I think it lies in between these two poles, and it requires innovation in how we do product management and product design as much as it requires innovation in what we do.

Exciting times! (Also, I’m hiring a Lead or Principal Product Manager and a Staff Product Designer, if you’re willing to relocate to Austin.)

The Workshop

This is a newsletter-only section where I share a half-baked idea in hopes that y’all who are smarter than me can work it out with me.

My Google theory is that Google is actually happy that generative AI-based answers to search is cannibalizing organic search and SEO traffic, because it will force companies to buy AdWords to make up for the lost traffic.

The decline of SEO as a channel is something I’ve written about before, but we’ve had some new folks join (hi!) since the last time, so here’s a quick refresher:

Google is an aggregator, matching people who want to find things (facts, products, etc) with the websites that offer them. Google won search because their PageRank algorithm was way, way better than the old Yahoo approach.

Because they won the battle for user demand, websites bent over backwards to get in front of that demand.

So an entire massive industry sprang up called SEO, to make it even easier for Google to find and rank your website, so that your site would get matched to people more often.

And then Google built one of the best business models ever in the history of humanity, selling ads to businesses that wanted more than organic search was delivering.

But Google doesn’t actually need websites to satisfy a lot of their search demand. They just need to generate good answers, which generative AI is pretty good at, now that the AI has gobbled up the entire internet.

A few large companies that constantly create new content, most famously Reddit but also many big media sites, already get this and are using their robots.txt files to block Google until Google pays them for access (in order to feed the Google’s AI with fresh content).

But most companies won’t be big enough to negotiate with Google. They remain dependent on Google for organic traffic, and their content is commodified enough that Google doesn’t really care if they block Google or not.

Which is a long way of saying, Google’s AI answers will be “good enough” to satisfy a decent number of searches, and Google’s answer to those companies who are negatively affected is to push them to AdWords. The old “Don’t be evil” is now “pay to play”.

1  “Great UX” was this puzzle that designers were asked to solve: what was the problem we were trying to help the user solve, and what was the best UX for solving that problem. We pulled out our tools like user research, customer journey mapping, usability testing with design prototypes, etc to unearth what JTBDs existing within users and present to them a compelling product that met their needs.

2  Let’s set aside the category of companies that is building their own LLMs from scratch. Now you’re really competing against Big Tech, except this is a technology that doesn’t have defensive moats intrinsically. The only differentiator is how much money you have to spend.