I've seen many product managers starting coding with Ai. Every body is soon builder. But I'm starting to think the building prototypes is the end game.

I'm in process of creating a new value prop around agents in business. Every day I remind myself: don't build. I know I could build the product. I've even built something like this in the past. What I don't know is the real market.

I must first gather some facts. But what are facts anyhow?

I know the question sounds silly. What are facts? Easy enough. Facts are the things that are true. You just go out and find out.

Clarity and focus have always been the scarce resource

In many ways, building the more punchy and more right product has always been the game. Sure some have built quicker and gained competitive advantage. Still, those who had the "better" product have come out ahead. And by better, I mean more suitable for the job at hand.

This knowledge of the suitability has always been scarce. We don't quite know. We don't spend the effort to listen and to understand. We take our best guess and we build.

Truth is subjective and segmented.

The trouble is that people do not think alike. Every has their own preference. They value different factors. And people don't even want the same things. So: in mathematics, there are truths and axioms. In real life lived by people, these facts start to become muddy. At best, you find segments and groups of people who think enough alike.

There is no absolute truths in lived life. Just shades of preference. Recurring occurrences where you might help with your product. Some of those moments you win. Some you lose. The more you win, the better the product.

Take an educated guess. You will be somewhat off. If you are lucky, you are directionally right. The trouble still is knowing which action to take to improve.

I don't trust anybody anymore who claims to know absolutely.

To fully leverage generative Ai, you need to turn outward, not in

The interesting bit about Ai is that it creates plausible solutions. The better our LLMs get, the more likeable the products are. But are they "true"?

By true, I mean that those products solve a real job for real customers. Ai thinks it did a marvellous piece of art. You might think too. How do you know?

You and Ai stuck with the same dilemma: is this true and real innovation? No answer to be found inside the company.

Evals and ai-driven iteration

Only the customers can really judge what is value and what is phantasy.

So: you ultimately somehow have to make the research on what are your customers, the segments and what they want next.

The nice bit with LLMs and generative Ai is that you can now turn large masses of unstructured information into concepts with relative speed. You can define concepts. These concepts are synthetic and generative. But you can define them. Then you can turn those into evaluations you can run with Ai. Basically, an evaluation (eval in short) takes an Ai output and scores it against criteria.

Next up, you check the Ralph Loop (google for it). Then you can let your Ai iterate to try improve your product against the criteria.

If you don't have evals, then you cannot let Ai loose to iterate autonomously toward the goal. You are all the time babysitting and nudging. But now that you have your evals and iteration, you let LLM run.

Once you have maxed out the result of this internal iteration, it is time to give the prototype to customers.

Connect the customer feedback and reactions back to the loop. The goal is not to take the improvements "as is". The key is to come up with richer and more accurate evals.

They say evals is the new black. I say so too.

Evals + iterations is what I'm going to do.

Your ceiling is how well you can capture the elusive "truth".