A friend of mine described today the act of steering Ai as steering without reading (all) the text. It was so well said that it just flows.
The problem many have with Ai is the walls of text. Ai and LLMs just are verbose. Correctness in turn in common sense is absence of erroneous claims in the text. The feeling is to have to read all the text to assure yourself the whole is correct.
There is just too much to read. Much of it repeated. Most of the time too much.
I don't read full documents
I have stopped for some time to read full documents at work. Some key works I still sometimes read when an even stronger memory imprint is needed.
Most of my work I consume with the LLM. I load the context. I explore the concepts and work with the LLM. I combine. I read in context of strategy or similar additional references.
But most of all: I mostly don't read to put concepts in my head. I use the LLM to recast those concepts into whatever I'm doing. There really are few words for these. But it is like building with abstract legos. Or holding the abstract Play-Doh from multiple sources. I approach the outcome and intermediate work I'm trying to put together.
I read to steer on high level. But I don't read all.
All the text is still necessary as it usually holds relevant information.
Sampling for quality
The problem with walls of text is of course precision and quality. Any sentence can be right or wrong. Typos are easy. They are just wrong. But sentences can be right or wrong based on context.
The quality of a bigger sample of text can be established by reading all of it.
Testers figured out a long time ago that not every use case, error condition and combination can be tested. That is mathematically impossible. And the testing game is all about spending scarce attention on the highest value findings.
Great quality has always been being very smart on what to look at and why. Testers don't test all things over and over again.
But how do testers conclude a product works right: that there are no errors? Well... many say that absence of error cannot even be proven. So we have risk. We check enough to have growing confidence and evidence of small problem rates.
Quality management is a sampling approach.
Same with text. You read what is most valuable to make the next move.
Much of the text is for the LLM itself.
The basic technique and grounding
I'm learning new vocabulary on my agentic journey. The concept of grounding is something that really helped me.
Grounding is that a claim is based on something. "I worked at Nitor" is a claim. But how would you know that is true or false. You need to go to the "ground" that is reliable. Maybe you would have to go to my work contract. Or payslip. I think you would not accept my LinkedIn profile as reliable ground.
Dreams and make-believe are not grounded. They just appear from thin air.
The basic technique with Claude is: "Hey Claude: take text XXX, find right grounding, check all claims for validity".
Matching mental models
Matt Pocock has once said: the plan (with LLM) is ready when you notice your mental model matches the LLM's "mental model". I had to use quotation marks as LLMs don't quite hold mental models, but close enough. When the LLM answers congruently with your own assumptions given the context, then you can notice a higher level alignment. Single sentences may not match stylistically or in all detail.
But you know you are aligned. In some different way: you sample on higher abstraction level. The exact text is not the point.
Now capture the alignment. Make it into ground and guardrails. Compound your context.
The agent supercharges your sampling. And steering is the capability you get
Agents are not humans. But agents can check thousandfold. You can combine both. You read a bit. The agent checks a lot.
The trick is curating your rules, context and even taste into format that you can apply with your agent. All of a sudden, your agent can do the grunt work your eyes and attention have been needed for. You sample enough to get a feeling. You sample enough to spot errors. Errors turn into repeatable rules.
And then something changes. You create text. Another agent goes in first. Loop until all appears fine. The agent is sampling.
Then you sample. But you sample to keep informed. You sample to learn what your system does not yet catch right.
You can steer throughout without reading and spot-fixing. A really interesting progression still for me.
PS: credits Juha Itkonen for the wording. PPS: Was going to write on "steering with outcomes": let that be a followup some day.