I was paying AI for answers and doing nothing with them
Every two days, Nova asks ChatGPT, Claude and Perplexity the questions a customer would ask about our clients. Nova is the AI we built here at Lefty, and she does the legwork. Who is the best kitchen remodeler in Kansas City. Who trains dogs near me. She checks whether our client got named, marks it yes or no, and moves on to the next one.
Since June that has added up to 30,833 answers across ten websites, and I paid for every one of them. Our AI bills since June have run past six hundred dollars, and asking those questions is a big piece of that. Running a business, I spend a good part of every week looking for anything I can stop paying for, and here I was paying for this on purpose.
Then I looked at what was coming back. Every one of those answers was a full paragraph of an AI explaining who it would hire and why, which businesses it named instead of ours, and which websites it read before it decided. We were saving every word of it and reading one.
I had already paid for the answers. Reading them cost nothing.
It was like finding money in a coat I already owned. So we built a place to read them, and since people keep asking me what that place is, here is the tour I would give a new guy on his first day.
First is the Foundry, and it is the part I am proudest of. Everything Nova brings back goes into the Foundry: every answer she collects, the server logs that show which AI robots visited which pages and when, and the record of every change we make to a client’s site.
The Foundry never treats a good result as the end of the conversation. When a client gets named, it asks why, which page the AI read before it decided, and what we have to protect so it happens again. When a client does not get named, it asks who did, why the AI picked them, and where it went to find its answer. One of our clients was pulled up as a source twenty-four times in one day of questions and named in the answer zero times. The Foundry is what caught that.
Then it zooms out. It tracks how many kinds of AI robots are out there and what each one does when it visits. It looks for patterns that repeat across every client and for the ones that only appear for one, because a fix that works for a contractor might do nothing for a dog trainer, and the job is knowing which is which before anybody spends a client’s money on it. Anything the Foundry thinks might be a rule does not get called a rule yet. It goes to the Lab.
The Lab runs on one rule. Before we look at the results, we put the prediction in writing, along with the result that would prove us wrong. We also use a method built at the CIA by an analyst named Richards Heuer, called Analysis of Competing Hypotheses. Instead of picking the explanation we like and collecting evidence for it, we list every explanation that could be true, including the embarrassing one where our own records are wrong, and score every piece of evidence by which explanations it rules out. Across eight clients we have weighed fifty of those explanations so far. Then the data decides, not me. Two of my own ideas have died in there. I was sure fake AI robots go after small websites more than the industry average. The data said no. I was sure a business whose name sounds like something else would get answered as that something else. It happened two times out of five, which is a no.
I like that those two died. A Lab where my ideas always win is a Lab that is not testing anything.
Last is the Casebook. Every finding worth a closer look gets a case file there in plain English, and we have opened fourteen so far. Most have since been retired, because they turned out wrong, stopped being true, or did not reproduce when we checked them a second time. When one survives everything we throw at it, it goes out as a research paper with my name on it, filed on Zenodo, a research archive run by CERN, with a permanent ID so anyone can check the work. There are three so far. One of them looked at a file called llms.txt that Google has told everybody nobody uses. Google was right about Google. It visited our sites 939 times in a month and never opened the file once, while Anthropic’s robot opened it more than a thousand times.
People ask me why a Kansas City company bothers writing research papers. There are two reasons.
The first is that nobody else has done it. If I wanted to open a regular marketing company tomorrow, search ads and backlinks and the rest, there are a million guides on how to do every part of it. With AI there is almost nothing. A survey of this entire field published this summer went through the research and could not find a single technique with a steady, lasting effect on getting a business found across the AI tools. If somebody had already built this, I would happily use theirs. Nobody has, so we do the work.
The second is how fast it moves. Stanford’s AI Index, which comes out of a university and is not selling anybody anything, found that on one hard coding test the best AI went from 60 percent to nearly 100 percent in a single year. Generative AI reached more than half the population within three years, faster than the personal computer or the internet did. We feel it from the inside too. In September we had to change two of the three AI engines we measure, because the companies behind them changed what they run.
A how-to written in the spring is a history book by the fall.
Here is what I am betting on. In a market this new, what separates people is who you can trust, and with AI the only way I know to earn that is to show the work: the numbers, the method, and the ideas that died.
Will other people copy this and maybe do it better? I hope so. That is competition, and it would make this field more honest than it is right now.
So if you are hiring anybody to help your business get found by AI, us included, ask them one question. When is the last time you changed your system and what caused you to change it? If they cannot answer that, they are guessing, and you are paying for the guess.
Nathan. Lefty Media Co, Kansas City.
Lefty Notes goes out every Tuesday morning. Subscribe on Substack and the next one lands in your inbox.
