Fair warning. This is the longest one of these I have ever sent you, and I am not sorry about it. Get a coffee. The good part is the part where I find out the intruder was me.
Last Tuesday I told you there is a robot in my logs that says it is Google and is not Google, and that I would get into it this week.
So I went looking for the liar.
I found something that looked a lot worse. Then I found out it was me.
The same computer kept showing up on more than one of my clients’ websites, and it gave a different name at every one. Picture one guy walking into three of your neighbors’ houses on the same afternoon and using a different name at each door.
That is not somebody reading your website. That is somebody checking which doors move.
I lined our client sites up against each other and that pattern came back fifteen times out of twenty one.
Fifteen out of twenty one is not a fluke. That is a finding, and it was going in a report with my name on it.
It was us.
It was our own machines. Some of it was the knocking I told you about last week, from before I taught it to sign its name. Most of it was me, sitting at my desk during the hosting migration, testing the doors by hand and pretending to be a crawler. Over and over, on every client, from the same few computers, wearing whatever name I needed that minute.
I built the thing that fakes a name in a log. Then I went hunting for a faker in the log. Then I found me.
The honest number was four out of twenty one, not fifteen.
Four is still four. There is something sitting on those four and I am still watching it. But eleven of the fifteen were me. I scared myself with my own work.
The sad part is I have said this exact thing in the last three or four of these.
It was my gut. I knew it was not right and I did not want to believe it, because I wanted to believe the data. The numbers were the easier road. In this business there is no playbook, so failing is more common than winning, and if I hide the failing I taint my own name and the industry I am trying to lead. I want to win. Winning is admitting the mistake and being honest about it. But some part of me wants to be able to say ha, I do know AI. That is just not possible right now.
So why did I catch it at all.
Every business I was in before this one came with a playbook. Somebody figured it out in 2003, everybody got handed the same copy, and you competed on how well you ran it. Boring, but you knew the rules.
There is no 2003 for AI. There is no book everybody plays by, because the thing is a couple of years old and it moves under you while you are looking at it. You cannot borrow the method from marketing either, because marketing has not figured this out yet. Everybody is guessing, me included.
So if the field is brand new, the way you study the field has to be brand new too. For me that meant going and finding somebody who already solved the hard part somewhere else entirely.
That is how a CIA method ended up in a newsletter about small business.
A man named Richards Heuer spent a career on one problem. How do you get to the truth when your information is incomplete and some of it is lying to you on purpose. He called it Analysis of Competing Hypotheses, which is a mouthful for something fairly simple.
Here is the version I run. Heuer wrote a book about it. I run five steps.
You name every explanation you can think of. Not the interesting ones. All of them, including the one that makes you look stupid.
You lay every piece of evidence next to that list. Including the thing you went looking for and did not find.
Then you take each piece of evidence and ask it one question. Does this tell my explanations apart, or does it fit all of them the same?
Anything that fits all of them is worth nothing. However good it feels.
Then you spend your time trying to kill explanations instead of proving one. Whatever you cannot kill is your answer.
The only reason I caught myself is that the list made me include “it is us.” That one is boring and it is embarrassing and it never lands on the list by accident. But it was on there, so it had to be killed like the rest of them, and it would not die.
So what did all that buy me, besides a smaller number and a rough morning.
A true one. And true is the part I keep forgetting to tell you about.
Every line in a log used to be a maybe. Now every line gets checked against the address ranges the AI companies publish themselves. Nine hundred and thirty four of them. The name in a log is a claim. The address is the evidence.
Once you can do that, the junk falls out on its own. Two thousand six hundred requests that were never AI at all. Fifteen hundred of those were somebody rattling a login page hunting for a password. Not a crawler. Never was. Under the old way of counting, all of it was traffic, and traffic is the kind of thing that ends up on a slide in front of you.
Of what was left, the ones lying about being a crawler came to about five percent.
The rest I have started calling chasers. Fake knocks that look like something, feel like something, and cost you a morning going after a machine that was never there.
Now add up what I have told you about in here since the middle of the summer.
My own AI called us a web development agency on our own website. The credentials we built for clients got deleted, and the people who deleted them were right. My chart said zero when the truth was a locked door. And now this one, where the intruder was me.
Every one of those was the instrument, not the work. And every one of them left something behind that is still running. The files got fixed. A person reads the accounts now instead of trusting a green dashboard. The door gets knocked on a schedule whether anything looks wrong or not. The check that caught me this week sits at the top of the file and runs before anything else does.
So the gear I use to tell a client what is happening to them is not the gear that was lying to me in July. That is not me saying we are good. It is me saying you can check us, and that you have watched me do it for two months.
That is the part I keep leaving out. I show you the hole every week and never the patch.
That is a lot of work to find out you are the problem. Hours of it, on a Tuesday, to take a number I liked and make it smaller.
I do not know for certain that this is the right way to study any of this. Nobody does. But I would rather borrow a method built for liars than run a playbook that was finished before any of this existed.
That check now sits at the top of our code with a note on it, because I did not want to lose the reason:
an accusation aimed at our own laptop is worse than useless, because it is the one finding nobody would think to double-check
Here is the part for you.
You have done this. You asked ChatGPT something, then you asked Claude the same thing to check it, and you felt better when they agreed.
They agree a lot. They learned from a lot of the same text. Two of them agreeing is not two opinions. It is closer to one opinion said twice in two different voices, and it lands like confirmation, because that is what confirmation feels like from the inside.
There is a name for it. Correlated error.
Agreement is the cheapest evidence there is.
It fits “this is true” and it fits “this is a popular thing that is wrong.” Which puts it at about zero.
The better question, and it is free:
What would prove this wrong?
Something that knows comes back fast and specific. Something that is guessing hedges, goes vague, or tells you it depends. You will hear the difference immediately. Try it on the next confident answer an AI gives you. Then try it on the next confident answer from a person selling you something. Same test. It works on me too, and you should use it on me.
We are building the shop around this, and not because it makes us look clever. We call our shot before we look at the data. We say out front what would prove us wrong. And when the answer comes back that we spent a morning accusing ourselves, we publish that too.
That is going to be uncomfortable roughly once a month.
It is cheaper than being confidently wrong in front of somebody who trusts me.
Next Tuesday, the harder version of this one. Being wrong is easy. A number comes back negative, a page falls over, you go fix it in an afternoon. I built a page in my own software where every line on it is accurate, and it has never once helped anybody do their job. I defended it for a week without noticing, because every time I got suspicious I checked a number, the number was right, and I walked away happy.
Go find the last thing you were sure about this week. Ask what would prove it wrong.
See you next Tuesday.
Nathan. Lefty Media Co, Kansas City.