There is a file you can put on your website called llms.txt. The idea is that it tells an AI who you are in plain language, instead of making it dig that out of your marketing.
There is also a large group of people online who will tell you it does nothing. That no AI reads it. That it is a fad invented by people selling AI services, which is a category I am standing in, so I have a reason to want them to be wrong and you should know that going in.
Last week I said I would get into the robots themselves this Tuesday. Which ones come to your site, what each one is for. That one is still coming. But on the way to writing it I found out my own tool could not see them, and I am not going to describe robots I cannot see. So this first.
We put one on every site we look after. So I went to find out who was fetching them, because I would rather know than argue.
Our own tool said nobody.
Zero requests. Not a handful. None, across seven sites, out of seventy five hundred logged robot visits.
That is a hell of a finding, and it was in a draft of this newsletter before lunch.
What saved me was small and it was not clever.
Before writing the sentence, I checked the file was there. If it does not exist then nobody fetching it means nothing. So I pulled it on every site that is supposed to have one. It is there. Every one, loads fine, content in it.
Good. Zero it is.
And then I ran into the part of my own rules I keep having to learn twice. I checked whether the file existed. I never checked whether my tool could see it.
Here is the boring mechanical reason, and it is worth two minutes because it is probably sitting in something you are paying for.
Our robot logger is a piece of software that runs inside WordPress. It wakes up when WordPress wakes up. That is fine for most of your website, because most pages get built by WordPress on the spot when somebody asks for one.
But llms.txt is not built by anything. It is a plain file sitting on a hard drive. When a robot asks for it, the web server hands it straight over and WordPress never wakes up at all.
So our logger never ran. Not once, for that file, on any site, ever.
And a piece of software that never runs does not report an error. It reports nothing.
Blind and quiet look the same on a screen.
Five of our sites had been blind for seventeen days and the number I was looking at stayed a calm healthy zero the entire time.
So I went underneath it, to the raw server log. The one the web server writes itself whether WordPress is awake or not. Seventy thousand lines.
Eight hundred and thirty eight verified requests for that file.
Seven hundred sixty seven of them from ClaudeBot, which is Anthropic’s crawler. Seventy from GPTBot, which is OpenAI’s. One from OpenAI’s search crawler. Spread across six sites, the busiest of them a property services company in Lee’s Summit.
Verified means we checked the address each request came from against the list those companies publish, rather than trusting the name it called itself. Two of the GPTBot ones failed that check and were somebody in a costume. I am telling you about those two because a number with no impostors in it is usually a number that got tidied.
One more thing, and this is what made me believe it.
We look after a site that did not have the file for most of that window. It was behind a coming soon page and there was nothing at that address. Robots asked for it eight times, got nothing all eight times, and not one of those was an AI crawler.
The thing is absent exactly where the file is absent. That is the check that separates a finding from a coincidence, and it is the one I would want somebody to run on me before I believed a word of this.
Here is where I stop, because this is the line I watch people sprint past.
I know the file was asked for. I know it was handed over. I do not know what happened to it after that, and neither does anybody showing you a chart about this. A server log tells you somebody knocked on the door and what you handed them. It does not tell you whether they read it in the car.
So llms.txt is read. Whether it does anything is a separate question and I do not have that answer yet. When I do, you will get it either way.
Six sites. About ten weeks. Our own fleet, not the internet.
Here is why this went the way it did instead of the other way.
Three weeks ago I told you about the rule we run everything through now, the one from the CIA analyst. Before you believe a number, you say out front what would prove it wrong, and then you go check that thing on purpose. It is slow and it is annoying and it is the reason the zero never left the building. The rule did not stop me from reading a blind sensor. Nothing stops that. It made me check the sensor before I put my name under it.
What it bought: 838 verified requests, two impostors caught and counted, and a control site that came back empty exactly where it should have. Every one of those can be checked by anybody with a server log. That is how deep we go for a file most people will tell you does nothing.
So if you have an llms.txt, go find your raw server log. Not your analytics. Analytics will never show you this. Search it for llms and see who has been coming by. It takes about a minute and it is your own data.
If your site is on Wix or Squarespace, you cannot run this check. They do not hand you the server log. And their analytics, same as Google Analytics on any platform, run on a script inside a browser, which a robot does not have. A crawler has never once shown up in a Google Analytics report and it never will. The tool is not lying to you. It is pointed at the porch.
And whatever tool you are trusting to tell you what AI is doing with your business, ask it one question before you believe the next number it hands you.
Can you see the thing you are counting?
Next Tuesday, the robots. Now that I can see them, I can tell you which ones come to your site, what each one is for, and which one decides whether an AI ever knows you exist.
See you next Tuesday.
Nathan. Lefty Media Co, Kansas City.