I was talking with a guy last weekend who is a lot deeper in tech than I am. He builds things that work. I break things and then send you a newsletter about it.
I brought up the robots on his website the way you would bring up weather, and he said sure, Google. I said not Google. I said there are a handful of them now, they work for the AI companies, they do not do the same job, and one of them is the difference between an AI having heard of him and an AI having no idea he exists.
He looked at me like I had told him his truck had a second engine.
Then he asked me which one, and I realized I had never laid this out in plain words for anybody, and I was not sure I could. If he did not know it, I cannot work out why anybody else would. So this is that post. It is two weeks late because the first time I sat down to write it, I found out my own logger could not see the robots, and I was not going to describe animals I could not see. Last week was about fixing the sensor. It reads the server’s own log now, the one the web server writes whether WordPress is awake or not. So now I can see them.
I wanted to hand you a list. There is no list.
OpenAI publishes four robot names. Anthropic publishes three. One of OpenAI’s four was not there the last time I looked, it checks ads, and I only know it exists because I went back to the page this weekend to make sure I had the count right. They add these and rename them and do not send a card. And on top of that, a good share of what shows up in your logs wearing one of those names is not that company at all. It is somebody else in a costume, which I told you about in August.
So I cannot give you a list. What I can give you is the three jobs, because the jobs have not changed even when the names did.
The first job is the reader. GPTBot, ClaudeBot. This one shows up, reads what you have, and hauls the text home so it can be taught to the next version of the model. It is slow, it works in seasons, and it is the one that answers the question I asked two weeks ago when we took the internet away. Whatever it took on its last trip through your site is all the machine knows about you when the search is switched off. This is the robot that decides whether AI knows you exist.
The second job is the librarian. OAI-SearchBot, Claude-SearchBot, PerplexityBot. This one builds an index, the way Google builds an index, so that when the AI goes searching in the middle of an answer it has somewhere to look. It moves faster than the reader and it forgets faster too.
The third job is the errand. ChatGPT-User, Claude-User. This one does not come on a schedule. It comes because a human asked a question right now and the AI went to your site to answer it. It is the closest thing in your logs to a citation. It is also, some of the time, you, testing your own site, which is the story from the week I went looking for a liar in my logs and found me.
Three jobs. Same companies, different names, and the names are not stable. That is the honest version.
Now here is what they are doing on our sites, checked by address, not by what they call themselves. Thirty five days of logs, six sites, and every count below is only the visits that came from an address the company publishes.
The reader from Anthropic keeps a schedule. Every two hours on a site it is watching it asks for three files. Your robots file, your llms file, your sitemap. Thirty six lines a day, like a clock. It is reading the map. Across our sites in those five weeks it read the map three thousand eight hundred times and opened two hundred pages. About nineteen map reads for every page.
And it has a watch list, and the watch list moves. A property services company in Lee’s Summit has had it every one of the thirty five days. Our own site had it, thirty six lines a day, and on September 9th it stopped. Eleven days of nothing. I went looking for why. I killed seven explanations with our own data in one afternoon. The door was open, the robots file was identical to the Lee’s Summit one, the same addresses were reaching that site the same day. Nothing.
Then it came back. Today. While I was working on this. And this time it did not read the map. It read the posts. The press page, the about page, five of these newsletters. Thirty seven requests, every one from an address Anthropic publishes, every one answered with the page.
I do not know why it left and I do not know why it came back, and nothing the company publishes explains either. I am telling you anyway, because the alternative is a newsletter that only reports the parts that make sense.
The librarian from Perplexity came to our sites exactly once. One address, three sites, four days in late August, asked for the robots file and nothing else, and has not been seen since on anything we host. The other visits wearing its name were scanners asking for password files.
Google’s AI reader has never come to any site we own. Not once in the old logger, and not once in thirty five days of the new one. Every visit wearing that name came from an address that was also wearing five other names on the same day. In August that name showed up on exactly two of our sites and for about an hour it looked like a finding about Google noticing us. Then we checked the address.
The errand robots are the ones I expected to see least, and they are the ones that surprised me. Ninety seven times in those thirty five days, a ChatGPT conversation reached out to one of our client sites in the middle of an answer. Four sites. Thirty one different days. Ninety one different addresses, every one inside OpenAI’s published range. Almost always the front page. I cannot tell you who asked or what they asked, the log does not have that. I can tell you it was not us testing, because our own site, the one we poke at more than any other, got none of them. Somebody out there asked a question and the machine went to a client’s front door to answer it. That is the closest thing to a citation I can show you, and I did not know it was there until this weekend.
Claude’s version of the same robot has not made one verified visit to any site we have. Forty visits wore its name. Every one came from an address that was not Anthropic’s, or from my own house.
And then the one that got a swear word out of me.
One of the sites we look after is on a managed host, a good one, the kind you pay for so you do not have to think about the server. In three weeks the reader from Anthropic knocked on that door one thousand and eighty seven times.
Six hundred and fifty seven times it was told to go away.
Sixty percent. Not by the client. By the host. The answer it sends is the one that means too many requests, come back later, except it is not about speed. We checked. The robot gets refused at a walk. It is being refused for who it is. And a good robot takes that answer, comes back later like it was told, and gets the same answer. So the one robot that decides whether AI knows this business was being turned away at the door six times out of ten, every day of the three weeks we have logs for, and there was no screen anywhere in that company or mine that showed it.
A robot that gets told to go away does not complain. It just does not know you.
That is the part for you, and it is small.
If you have a raw server log, and I told you last week where to find it and why your analytics will never show you this, search it for GPTBot and ClaudeBot. Then look at the column next to it with the three digit number. 200 means you handed it what it asked for. 429 or 403 means you sent it away. You can have the file on your site, the map, the plain language description, all of it, and if that column says 429 none of it ever left the building.
I do not have a fix for you yet, and I am not going to pretend the fix is one setting. On that client’s site it is a conversation with the host that has not happened. When it does, and whichever way it goes, you will hear about it here.
What all of this bought, since I keep telling you the mistake and forgetting to tell you the part after. The sensor is fixed and it reads from underneath WordPress now. Every robot count in this post was checked against the address ranges the companies publish, not against the name in the log. Two things I believed when I sat down to write this were wrong by the time I finished, that the errand robots never come and that our site had been dropped for good, and both got fixed because I went back to the log before I hit send instead of after. The sixty percent is out of one thousand and eighty seven. Anybody with the same log could get the same numbers, and that is the only kind of number I want to hand you.
Next Tuesday is something lighter. Somebody asked me whether I would rather work with my friends or have coworkers, and answering it got me into an argument with a robot that I lost.
See you next Tuesday.
Nathan. Lefty Media Co, Kansas City.