For months I have been putting special files on my clients’ websites, files written for AI instead of for people. The idea is simple. Hand the AI a clean, honest version of who this business is, so when somebody asks an assistant about them, the answer is sitting right there.
And for months, almost everyone told me I was wasting my time.
The research said it. The big studies said these files barely get read. Even the AI tools I work alongside every day told me to my face that the major engines were not bothering with them. Smart people, good data, all pointing the same direction. I probably should have folded.
But I had a stubborn feeling I could not shake. I kept watching AI tools read my clients’ words back almost exactly the way we wrote them. That does not happen by accident. So instead of arguing about it, I did the only honest thing I know how to do. I built a tool to settle it.
I put a small piece of code on my clients’ sites that records every single time an AI crawler comes and reads that file. The bot, the timestamp, all of it, straight to a dashboard I can watch.
Then I waited.
It took less than a day. The logs filled up. ClaudeBot from Anthropic, eighteen and twenty-four times on a single site. Bing’s crawler. Apple’s crawler, the one that feeds Siri. And the one that stopped me cold: OpenAI’s search bot, the exact crawler that fetches pages to build ChatGPT’s answers. Pulling my clients’ files. On its own. Over and over.
Here is what nobody tells you. The studies are right. The average AI file gets ignored. And they are wrong that it has to. The difference is one thing, whether anyone gave that file a door. Most people drop it on the site and walk away, so the crawlers never find it, sitting alone in the dark with no reason to come look. We do not walk away. We build the door, we point the crawlers at it, and then we watch them show up.
Everyone is right that the average file gets ignored. And everyone is wrong that it has to be.
So here is what I actually know now, with receipts instead of opinions.
The files do their job. They teach the AI who your business is, in plain terms it can read fast. That is the first half of getting found, being known. My clients are getting crawled by the same engines that answer millions of questions a day.
I will be honest about the second half too, because I do not sell magic. Being known is not the same as being chosen. When someone asks AI for the best in town, getting picked still comes down to your reputation out in the world, your reviews, your records, the people who vouch for you. The file gets you in the room. Your reputation gets you picked.
But I am done apologizing for the files. I have a dashboard full of OpenAI and Anthropic timestamps that says they are being read every single day.
I trusted a feeling, built a tool to check it, and the tool sided with the feeling. That does not make me right about everything. It makes me a little less willing to fold the next time the room tells me I am wrong.