AI assistants now read online stores more than twice as often as Google. Most retailers can't see it

For twenty years, online retailers have built their websites for one reader above all others: Google. Page titles, product descriptions, site speed and structured data have all been tuned for the crawler that decides who appears on the first page of search results.

That reader is no longer the busiest visitor on many websites.

My company runs its own websites alongside the stores we build for merchants. This month we went through the raw server logs for two of our sites, line by line, to see who was actually reading them. Between 1 and 27 September, crawlers from OpenAI, Anthropic, Perplexity and Meta made 6,826 requests. Googlebot made 2,810. AI systems read our pages almost two and a half times as often as the search engine most businesses still optimise for.

More telling than the volume is the kind of visit. 859 of those requests came from assistants fetching a page live, in the moment, because a person had just asked ChatGPT, Claude or Perplexity a question and the assistant went to find the answer. Those are not background crawls. Each one is a potential customer asking an AI tool for a recommendation, and the tool reading a business website to decide what to say.

For online retailers in New Zealand and Australia, that changes the question. It is no longer only "do we rank on Google?" It is also "when an AI assistant reads our store, what does it find?"

Most businesses have never looked

Almost no merchant I speak to has checked. Analytics tools such as Google Analytics do not show these visits at all, because most AI crawlers do not run the tracking scripts that analytics depends on. The only place they appear is the server's own access log, which few business owners ever open.

When we looked closely at our own logs, three things stood out. Each applies to most online stores.

First, a meaningful share of "AI" traffic is not AI at all. About one in ten requests carrying an AI company's name were not reading pages. They were probing for configuration files, passwords and known software weaknesses. Attackers have learnt that many websites wave through anything that calls itself an AI crawler, so they borrow the name. A bot's name proves nothing. Several AI companies publish the network addresses their crawlers use, and checking traffic against those lists is the reliable way to tell the real ones from impostors.

Second, genuine AI crawlers hit errors far more often than you would expect. Of the remaining requests, nearly one in five ended in an error. Some reached pages that no longer existed. Some followed broken links that only a machine would find. Some were turned away by security rules or rate limits that treated a legitimate assistant like an attacker. Every one of those is a moment when an AI tool tried to learn about a business and came away with nothing.

Third, the settings that block AI tools are often switched on without anyone deciding to. At least one major content delivery network now blocks AI crawlers by default on new websites, and many hosting and security tools offer one-click blocking that is easy to switch on and forget. The website still works perfectly for people, so nobody notices. Meanwhile, AI assistants are told nothing, and the business quietly drops out of the answers.

Blocking is a decision, not a default

There are reasonable arguments for blocking some AI crawlers. Businesses with original research or premium content may not want it used to train models without payment. That is a legitimate commercial choice.

For most retailers, though, being readable is the point. A product that an assistant cannot read is a product it cannot recommend. The important distinction is between crawlers that gather training data and assistants that fetch a page to answer a live question. Many businesses block both without realising they are different, and lose the second, which is the one that sends customers.

Whatever you decide, it should be a decision someone in the business has made on purpose, not a setting inherited from a hosting plan.

What retailers should check this quarter

None of this needs a new platform or a large budget. It needs a few hours and the right questions.

1. Read your access logs, or ask whoever manages your website to. Look for the named AI crawlers and count how often they visit, which pages they request, and how often they receive an error.

2. Check your robots.txt file and your security settings together. The file might invite AI crawlers in while a firewall or content delivery network turns them away. Both have to agree.

3. Verify before you trust. Where AI companies publish their crawler address ranges, match your traffic against them. Treat anything that claims to be an AI crawler from somewhere else as hostile.

4. Fix the errors AI tools actually hit. Broken internal links, removed products without redirects and pages that time out all cost you with AI assistants as much as with shoppers.

5. Make your key facts easy to quote. Assistants favour pages that state plainly what a product is, what it costs, who it suits, and how shipping and returns work. Clear, specific, factual text is cited. Vague marketing language is skipped.

6. Test the result the way a customer would. Ask the major assistants about your category and your products, and see whether your business appears, and whether what they say is accurate.

The new front door

Search engines are not going away, and Google remains the largest single source of traffic for most online stores. But the way people find products is splitting across more channels, and AI assistants are becoming one of the most important of them.

The retailers who do well over the next few years will not necessarily be the biggest or the best funded. They will be the ones who noticed early that a new kind of reader had arrived, checked what it could see, and made sure the answer was their own.

Pratham Jani is the founder of Tripster Developers, a Shopify agency and software company founded in Auckland, New Zealand, in 2016, with offices in Sydney and Atlanta. A team of certified Shopify experts, Tripster has worked with more than 4,000 merchants across New Zealand, Australia, the United States and Canada, from independent stores to enterprise Shopify Plus brands.