www.simplify-data.de
When people want to know something today, they no longer necessarily open ten browser tabs. Many ask an AI assistant directly, which summarises, compares and, if asked, acts on the answer. For website owners this means a growing share of requests no longer comes from people but from machines reading on their behalf. This article uses Cloudflare's figures to show how fast this is developing and what businesses can take from it.
From a list of results to an answer
Classic search returns a list of links, and the user decides which pages to visit. With AI assistants and agents this division of labour changes: the assistant fetches pages itself, reads them, summarises them and returns a finished answer. Whether the source is still clicked is an open question. Two things matter for businesses: your content has to be readable for these systems, and it has to convince as a source.
What Cloudflare's data shows
Cloudflare operates one of the largest networks on the internet and publishes annual analyses of global traffic with Cloudflare Radar. The 2025 Year in Review (data covering 2025) shows several things:
- AI bots are a noticeable share of traffic. On average over the year, AI bots made up about 4.2 percent of all HTML requests, and Googlebot alone about 4.5 percent. On 2 December 2025, 47 percent of HTML requests came from humans and 44 percent from bots that are not AI services.
- Agent-driven requests grow fastest. Cloudflare distinguishes crawlers that collect content for training from requests triggered by a user ("user action"), for example when an assistant fetches a page for a specific question. These requests rose by more than 15x during 2025. Requests from ChatGPT-User reached up to 16 times the level at the start of the year.
- Training remains large. According to Cloudflare, collecting training data makes up the overwhelming majority of AI crawling, at times 7 to 8 times search crawling. OpenAI's GPTBot increased its requests by 305 percent between May 2024 and May 2025.
- Overall crawler traffic is growing. Between May 2024 and May 2025, crawler traffic from AI and search services rose by 18 percent according to Cloudflare, and Googlebot's requests alone by 96 percent.
These figures describe traffic on Cloudflare's network and are not a measurement of the entire internet. They do show the direction clearly: machine readers are increasing, and requests triggered directly by a user are increasing fastest.
Many fetches, few visitors
Particularly revealing is the "crawl-to-refer ratio" that Cloudflare has published since July 2025. It shows how often a service fetches pages compared with how often it sends visitors back. The range is wide: for Google the ratio was at times around 3 to 1 and rose to as much as 30 to 1. For OpenAI the peak was around 3,700 to 1, and for Anthropic at times considerably higher. In other words, AI services read far more pages than they refer visitors to. Anyone who measures success only by visitor numbers therefore sees only part of what happens to their content.
The numbers alone do not show how much visits to individual websites decline. It is plausible, however, that people more often receive a finished answer and visit the source less often. That is our assessment, not a statement by Cloudflare.
What businesses should do now
- Decide who may read. Your robots.txt defines which crawlers have access. Cloudflare reports that AI crawlers such as GPTBot, ClaudeBot and CCBot are the ones most often fully disallowed. That can make sense, for example regarding training, but it should be a conscious decision: if you lock assistants out, you will not appear in their answers.
- Make content machine-readable. A clear HTML structure, meaningful headings and content that is already in the delivered HTML rather than loaded later via JavaScript help both people and machines.
- Maintain structured data. Schema.org markup for organisation, services, articles and FAQs makes information unambiguous.
- Provide answers instead of platitudes. Concrete facts, prices, processes and contacts are easy for assistants to cite; vague marketing language is not.
- Keep the technology in order. Fast response times, clean sitemaps and reachable pages are the basis for crawlers being able to fetch your content at all.
- Look at new metrics. Besides visitors, it is worth checking your server logs to see which bots read which pages.
Checking your visibility with Simplify Data
In our SEO audit we check not only classic ranking factors but also how accessible your website is to AI search systems and agents. The free SEO quick test gives a first impression. We work for companies in Kiel, Schleswig-Holstein and Hamburg, and remotely across Germany. More about the SEO audit