Skip to content
Cloudflare Docs

Analyze AI traffic

AI Crawl Control metrics provide you with insight on how AI crawlers are interacting with your website (Cloudflare zone).

To view AI Crawl Control metrics:

  1. Log in to the Cloudflare dashboard ↗, and select your account and domain.
  2. Go to AI Crawl Control.

You can find meaningful information across both Crawlers and Metrics tabs.

View AI crawler activity

The Crawlers tab provides you with the following information:

  • Total number of requests to crawl your website from common AI crawlers
  • Number of requests made by each AI crawler
  • Number of robots.txt violations for each crawler

View AI Crawl Control metrics

The Metrics tab provides you with the following metrics to help you understand how AI crawlers are interacting with your website.

MetricDescription
Total requestsThe total number of requests to crawl your website, from all AI crawlers
Allowed requestsThe number of crawler requests that received a successful response from your site
Unsuccessful requestsThe number of crawler requests that failed (HTTP 4xx or 5xx) as a result of a blocked request, other security rules, or website errors such as a crawler attempting to access a non-existent page
Overall popular pathsThe most popular pages crawled by AI crawlers, from all AI crawlers
Most active AI crawlers by operatorsThe AI crawler owners with the highest number of requests to access your site
Request by AI crawlersA graph which displays the number of crawl requests from each AI crawler
Most popular paths by AI crawlersThe most popular pages crawled by AI crawlers, for each AI crawler
ReferralsA graph which displays the number of visits that were directed to your site from each AI operator
ReferersThe list of referers who directed visits to your site

Filter date range

You can use the date filter to choose the period of time you wish to analyze.

Filter options:

  • Past 24 hours

The values of the AI Crawl Control metrics will update according to your filter.