The Agentic Web Index

Traffic by Agent Type

Hover over each agent type for more information about what they do

Spoofed Traffic by Agent Identity

The percentage of impersonated website traffic for each agent identity over time

Traffic by AI Chat Platform

Referral activity by AI chat platform over time

Data Scope

The Index is updated daily with completed days of traffic, security, and referral data from more than 5,000 websites using Agent Analytics and AI Chat Referral Tracking. The current partial day is excluded. Percentage-change tags compare the current period with the preceding period of the same duration. Agent names, operators, and classifications come from the Agent Directory, which is updated as new agents are discovered or existing agents change. Website categories follow the taxonomy used by Google AdSense. Participating websites are not a random sample of the entire web, and the qualifying set can change as websites connect, disconnect, or cross activity thresholds. Results characterize the observed network and broader directional trends; they should not be interpreted as a precise census of global web traffic.

Qualification & Aggregation

Only websites meeting minimum activity and data-quality requirements are included. Internal, test, incomplete, or anomalous data is excluded. Bot traffic percentages use total server traffic as their denominator. AI chat referral percentages use estimated human traffic, calculated by excluding identified bot visits from total server traffic. Rates are calculated for each qualifying website first, then averaged across websites and completed days. This gives each website equal weight regardless of traffic volume and prevents a small number of high-traffic websites from dominating the results. Daily charts are not smoothed, allowing normal seasonality to remain visible.

Measuring Robots.txt Effectiveness

Robots.txt Effectiveness estimates how much an agent's share of website traffic falls on websites that block it sitewide using Disallow: /. It measures the observed outcome of the rule, not the agent's intent. Each day, Known Agents averages the agent's share of traffic across all qualifying websites where it is allowed, whether or not the agent visited them that day. This baseline is scaled to each blocked website's traffic and compared with observed disallowed visits. A day qualifies only with sufficient agent activity across multiple allowed websites and expected traffic across multiple blocked websites. Scores require qualifying observations consistently across multiple recent completed days. For agents supporting authentication, only verified traffic is attributed.

Each qualifying website-day contributes equally. A 0% score means no measurable traffic reduction; 100% means no disallowed visits were observed. Zero observed traffic counts only when the baseline predicts meaningful activity, and agents without enough broadly distributed traffic receive no score. The headline metric gives qualifying agents equal weight. Because this is observational, it cannot distinguish technical enforcement from voluntary behavior or prove causation. Top Blocked Bots is calculated separately using robots.txt scans of Similarweb's top 1,000 websites.

Identifying Spoofed Bots

Spoofing statistics measure traffic from visits that claim the identity of a known agent but fail a supported authentication method, such as published IP verification or HTTP message signatures. Each agent's daily rate is calculated against total server traffic for every qualifying website, then averaged across websites. A failed check indicates that the visit was likely impersonating the named agent; it does not identify the software or operator that actually made the request. Agents without a supported authentication method are not included in these measurements.

AI Chat Citations & Referrals

AI chat referral statistics count directly observed human visits carrying a recognized AI platform in the referring URL or campaign source. Visits without usable referral information cannot be attributed to an AI platform. Citation statistics are estimates based on requests from agents known to retrieve content for AI platforms. Those requests indicate that content may have informed a response, but they do not confirm that a source appeared as a citation to a user. Because AI platforms do not provide a complete public record of their sources, citation results should be interpreted as directional patterns rather than exact citation counts.