Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot

The Agentic Web Index The internet is rapidly evolving from an environment built primarily for humans, into one increasingly used by machines. See how AI agents, crawlers, scrapers, and other bots are reshaping the way information is discovered, accessed, and used across the web. Overview Key ecosystem metrics across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking. Bot vs. Human Traffic Agentrification AI Chat Referral Volume Robots.txt Effectiveness Traffic by Agent Type Hover over each agent type for more information about what they do Top Agent Types Agent types with the most activity Top Visiting Agents Agents with the most activity Top Operators Operators with the most activity AI Scraping Activity These bots scrape website content to train AI models. Some belong to AI companies, while others belong to third-party services that resell the data. Automatic Robots.txt can block unwanted scraping. Included agent types include AI Data Providers and AI Data Scrapers. Traffic by Agent Type AI scraping activity by agent type over time Top Visited Website Categories Website categories with most activity Top Agents Agents doing the most AI scraping Top Operators Operators doing the most AI scraping AI Fetching Activity These bots fetch website content in real time to power AI assistants, coding agents, and other retrieval-augmented generation (RAG) tasks. Pages inform responses on the spot, such as when an assistant summarizes an article or a coding agent references documentation. Included agent types include AI Assistants and AI Coding Agents. Traffic by Agent Type AI fetching activity by agent type over time Top Visited Website Categories Website categories with most activity Top Agents Agents doing the most AI fetching Top Operators Operators doing the most AI fetching AI Search Indexing Activity These bots crawl website content so it can be surfaced in AI search engines and AI-generated answers. Those answers often include citations or links back to the source pages. Included agent types include AI Search Crawlers. Traffic by Agent Type AI search indexing activity by agent type over time Top Visited Website Categories Website categories with most activity Top Agents Agents doing the most AI search indexing Top Operators Operators doing the most AI search indexing AI Browsing Activity These bots use browsers to autonomously navigate websites, click through pages, and make decisions to complete tasks for people. Agentic UX best practices and Google PageSpeed Insights help evaluate how well websites support them. Included agent types include AI Agents. Average Session Duration Average Pages per Session Traffic by Agent Type AI browsing activity by agent type over time Top Visited Website Categories Website categories with most activity Top Agents Agents doing the most AI browsing Top Operators Operators doing the most AI browsing Robots.txt & Compliance See which robots.txt rules are set across the web and how well agents follow them. An agent's Robots.txt Effectiveness measures the effectiveness of a disallow rule for it by estimating how much the agent reduces its traffic after it's blocked. Effectiveness (Overall) Effectiveness (AI Scrapers & Data Providers) Top Rule-Following Agents Agents with the best Robots.txt Effectiveness percentages Top Rule-Breaking Agents Agents with the worst Robots.txt Effectiveness percentages Robots.txt Rules by Top Blocked Agent The percentage of the top 1,000 websites blocking each agent in robots.txt over time Top Blocked Agents Agents blocked by the most top websites Spoofing & Security See which agents are most frequently impersonated, and how spoofing activity changes over time. A visit is considered spoofed when it claims a recognized agent identity but fails that agent's supported authentication method, such as verified IP or Web Bot Auth. Spoofed Traffic by Agent Identity The percentage of impersonated website traffic for each agent identity over time Top Spoofed Agent Identities The most impersonated agent identities Recent Top Targeted Paths Examples of recent top targeted request paths AI Chat Referrals See which AI platforms like ChatGPT, Perplexity, and Gemini cite websites and send them human referral traffic. Citations are estimated. Google's guide explains how websites can optimize their content to be more visible in AI chat responses (GEO). Traffic by AI Chat Platform Referral activity by AI chat platform over time Top Mentioned (Cited) Website Categories Website categories most frequently cited in AI chat responses Top Clicked (Referred) Website Categories Website categories receiving the most referrals from AI chat Methodology Data Scope The Index is updated daily with completed days of traffic, security, and referral data from more than 5,000 websites using Agent Analytics and AI Chat Referral Tracking. The current partial day is excluded. Percentage-change tags compare the current period with the preceding period of the same duration. Agent names, operators, and classifications come from the Agent Directory, which is updated as new agents are discovered or existing agents change. Website categories follow the taxonomy used by Google AdSense. Participating websites are not a random sample of the entire web, and the qualifying set can change as websites connect, disconnect, or cross activity thresholds. Results characterize the observed network and broader directional trends; they should not be interpreted as a precise census of global web traffic. Qualification & Aggregation Only websites meeting minimum activity and data-quality requirements are included. Internal, test, incomplete, or anomalous data is excluded. Bot traffic percentages use total server traffic as their denominator. AI chat referral percentages use estimated human traffic, calculated by excluding identified bot visits from total server traffic. Rates are calculated for each qualifying website first, then averaged across websites and completed days. This gives each website equal weight regardless of traffic volume and prevents a small number of high-traffic websites from dominating the results. Daily charts are not smoothed, allowing normal seasonality to remain visible. Measuring Robots.txt Effectiveness An agent's Robots.txt Effectiveness estimates the reduction in its request rate associated with a full disallow rule. For each completed day, Known Agents establishes an agent-specific baseline from qualifying websites where that agent is allowed, adjusts the baseline for the overall traffic of each website where the agent is disallowed, and compares the expected activity with the activity actually observed. Only website-day observations with sufficient site traffic, agent activity, cross-site coverage, and expected volume qualify. Scores also require repeated observations across multiple websites and days. When an agent publishes a supported authentication method, only verified traffic is attributed to it. Each qualifying website-day contributes equally. Scores range from 0%, meaning no measurable reduction, to 100%, meaning no qualifying requests were observed where the agent was disallowed. The headline Robots.txt Effectiveness metric gives each qualifying agent equal weight. Because this is an observational estimate rather than a controlled experiment, it measures an association with robots.txt rules but does not claim that robots.txt caused every observed difference. Top Blocked Bots is calculated separately using daily robots.txt scans of Similarweb's top 1,000 websites. Identifying Spoofed Bots Spoofing statistics measure traffic from visits that claim the identity of a known agent but fail a supported authentication method, such as published IP verification or HTTP message signatures. Each agent's daily rate is calculated against total server traffic for every qualifying website, then averaged across websites. A failed check indicates that the visit was likely impersonating the named agent; it does not identify the software or operator that actually made the request. Agents without a supported authentication method are not included in these measurements. AI Chat Citations & Referrals AI chat referral statistics count directly observed human visits carrying a recognized AI platform in the referring URL or campaign source. Visits without usable referral information cannot be attributed to an AI platform. Citation statistics are estimates based on requests from agents known to retrieve content for AI platforms. Those requests indicate that content may have informed a response, but they do not confirm that a source appeared as a citation to a user. Because AI platforms do not provide a complete public record of their sources, citation results should be interpreted as directional patterns rather than exact citation counts. Frequently Asked Questions Can journalists and media organizations use this data? Absolutely. You may cite The Agentic Web Index with attribution and a link to this page. For interviews, fact-checking, background context, or a more specific breakdown for a story, contact us and include your deadline. Do you work with researchers? Absolutely. We welcome thoughtful research into how agents and bots are changing the web. Tell us about your research question, timeframe, and intended use. Depending on the scope and data constraints, we may be able to provide additional context, compare approaches, or explore a joint analysis. Can I request a specific analysis? Yes. If you need a breakdown by agent, operator, activity type, website category, or time period that is not shown here, contact us. When the underlying data supports it, we can examine the question and provide a focused analysis. How do I see these trends on my own website? Agent Analytics shows which agents and bots visit your website, what they access, and how their activity changes over time. AI Chat Referral Tracking measures the human traffic arriving from AI chat platforms. Automatic Robots.txt helps manage which bots can access your content.