crawlers

AI agents are starting to sign their requests under the Web Bot Auth drafts at the IETF, ChatGPT among them. The 51Degrees agent signature properties report whether the proof checked out, so you can let a verified agent in and keep the copycats out.

This blog explains how Naver’s web crawlers, including Yeti, Ads-Naver, and Blueno, interact with website content. It outlines their different roles, the crawler usage values assigned to each, and how they affect content visibility across Naver’s search and services.

This blog explains how DuckDuckGo’s web crawlers, including DuckDuckBot and DuckAssistBot, interact with website content. It outlines their different roles, how they follow robots.txt directives, and the management of crawler access and content visibility.

This blog explains how Apple’s web crawlers, including Applebot and Applebot-Extended, work across its ecosystem to surface content in Siri, Spotlight, and Safari. It outlines the different crawlers, their purposes—such as search indexing, feed updates, and AI training—and how enabling them can improve content visibility, traffic, and control over how content is used, particularly in generative AI.

Find out how we classify Perplexity's AI web crawlers.

Find out how we classify Meta's crawlers and AI content scrapers.