A crawler visiting a website can advertise where it came from via the User-Agent string. A product token can be added to determine the name and vendor, and a URL provided to give more information to the website. Here are some examples from popular crawlers.
| Crawler | Operator | User-Agent |
|---|---|---|
| Googlebot | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
|
| Bingbot | Microsoft | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 |
| GPTBot | OpenAI | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot |
| Applebot | Apple | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| PerplexityBot | Perplexity | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
Each string is as its operator publishes it. Bing puts the current Microsoft Edge version where W.X.Y.Z is.
But there's a problem. User-Agents can be spoofed and there's no guarantee that the request is truly being operated by the entity claimed. This has become a problem for good crawlers that want to ensure website operators know they are who they claim to be.
Web Bot Auth is the answer and 51Degrees has made it super simple to consume.
The crawler operator signs each request with a private key only it holds and publishes the matching public key at a well-known address on its own domain. If the signature checks out against that public key, the agent's private key signed the request. The 51Degrees agent signature properties report the result.
Who signs today
OpenAI's ChatGPT agent does. That's the agent that browses on a person's behalf, not GPTBot from the table. It publishes its key at chatgpt.com/.well-known/http-message-signatures-directory. Cloudflare, whose engineers wrote the first drafts, runs a directory of its own in the same format. The protocol is an IETF working group draft in two parts, one on how to sign requests and one on how to publish keys.
GET https://chatgpt.com/.well-known/http-message-signatures-directory
{
"keys": [
{
"crv": "Ed25519",
"kty": "OKP",
"x": "P_1c4SX6fCavDcq6FRegWcVvaOKsp_aXHPEN1FcKDIA",
"kid": "ArwjqcCqtA5oxEWkzXGC-To_0A9whHmajCHtfe8I9aY",
"use": "sig",
"nbf": 1735689600,
"exp": 1793801648
}
],
"signature_agent": "https://chatgpt.com",
"purpose": "ai"
}exp), and OpenAI keeps moving that date forward, so the one you fetch will differ.Not many crawlers sign yet. We expect that to change and in the near future the absence of a signature will be a sign that the crawler is not legitimate.
Checking the signature
Three extra headers travel with a signed request: Signature, Signature-Input, and Signature-Agent, the last naming the agent's domain. 51Degrees open-source packages fetch the agent's public keys from that domain, cache them, and check the signature. After the first fetch the check is a cache hit and cheap enough to run on every request where the signature is present.
We tried to fool it. We signed a request with the test key published in RFC 9421 and said it came from ChatGPT. Our check fetched OpenAI's real key directory, found no key that matched, and answered Invalid with the reason UnknownKey. When the same request named Cloudflare's research service instead, which publishes that test key, it read Verified.
The check fits RFC 9334, the IETF's architecture for remote attestation. An attester produces signed evidence about itself, a verifier checks that evidence, and a relying party acts on the result. Here the agent is the attester, 51Degrees is the verifier and your website is the relying party. Each agent publishes its own key on its own domain, and no central body hands them out.
That's decentralized and we like decentralized. 😊
We think this pattern will spread well beyond bots. Ad requests are an obvious next step, where a publisher's server could prove to a buyer that a bid request really came from it.
The properties
The main property, AgentSignature, always has one of five values.
- Verified. The signature checked out against the agent's published key.
- Invalid. A signature was sent and failed, for example expired, malformed, or signed for another site.
- Unverified. A signature was sent but could not be checked, for example because the agent's key directory could not be fetched.
- Timeout. The key directory was still being fetched when the answer was needed. The fetch carries on, so the next request from the same agent finds it cached.
- Absent. No signature. Most traffic today.
AgentSignatureReason says why, with seventeen codes. If the agent publishes an agent card, a description of itself, you also get its name, the token it answers to in robots.txt, and the purpose it declares. The other properties cover the key, the algorithm, the nonce, and the timing. All twelve are in the Property Dictionary.
Telling the real ChatGPT from a copy
- A verified ChatGPT agent. Someone asked it to fetch the page, OpenAI says, so treat it as a visitor. Send it to your AI notice or license terms, as our own site does at 51degrees.com/ai-notice, rather than a CAPTCHA or a block.
- Anything else calling itself ChatGPT. An allow list keyed on ChatGPT's User-Agent lets in anyone who copies the string. With the signature checked, a copy never reads Verified, so it gets no special treatment and your IsCrawler and IsArtificialIntelligence rules decide what happens to it.
What we don't like about purpose
Another draft, the Signature Agent Card, lets an agent describe itself. The working group hasn't adopted it yet. One of the things a card can carry is a purpose, "the intended use of collected data". ChatGPT already puts a purpose in its key directory, shown above. It says ai.
What does ai allow? Training a model, or answering one person's question? The draft doesn't say. It sets no values and offers the IETF's AI preferences vocabulary as "one possible vocabulary". The first draft of that list appeared in March 2025, and the current one still says its contents do not reflect the consensus of the working group.
We told the IETF in August 2025 that licensing can't be reduced to a few words. Copyright is more complicated than that. Back then the list covered training a model and said nothing about grounding, where an AI reads someone's article to build its answer.
Nothing makes the agent stick to its purpose either. The draft calls purpose "a claim made by the Signature Agent" and says clients "MUST NOT treat these values as verified behavior". So there's no contract, and no remedy if the agent does something else with your content.
A one-word purpose isn't a license. A license says what's allowed and on what terms, and a website can publish its own and point to the exact version. The Movement for an Open Web does that with the Search Only Contract, which permits search and needs a separate license for AI use.
Not a human check
The working group's own draft says the protocol "does not authenticate human users". Whether a person is reading the page is a separate question, and the HumanConfidence property, returned by the same call, answers it. Today an absent signature says nothing bad about a sender, because most agents and every browser send none.
Getting it
The twelve agent signature properties are in the Bespoke plan of the cloud service, where you ask for them by name like any other property. On-premise, the agent signature element ships in the .NET Pipeline, and one builder call adds it.
var pipeline = new PipelineBuilder(loggerFactory)
.AddFlowElement(new AgentSignatureElementBuilder(loggerFactory).Build())
.Build();
Every request the pipeline processes then carries the properties.
What's still missing
Web Bot Auth is a good step in the right direction. You can now tell the real ChatGPT agent from a copy, which a User-Agent string never let you do. It still doesn't tell you what OpenAI will do with your content once the agent has fetched it.
How does one company trust another? Contracts.
Find out more
The properties are listed in the Property Dictionary. For the wider picture on crawlers and AI agents, including options that go beyond robots.txt, see our AI solutions page.