Bots and humans

A crawler collects pages in bulk for its operator. A scraper hides by posing as a browser. An AI agent works for one person on one task. Human Confidence returns High, Medium, or Low confidence that a person is reading the page. Add the value to your analytics before deciding what each visit can access.

The panel at the top of this page shows canned requests and the cloud's answer for each. They follow the public HumanConfidence derived property, which you can read in full and try in the script tester. Declared crawlers and browsers driven by a web driver are Low, and the answer is Unknown until the 51Degrees JavaScript has run. Otherwise it depends on whether the page was in view, how recent the browser is, and what the IP address says about the people who use it.

const CLOUD = 'https://cloud.51degrees.com/api/v4/';
const KEY = process.env.RESOURCE_KEY;

const headers = await cloudHeaders();
const evidence = new URLSearchParams();
for (const [name, value] of Object.entries(req.headers)) {
  if (headers.has(name)) evidence.set(name, value);
}
for (const [name, value] of Object.entries(req.body ?? {})) {
  if (name.startsWith('51D_')) evidence.set(name, value);
}
evidence.set('client-ip', ip);
const response = await fetch(CLOUD + KEY + '.json', {
  method: 'POST', body: evidence,
  signal: AbortSignal.timeout(TIMEOUT_MS)
});
const json = await response.json();
if (!response.ok) throw new Error(JSON.stringify(json.errors));
answer = json.derived?.humanconfidence ?? 'Unknown';

if (human.answer === 'High') return next();
if (human.answer === 'Low')
    return res.redirect('/automated-access');
res.redirect('/challenge');

CLOUD = 'https://cloud.51degrees.com/api/v4/'
KEY = os.environ['RESOURCE_KEY']

headers = cloud_headers()
evidence = {name.lower(): value
            for name, value in request.headers.items()
            if name.lower() in headers}
evidence.update({name: value
                 for name, value in request.form.items()
                 if name.startswith('51D_')})
evidence['client-ip'] = ip
body = urllib.parse.urlencode(evidence).encode()
with urllib.request.urlopen(CLOUD + KEY + '.json', body,
                            timeout=TIMEOUT_SECONDS) as response:
    derived = json.load(response).get('derived', {})
answer = derived.get('humanconfidence', 'Unknown')

if human['answer'] == 'High':
    return view(*args, **kwargs)
if human['answer'] == 'Low':
    return redirect('/automated-access')
return redirect('/challenge')

const CLOUD = 'https://cloud.51degrees.com/api/v4/';

$evidence = [];
foreach (getallheaders() as $name => $value) {
    if (isset($headers[strtolower($name)])) {
        $evidence[strtolower($name)] = $value;
    }
}
foreach ($_POST as $name => $value) {
    if (str_starts_with($name, '51D_')) {
        $evidence[$name] = $value;
    }
}
$evidence['client-ip'] = $ip;
$json = cloud(getenv('RESOURCE_KEY') . '.json', [
    'method' => 'POST',
    'header' =>
        'Content-Type: application/x-www-form-urlencoded',
    'content' => http_build_query($evidence),
]);

$answer = $json['derived']['humanconfidence'] ?? 'Unknown';

if ($human['answer'] === 'Low') {
    header('Location: /automated-access');
    exit;
}
if ($human['answer'] !== 'High') {
    header('Location: /challenge');
    exit;
}

Developers start with the Human Confidence overview and its quick start. Retailers can read what it means for e-commerce. Human Confidence comes with the Pro plan, our cloud service, or with the Bespoke plan on your own servers.

Read the Human Confidence quick start

To start identifying bot traffic, you will need our IsCrawler property. Our CrawlerName property can provide additional information on the crawler.

Googlebot and Bingbot collect pages for search indexes, and other crawlers collect them to train AI models. CrawlerUsage says what each one collects pages for, so you can decide which uses of your content you allow.

Tell crawlers your decision in your robots.txt. Our robots.txt generator writes the file from the uses you allow, and emails you an update when crawlers or their uses change.

See the crawler properties

Decide which crawlers you want. A claimed crawler name alone is not proof of who sent the request, so verify the search engine crawlers you admit before you allow access, as our post on how likely your traffic is to be from real humans explains.

Bots posing as people sometimes claim an unusual device or a very old browser. Combine 51Degrees properties into a bot filter tailored to your audience, as our guide to filtering bots from your analytics with User-Agent data shows.

• Location data and device detection can inform your access rules. Neither identifies a fraudster on its own.
• Use Human Confidence to decide whether to serve content, request a login, or restrict a visit. The value is a confidence, so choose a policy suited to your audience.
• Segment analytics by declared crawler and Human Confidence. Review those segments before treating every visit as a person.

See how our customers use real-time data

We support cloud and on-premise deployment. For cloud use, read about the Resource Key, then create a Resource Key with your chosen properties. Add your License Key when your plan requires one.

We have documentation for each API language to get you started with on-premise device detection. You can access a free trial by contacting us for more information.

Read the device detection quick start

• Various deployment options in a range of API languages, all provided open-source and free via GitHub.
• We're an ISO 9001 and 27001 certified company, so you can trust in our high-quality service.
• Our Data team update our device database of 1,295,686 combinations daily. We source our data from real devices, vendors, or optional usage sharing. We don't cut corners by scraping the web.

See the crawler database