Policy Engine Overview

BotGuard for Application’s Policy Engine lets you configure custom rules to modify which responses are returned by the Mitigation API. For example, you can change the criteria that trigger a block or allow response, or create a unique response like mfa to prompt users with a multi-factor authentication challenge.

Some common uses of the Policy Engine include:

• Defining a list of trusted bots and allowing them to access your site.
• Creating allow lists to prevent suspected false positives from being blocked.
• Making adjustments to the default block/allow criteria.
• Applying custom actions (like mfa or throttle) to specific types of traffic.

You can create and edit policies in the Configure section of the HUMAN Dashboard or by submitting a request to the Policy Engine API.

To use a custom policy, each request you make to the Mitigation API must specify the name of the custom policy you wish to apply. If a request does not include the name of a policy, HUMAN's default policy will be applied. This default policy uses the following logic:

if decision.bot then block
default allow

When this default policy is applied, any traffic that HUMAN classifies as a bot will receive a block response; all other traffic will receive an allow response. You can configure your site or application to receive these responses from HUMAN and take action accordingly (by either allowing the event or blocking it from proceeding it further).

Policy structure

Each policy consists of a policy name, a policy version, zero or more rules, and a default action.

Each rule consists of a rule label, a match expression, and an action. Match expressions can reference HUMAN’s decisioning data or any custom data provided in the client_ds object in the request payload. Logical operators, scalar expressions, and regular expressions are supported; rules can also reference an inline list of values or an external set.

When a policy is triggered, its rules are evaluated one at a time from top to bottom. If a rule's condition is met, the action for that rule is returned and no further rules will be evaluated. If no rules' conditions are met, the default action is returned.

A policy can be up to 10KB in size. You can only apply one policy to each request, but your account may have up to 10 policies in total.

Versions

The HUMAN Dashboard saves a running history of updates to each policy. You can roll back to a previous version of a policy (for example, to undo recent updates) at any time.

Sets

If you need to reference a large number of values in a rule, you can store those values in an external list called a set. Each set can be up to 100KB in size and supports the value types ip, string, and uint; sets are most commonly used to manage lists of IP addresses or user IDs.

You can create and maintain sets through the HUMAN Dashboard or by directly calling the Mitigation API.

Example policy

You can get started with the Policy Engine by using the following example as a template. This basic policy uses one allow list (which is stored in a set) to prevent suspected false positives from being blocked, uses another allow list (also stored in a set) to prevent internal IP addresses from being blocked, and uses specific criteria to prevent your own "safe" bots from being blocked.

Example policy structure

The below example policy has four rules and a default action of allow. These four rules are evaluated from top to bottom until a match is found. Events that don't match any of these rules will receive a default response of allow.

1. The first rule, allowedUsers, checks for a match between the event's user ID (which is stored in the variable clientds.ui) and a set of allow-listed users. If the event's user ID matches any value from this list, the Mitigation API will return an allow response.
2. The second rule, allowIPDs, follows a similar logic as allowedUsers, but instead checks for a match between the event’s IP address (which is stored in the variable clientds.ip) and a set of allow-listed IP addresses. If the event’s IP address matches any value from this list, the Mitigation API will return an allow response.
3. The third rule, throttledBots, applies a custom action to any events that match the specific criteria. In this example, an event must meet two criteria to return the specified action: if an event's user agent matches a specified expression, and its IP address matches a value in a specified array, the Policy Event will return the custom action throttle. (Note: because this is a custom action rather than a default action, its return value follows the syntax of "action": "throttle".)
4. The fourth and final rule, blockedBots, returns a block response for any traffic that has been flagged as a bot.

Example policy syntax

The following sample shows the full body of our example policy, written in the Policy Engine’s domain-specific language:

version 1
allowedUsers:
if clientds.ui in allowed_users_set then allow

allowedIPs:
if clientds.ip in allowed_ips_set then allow

throttledBots:
if and(
  clientds.ua ~ /^*my_custom_safe_bot*$/,
  clientds.ip in ["1.2.3.4", "5.6.7.8"]
) then action(“throttle”)

blockedBots:
if decision.bot then block

default allow

When all four rules are evaluated as a group, this policy first allows any users in the set allowed_users_set, then allows any IPs in the set allowed_ips_set, then returns a custom action of "throttle" to any event whose user agent matches the regular expression ^*my_string_safe_bot*$. Any remaining users who have been flagged as a bot and did not meet the criteria of the previous three rules will receive a block response.

Advanced features

The Policy Engine’s domain-specific language (DSL) supports custom actions, multiple logical operations, and variables. The following sections contain more information about these features.

Safe bots

HUMAN maintains a list of non-malicious bots that serve specific purposes, such as automated tools used for search engine indexing, uptime monitoring, and analytics. Safe bot data is accessible on a per-event basis through the Mitigation API, or you can view more information about the safe bots comprising your traffic through the HUMAN Dashboard.

You can also use this data in the Policy Engine to create rules that allow certain types of bots to access your website or application. The following examples show several ways that you can build custom policies that use these safe bot designations.

Allow all safe bots

This sample policy will return an allow response for any bots or scripts that HUMAN has identified as safe. Any bots or scripts that have not been deemed safe will continue to receive a block response.

blockBadBots:
if and(
  decision.bot,
  not decision.entity_fingerprint.safe
) then block

default allow

Only allow crawlers

This sample policy will return an allow response for any crawlers that HUMAN has identified as safe; however, all other types of bots and scripts, including non-crawler bots that HUMAN has identified as safe, will receive a block response.

blockNonCrawlers:
if and(
  decision.bot,
  decision.entity_fingerprint.class != "crawler"
) then block

default allow

Only allow certain aggregators

This sample policy will return an allow response for certain aggregators that HUMAN has identified as safe; however, two specific aggregators (in this case, Mint and Yodlee) will still receive a block response. All other types of bots or scripts, including bots that HUMAN has identified as safe, will also receive a block response.

allowSomeAggregators:
if and(
  decision.entity_fingerprint.class = "aggregator",
  decision.entity_fingerprint.name not in ["Mint", "Yodlee"]
) then allow

blockOtherBots:
if decision.bot then block

default allow

Supported variables

The decision object and the clientds object contain a variety of variables that you can incorporate into your policies. These variables include basic information about the signals collected for each event and information about HUMAN's threat analysis.

decision

The decision object contains the following variables:

{
    "bot": "boolean",
    "error": "boolean",
    "product": "string",
    "timestamp": "int64",
    "challenge": {
        "captcha": {
            "loaded": "boolean",
            "completed": "boolean"
        }
    },
    "errorReason": "string",
    "ivtTaxonomy": {
        "botCategory": "map[string]boolean",
        "botSubcategory": "map[string]boolean",
        "factCategory": "map[string]boolean",
        "factSubcategory": "map[string]boolean",
        "threatProfile": "string"
    },
    "threatProfile": "string",
    "threatCategory": "map[string]boolean",
    "asn": "uint",
    "country": "string"
}

clientds

The client_ds object contains the following variables. For more information about each variable, see our Mitigation API documentation.

{
  "et": "string",
  "ip": "string",
  "asn": "uint",
  "country": "string",
  "mo": "string",
  "pd": "string",
  "url": "string",
  "ua": "string",
  "client_error": "boolean",
  "event_success": "boolean",
  "pw_match": "boolean",
  "server_error": "boolean",
  "user_exists": "boolean",
  "validation_error": "boolean",
  "ap": "string",
  "ck": "string",
  "custom": "map[string]string",
  "dv": "string",
  "endpoint": "string",
  "fi": "string",
  "ref": "string",
  "si": "string",
  "username": "string",
  "ui": "string"
}

Custom actions

Actions are the strings that the Mitigation API returns in its response payload. These actions describe HUMAN’s recommendation for how you should respond to any given event. By default, the Mitigation API only returns the actions block and allow, but you can use the Policy Engine to create and return custom actions of your choosing. You must define custom actions by using the syntax action("action_name"). For example, the following rule returns the custom action throttle if an event originates from one of the listed ASNs:

throttleASN:
if decision.asn in [1, 2, 3, 4] then action("throttle")

Since block and allow are HUMAN-defined responses, each of these responses has a shortcut syntax. For example, the following statements are equivalent:

• if clientds.ui in ["userID1"] then block
• if clientds.ui in ["userID1"] then action("block")

len

You can use the len operator to create rules based on the length of an array.

For example, you can determine how many threat categories were associated with an event by checking the length of the threatCategory array. Since events associated with a high number of threat categories tend to return more confident decisions than events with a low number of threat categories, you could use len to create rules that only apply to events whose threatCategory array contains a certain number of items. You can also use len to create rules based on any custom fields that contain arrays.

The following sample will only return the block action for events that were classified as a bot and were associated with three or more threat categories:

highPrecisionBlock:
if and(
  decision.bot,
  len(decision.threatCategory) > 2
) then block

samplePercent()

You can use the samplePercent() function to create rules that only run a certain percentage of the time. For example, the syntax samplePercent(74) will evaluate to true 74% of the time its criteria are met. In other words, if the rule's criteria are met 100 times, the rule will be executed approximately 74 of those times.

Every time samplePercent() is called, the Policy Engine generates a random number from 0 to 100. If the generated number is less than the value passed through samplePercent(), the rule will evaluate to true. If the generated number is greater than or equal to the value passed through samplePercent(), it will not execute. Values passed through samplePercent() must be between 0 and 100 inclusive.

The following sample uses samplePercent(10) to create a rule that will return a block response 10% of the time:

randomBlock:
if and(
  decision.bot,
  samplePercent(10) 
) then block