NGINX Lua Modules

Without additional backend development, HUMAN Security's BotGuard for Applications product can be integrated into an NGINX reverse proxy using the lua-nginx-module.

Prerequisites

  • NGINX that has been installed with the LuaJIT. We recommend OpenResty's LuaJIT2, but there may be better alternatives depending on your operating system.
  • LuaRocks - the lua package manager.
  • A backend service protected by NGINX.

Checklist

This is a list of things worth checking before you start onboarding. Please speak to your HUMAN representative about this.

  • Do you use multipart uploads, or have binary file upload capabilities that you wish to protect? We support all upload types as long as you are configured to use signal cookies, or signal headers. However, if you are configured to use signals injected into the body of requests, we do not support multipart file uploads.
  • Does your server backend do gzip compression? We cannot inject scripts into HTML responses if the backend server is compressing the response. We advise that you compress at the NGINX level rather than at the server level. If you cannot disable this, then you will need to inject the script tags manually into your responses.

Getting Started

Your account manager can provide a software package which includes the following files:

  • README.md - contains this information and a link to this documentation
  • nginx.example - an example nginx config file for use with envsubst (see documentation)
  • lua-plugins/injector.lua - injects a script into the <head> of an HTML document
  • lua-plugins/mitigation.lua - requests an ACTION from the mitigation API
  • lua-plugins/mitigation/ - contains the plugin modules
  • lua-plugins/tests/ - contains the unit tests

In your environment you will need to use lua-rocks to install some libraries

luarocks install lua-resty-http \
&& luarocks install lua-resty-cookie \
&& luarocks install lua-resty-session

Configuration

Although most of the plugin variables can be easily configured in NGINX server blocks, individual NGINX http blocks may require some specific configuration.

  1. The release contains the required BotGuard for Applications Lua modules. Once the Lua modules are copied on to the file system, NGINX must be configured with their location. The below NGINX configuration example snippet, which should be located within an http{} block, indicates that the Lua modules are located at /etc/lua-plugins/.

    lua_package_path "/etc/lua-plugins/?.lua;;";
more_clear_headers Server;
server_tokens off;
    • Place this in the http{} block of your NGINX configuration.
    • The lua_package_path should be set to the directory in which themitigation/ directory (mentioned above) resides. In our case the lua-plugins directory was decompressed into the /etc/ directory, and as the mitigation/ directory resides there, this is where we point Nginx to.
    • The following two lines:
    more_clear_headers Server;
server_tokens off;

    are optional, but tell NGINX to not set the Server header so at not to give away the configuration. Note however, this would require NGINX to be compiled with the more_clear_headers flag set. We don't recommend exposing information about the NGINX server so this might be something to consider.

  2. DNS resolution. Lua needs to be given explicit capability by NGINX to be able to do DNS resolution. We recommend setting this within the server block of the application you wish to protect. You can use whichever resolver you wish, in this case we are using Google's DNS server however any that is publically available will work.

    resolver 8.8.8.8;

  3. Server block configuration:

    Configuration Required Type Default Example Description
    $block_redirect_status_code true string "307" "302" The HTTP code that you would like to be sent with the redirect. If this is not set, the code will be 307 Moved Temporarily
    $block_redirect_url false string / /error This is the url that any blocked request will be redirected to. If it's not set, then the Mitigation API lua plugin will redirect to the root of the domain.
    $block_spa_response_body false string '{"error":"bad request"}' '{"error":"you have entered an incorrect password"}' Specifies the default body to respond with on blocking an SPA request.
    $block_spa_response_code false string "400" "200" Specifies the http response code that will accompany the SPA response payload.
    $custom_fields true string NONE '{"some_field": "some_value"}' Custom fields sent to the Mitigation API.
    $detection_tag_ci false string NONE CUSTOMER_ID HUMAN Security will have provided you with a customer ID. Although this is not a secret, we recommend setting it as an environment variable, however it is fine to hardcode this value.
    $detection_tag_dt false integer NONE DETECTIONTAGID You will have been provided with a TAG ID. You can set this within the nginx.conf here.
    $detection_tag_host false string NONE sub.example.com This is a host that either is a CNAME on your organisation's domain that points to HUMAN Security's Mitigation engine, or a domain that HUMAN Security has provided you with. In either case, the actual domain should be obfuscated.
    $detection_tag_mo false string "2" "2" This is the tag mode. This should be always "2" for active interception.
    $detection_tag_path false string NONE /ag/CUSTOMER_ID/clear.js The path that the tag will live at. This is another obfuscated path that will either be configured on a CNAME at your domain, or will be provided to you by HUMAN Security.
    $detection_tag_si false integer NONE SITE_ID An identifier set by the customer (you) to identify the site internally.
    $detection_tag_spa false string "0" "0" Specifies if the integration is within a single page application or not.
    $global_override false string "" "true" Enabling this will bypass all mitigation and scraping protection. Use this to quickly flip the state of the mitigation interception.
    $log_prefix false string "MITIGATION-API" "SCRAPING-API" Certain logging information (e.g. decision making) will be prefixed with this.
    $mitigation_api_et true string NONE 1 A value representing the type of interaction / transaction to be protected.
    $mitigation_api_key false string NONE API_KEY HUMAN Security will have provided you with an API Key. This value should be set as an environment variable and not be hardcoded.
    $mitigation_api_policy_name false string NONE allow_for_login A policy that the customer has set up in the mitigation API.
    $mitigation_api_ssl_verify false string "true" "false" Disable verifying SSL connection certificates.
    $signal_headers false integer "1" "1" Informs the BotGuard tag whether or not to use signal headers as opposed to injecting headers within the payload of a request. Set to "0" to disable signal headers.

    Note about signal headers

    • If you are using signal headers, you need to allow underscores in headers. Add the following to your server block:
      underscores_in_headers on; #required for signal headers
    • NGINX by default doesn't allow headers of more than a certain size. It is necessary to define larger headers and payloads as part of your server blocks. The size of uploads are up to you, here it is set to 100M.
      # buffers for headers and body
  client_header_buffer_size 512k;
  large_client_header_buffers 64 512k;
  client_max_body_size 100M;
  proxy_busy_buffers_size   512k;
  proxy_buffers   4 512k;
  proxy_buffer_size   256k;

  4. Remote Address - depending on how your environment is set up, it can be useful to add the following to your server blocks so that NGINX will set the headers correctly. The correct values of client's IPs are required as part of requests to the Mitigation API. In the case where the client is proxied, this will expose their real IP address as opposed to the IP address of the proxy.

    proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;

  5. Injecting the script tag in HTML responses - the modules will attempt to inject the script tag into any HTML response. Add the following to your server block in order to reset the content header so that NGINX will recalculate it, and define the script that will do the injection.

    header_filter_by_lua_block {
    ngx.header.content_length = nil;
}
body_filter_by_lua_file /etc/lua-plugins/injector.lua;

  6. Intercepting the request and forwarding data to the Mitigation API - this next block tells to NGINX to pass any request to the mitigation API. The plugin itself will decide whether it should act on the request based on the method. Currently, only POST, PUT and PATCH request methods are supported.

    lua_need_request_body on;
access_by_lua_file /etc/lua-plugins/mitigation.lua;

    The following table lists the expected response of the plugins using default configuration settings.

    Situation Default Code Default Response Default Effect Headers Set
    action = block 307 Redirects to headers["Referer"] or / depending on whether or not headers["Referer"] is set Client is redirected to root path X-HMN-MITIGATION-RESULT
    action = allow N/A Let request through Client continues to backend route X-HMN-MITIGATION-RESULT
    Error during processing N/A Let request through Client continues to backend route. Customer to decide what to do X-HMN-MITIGATION-ERROR
    Unexpected Status Code N/A Let request through Client continues to backend route. Customer to decide what to do. Header informs client backend of the received status code (expected status code is 200) X-HMN-MITIGATION-STATUS

SSL Verification

SSL verification will be the default as part of your NGINX configuration. This is slightly beyond the scope of this documentation, however if your server has the ca-certificates package installed and you can generate a signed pem certificate you can get NGINX to verify SSL connections to the mitigation engine.

On Linux this would involve something like the following:

apk update && apk --no-cache add ca-certificates && rm -rf /var/cache/apk/*
cp /nginx-selfsigned.crt /etc/ssl/certs/nginx-selfsigned.pem
update-ca-certificates

Once that is done, you can add the following to your NGINX http block before your server block:

lua_ssl_verify_depth 2;
lua_ssl_trusted_certificate /etc/ssl/certs/nginx-selfsigned.pem;

Alternatively we have added the ability to disable SSL verification if you trust the connection between your server and your DNS resolver. If that is the case, you can set the environment variable to disable SSL verification in your server block:

set $mitigation_api_ssl_verify "false";

Using environment variables in Nginx configurations

It can be useful to set the parameters based on environment variables. In the following example the parameters are set by templating the values from environment variables:

# required variables
set $mitigation_api_key "$MITIGATION_API_KEY";
set $detection_tag_ci "$DETECTION_TAG_CI";
set $detection_tag_dt "$DETECTION_TAG_DT";
set $mitigation_api_et "1";
set $detection_tag_si "12345";
set $detection_tag_host "$DETECTION_TAG_HOST";
set $detection_tag_path "$DETECTION_TAG_PATH";
set $detection_tag_spa "0";
set $detection_tag_mo "2";

# optional variables
set $block_redirect_url "$BLOCK_REDIRECT_URL";
set $block_redirect_status_code "$BLOCK_REDIRECT_STATUS_CODE";
set $custom_fields "$CUSTOM_FIELDS";

The right hand side values get replaced with environment variables using envsubst. Below is a very basic example of using envsubst to replace two specific variables in a conf file and output a new conf file:

envsubst '$DETECTION_TAG_CI $DETECTION_TAG_DT' < nginx.conf.template > nginx.conf

Examples

All examples and conf files will need the following set:

set $mitigation_api_key "API_KEY"; # your API key
set $mitigation_api_et "1"; # the event type
set $detection_tag_ci "CUSTOMER_ID"; # your customer ID
set $detection_tag_dt "DETECTION_TAG_ID"; # your tag ID
set $detection_tag_si "SITE_ID"; # a site identifier, specified by the customer

Catch All

The following is the most basic example. It will send all non-GET requests to the mitigation api and inject the script tag on all responses that contain a </head> and/or <body> tag. This assumes that you have unzipped the release to /etc/lua-plugins.

worker_processes auto;
pcre_jit on;

events {
    worker_connections 1024;
}

http {

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    include mime.types;
    default_type application/octet-stream;
    gzip on;

    access_log /dev/stdout;

    lua_package_path "/etc/lua-plugins/?.lua;;";
    more_clear_headers Server;
    server_tokens off;

    server {
        listen 3000;
        server_name some.example.com localhost;
        resolver 8.8.8.8;
        client_header_buffer_size 8k;
        large_client_header_buffers 8 64k;
        error_log /dev/stdout debug;

        # generic properties
        set $log_prefix "MITIGATION-API";
        set $global_override "$MITIGATION_GLOBAL_OVERRIDE";
        set $global_debug "$MITIGATION_GLOBAL_DEBUG";

        # required variables
        set $mitigation_api_key "API_KEY";
        set $detection_tag_ci "CUSTOMER_ID";
        set $detection_tag_dt "DETECTION_TAG_ID";
        set $mitigation_api_et "1";
        set $detection_tag_si "SITE_ID";
        set $detection_tag_host "sub.example.com";
        set $detection_tag_path "/ag/CUSTOMER_ID/clear.js";
        set $detection_tag_spa "0";
        set $detection_tag_mo "2";

        #signal method
        set $signal_headers "1";

        location ~* \.(?:ico|css|js|gif|jpe?g|png|woff2|woff|ttf)$ {
            root /usr/share/nginx/html;
            index index.html index.htm;
        }

        location ^~ / {
            default_type text/html;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;

            header_filter_by_lua_block {
                ngx.header.content_length = nil;
            }
            body_filter_by_lua_file /etc/lua-plugins/injector.lua;
            lua_need_request_body on;
            access_by_lua_file /etc/lua-plugins/mitigation.lua;

            proxy_pass http://localhost:$BACKEND_PORT;
        }
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
            root html;
        }
    }
}

Route Management

The following example is very similar to the one above, however it defines some major differences which are listed below and commented within the example for easier reading.

  1. Variables that are shared between endpoints are part of the server, and not location block.
  2. An NGINX location block to handle signup attempts that will be redirected to if a signup is blocked by the mitigation API.
  3. The /signup route is a vanilla HTML/CSS website and redirects a blocked user to a /catch endpoint, however it informs the client that the redirect is a 200. This code is configured to deceive the client rather than inform them.
  4. Different routes to define different configurations for /login vs /signup.
  5. The /login route is an SPA and defines a response code and body to respond with when a request is blocked.
worker_processes auto;
pcre_jit on;

events {
    worker_connections 1024;
}

http {

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    include mime.types;
    default_type application/octet-stream;
    gzip on;

    access_log /dev/stdout;

    lua_package_path "/etc/lua-plugins/?.lua;;";
    more_clear_headers Server;
    server_tokens off;

    server {
        listen 3000;
        server_name some.example.com localhost;
        resolver 8.8.8.8;
        client_header_buffer_size 8k;
        large_client_header_buffers 8 64k;
        error_log /dev/stdout debug;

        location ~* \.(?:ico|css|js|gif|jpe?g|png|woff2|woff|ttf)$ {
            root /usr/share/nginx/html;
            index index.html index.htm;
        }

        # 1. Variables that are shared between endpoints are part of the server, and not location block
        set $mitigation_api_key "API_KEY";
        set $mitigation_api_et "1";
        set $detection_tag_ci "CUSTOMER_ID";
        set $detection_tag_dt "DETECTION_TAG_ID";
        set $detection_tag_host "sub.example.com";
        set $detection_tag_path "/ag/CUSTOMER_ID/clear.js";


        # set a policy if there is one that you would like to apply
        set $mitigation_api_policy_name "$MITIGATION_API_POLICY_NAME";

        #signal method
        set $signal_headers "1";

        location ^~ /catch {
            add_header Content-Type text/html;
            header_filter_by_lua_block {
                ngx.header.content_length = nil;
            }
            body_filter_by_lua_file /etc/lua-plugins/injector.lua;
            return 200 '<html><body><h1>You have been caught!</h1></body></html>';
        }

        # 2. An NGINX location block to handle signup attempts that will be redirected to if a signup is blocked by the mitigation API
        location ^~ /signup {
            default_type text/html;

            # required variables
            set $detection_tag_spa "0";
            set $detection_tag_mo "2";
            set $detection_tag_si "SITE_ID";

            # 3. The /signup route is a vanilla HTML/CSS website and redirects a blocked userto a /catch endpoint, 
            # however it informs the client that the redirect is a 200. This code is configured to deceive the client rather than inform them.
            set $block_redirect_url "/catch";
            set $block_redirect_status_code "200";

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;

            header_filter_by_lua_block {
                ngx.header.content_length = nil;
            }
            body_filter_by_lua_file /etc/lua-plugins/injector.lua;
            lua_need_request_body on;
            access_by_lua_file /etc/lua-plugins/mitigation.lua;

            proxy_pass http://localhost:$BACKEND_PORT;
        }

        # 4. Different routes to define different configuration for /login vs /signup
        location ^~ /login {
            default_type text/html;

            # required variables
            set $detection_tag_spa "1";
            set $detection_tag_mo "2";
            set $detection_tag_si "SITE_ID";
            # 5. The /login route is an SPA and defines a response code and body to respond with when a request is blocked in the case of an SPA
            set $block_spa_response_code "200";
            set $block_spa_response_body '{"success":"you are now logged in"}';

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $remote_addr;

            header_filter_by_lua_block {
                ngx.header.content_length = nil;
            }
            body_filter_by_lua_file /etc/lua-plugins/injector.lua;
            lua_need_request_body on;
            access_by_lua_file /etc/lua-plugins/mitigation.lua;

            proxy_pass http://localhost:$BACKEND_PORT;
        }

        error_page 500 502 503 504 /50x.html;

        location = /50x.html {
            root html;
        }
    }
}