Configuring Alerts

Alerts can be configured using Watcher which is provided as part of the Elastic Stack.

To establish alerting, a custom alert can be built to email on specific conditions for a configured job. To do this requires some prerequisite items to be configured and knowledge of the job configurations.

Prerequisites

  1. Ensure that Watcher is installed within Prelert Behavioral Analytics for the Elastic Stack

Elasticsearch Watcher 2.1 is required. Full product documentation is available here. To install Watcher with a 30 day evaluation license do the folowing:

cd $PRELERT_HOME/cots/elasticsearch
bin/plugin install license
bin/plugin install Watcher
  1. Configure watcher to send email alerts

To configure watcher to send emails, use the following and replace blocks of < > with appropriate values for your environment. Then paste it at the end of elasticsearch.yml and restart Elasticsearch:

watcher.actions.email.service.account:
prelertalerts:
email_defaults:
from: <EMAIL-COMING-FROM>
smtp:
auth: true
starttls.enable: true
host: <SMTP-HOSTNAME>
port: <SMTP-PORT>
user: <USERNAME>
password: <PASSWORD>
watcher.actions.email.html.sanitization.enabled: false

Configure an alert for a Job

Example - Alerting for DNS Tunneling

As an example, imagine a job is configured called dns_tunnel which looks for anomalies in the information content of DNS queries.

To configure alerts for this example, run the following curl command replacing <IP-OF-PRELERT-SERVER> in the body content with the address for the system running Prelert:

curl -XPUT 'http://localhost:9200/_watcher/watch/prelertdnstunnelalert' -d '{
    "trigger" : {
        "schedule" : { "interval" : "1d" }
    },
    "input" : {
        "search" : {
            "request" : {
                "indices" : [ "prelertresults-dns_tunnel" ],
                "body" : {
                    "query" : {
                        "type" : { "value": "record" }
                    },
                    "filter" : {
                        "range" : {"normalizedProbability" : {"gte" : "75"}}

                    }
                }
            }
        }
    },
    "condition" : {
        "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
    },
    "actions" : {
        "send_email" : {
            "email" : {
                "to" : [ "joe@mycompany.com", "sara@mycompany.com", "tom@mycompany.com” ],
                "subject" : "DNS tunneling anomaly detected",
                "body" : {
                    "html" : "<html><body>Possible DNS tunneling from{{ctx.payload.hits.hits.0._source.clientip}} to {{ctx.payload.hits.hits.0._source.highest_registered_domain}}.Click <a href=\\"http://<IP-OF-PRELERT-SERVER>:5601/app/prelert#/anomalyexplorer?_g%3D(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-30d,mode:quick,to:now))%26_a%3D(filters:!(),options:(darkTheme:!f),panels:!((col:1,id:Prelert-Anomaly-timeline,panelIndex:7,row:1,size_x:12,size_y:3,type:visualization),(col:1,id:Prelert-Influencers-heatmap,panelIndex:8,row:4,size_x:12,size_y:5,type:visualization),(col:1,id:Prelert-Anomaly-Summary,panelIndex:9,row:9,size_x:12,size_y:9,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:&apos;jobId:dns_tunnel\*&apos;)),title:&apos;Anomaly%2520Explorer&apos;,uiState:())&amp;_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-30d,mode:quick,to:now))&amp;_a=(filters:!(),options:(darkTheme:!f),panels:!((col:1,id:Prelert-Anomaly-timeline,panelIndex:7,row:1,size_x:12,size_y:3,type:visualization),(col:1,id:Prelert-Influencers-heatmap,panelIndex:8,row:4,size_x:12,size_y:5,type:visualization),(col:1,id:Prelert-Anomaly-summary,panelIndex:9,row:9,size_x:12,size_y:9,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:&apos;jobId:dns_tunnel\*&apos;)),title:&apos;Anomaly%20Explorer&apos;,uiState:())\\">here</a> to investigate.</body></html>"

                },
                "attach_data" : true
            }
        }
    }
}'

The email that would get sent would look like the following example:

From: support@mycompany.com
Sent: Friday, February 12, 2016 11:54 AM
To: joe@mycompany.com, sara@mycompany.com, tom@mycompany.com
Subject: DNS tunneling anomaly detected

Possible DNS tunneling from 10.77.1.12 to *base.net*. Click *here* to investigate.

This alert is configured to search across all time once per day. A real world use case would search the last few minutes and re-run every few minutes.

To force a DNS tunneling alert search to occur, run the following command:

curl -XPUT 'http://localhost:9200/_watcher/watch/prelertdnstunnelalert/_execute' -d '{
    "action_modes" : {
        "_all" : "force_execute"
    },
    "record_execution" : true
}'

To delete the DNS tunnelling alert rule:

curl -XDELETE 'http://localhost:9200/_watcher/watch/prelertdnstunnelalert'

Example - Alerting for transaction drop-off rates

To configure alerts for an Anomaly Search Job called it_ops_kpi, that detects drops in transaction rates, run the following curl command. Be sure to replace <IP-OF-PRELERT-SERVER> in the body content for the system running Prelert.

curl -XPUT 'http://localhost:9200/_watcher/watch/prelertitopsalert' -d '{
    "trigger" : {
        "schedule" : { "interval" : "1d" }
    },
    "input" : {
        "search" : {
            "request" : {
                "indices" : [ "prelertresults-it_ops-kpi" ],
                "body" : {
                    "query" : {
                        "type" : { "value": "record" }
                    },
                    "filter" : {
                        "range" : {"normalizedProbability" : {"gte" : "75"}}
                    }
                }
            }
        }
    },
    "condition" : {
        "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
    },
    "actions" : {
        "send_email" : {
            "email" : {
                "to" : ["joe@mycompany.com", "sara@mycompany.com", "tom@mycompany.com”],
                "subject" : "Anomaly in online purchases KPI detected",
                "body" : {
                    "html" : "<html><body>Online purchase rate now {{ctx.payload.hits.hits.0._source.actual}} per minute, typically {{ctx.payload.hits.hits.0._source.typical}} per minute. Click <a href=\\"http://
://<IP-OF-PRELERT-SERVER>:5601/app/prelert#/anomalyexplorer?_g%3D(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-30d,mode:quick,to:now))%26_a%3D(filters:!(),options:(darkTheme:!f),panels:!((col:1,id:Prelert-Anomaly-timeline,panelIndex:7,row:1,size_x:12,size_y:3,type:visualization),(col:1,id:Prelert-Influencers-heatmap,panelIndex:8,row:4,size_x:12,size_y:5,type:visualization),(col:1,id:Prelert-Anomaly-summary,panelIndex:9,row:9,size_x:12,size_y:9,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:&apos;jobId:it_ops*&apos;)),title:&apos;Anomaly%2520Explorer&apos;,uiState:())&amp;_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-30d,mode:quick,to:now))&amp;_a=(filters:!(),options:(darkTheme:!f),panels:!((col:1,id:Prelert-Anomaly-timeline,panelIndex:7,row:1,size_x:12,size_y:3,type:visualization),(col:1,id:Prelert-Influencers-heatmap,panelIndex:8,row:4,size_x:12,size_y:5,type:visualization),(col:1,id:Prelert-Anomaly-summary,panelIndex:9,row:9,size_x:12,size_y:9,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:&apos;jobId:it_ops*&apos;)),title:&apos;Anomaly%20Explorer&apos;,uiState:())\\">here</a>
to investigate.</body></html>"
                },
                "attach_data" : true
            }
        }
    }
}'

The email that would get sent would look like the following example:

From: support@mycompany.com
Sent: Friday, February 12, 2016 11:54 AM
To: joe@mycompany.com, sara@mycompany.com, tom@mycompany.com
Subject: Anomaly in online purchases KPI detected

Online purchase rate now 633.0 per minute, typically 3700.8 per minute. Click *here* to investigate.

This alert is configured to search across all time once per day. A real world use case would search the last few minutes and re-run every few minutes.

To force an IT OPS KPI alert search to occur, run the following command:

curl -XPUT 'http://localhost:9200/_watcher/watch/prelertitopsalert/_execute' -d '{
    "action_modes" : {
        "_all" : "force_execute"
    },
    "record_execution" : true
}'

Then to delete it use:

curl -XDELETE 'http://localhost:9200/_watcher/watch/prelertitopsalert'