Insight Monitor Configuration

An Insight is a collection of anomalies which can be manually or automatically created. They tell the story of your data.

There are two types of Insight that can be automatically created.

  • Influencer Insights - these are a collection of anomalies that share the same influencer.
  • Bucket Insights - these are a collection of anomalies that occur close in time.

Insight Monitors are scheduled searches that automatically create Insights from anomaly search results. Use your system domain knoweldge to configure the Insight Monitor to capture Insights that are important to you. For example, Insights can be triggered to be created when anoamlies occur in KPI measures. Additionally these KPI Insights can be configured to include context from associated anomaly searches which aide in troubleshooting.

To create a new Insight Monitor

  1. In the top level menu, click on Configure then Insight Monitors
  2. Select New Search and a tabbed dialog will open

Details

Field Description
Monitor name Unique name for the Insight monitor. Must not contain whitespace or characters that cause problems in file names. Cannot be edited.
Display name Description for this rule. Should be display-friendly.
Monitor type

Select Influencer or Bucket.

This defines the type of Insight monitor and subsequent configuration options.

Search names

Names of searches whose results will be scanned for Insights for the specified influencer type. Multiple searches should be selected.

Note: A search may be defined as “context only”. This is a non-scoring search, i.e. its anomalies will be included in the Insight to provide contextual information, however its score will not be used to trigger or score the Insight.

Influencer

Influencer Monitors only.

The influencer field that will be used to search for Insights. e.g. user. Only one influencer may be specified.

Number of evaluated scoring searches

Applicable to Bucket Insights only.

The number of selected, scoring searches that will be used to trigger and score the Insight, where a scoring search is one that is not marked for “context only”.

For example if set to 2 and several searches are being monitored, then the average of the top 2 scoring searches will be used to calculate its score and trigger Insight creation.

Insight score threshold

The threshold that determines whether an Insight is created.

For Influencer Insight, this is the sum of influencer scores for the maximum Insight span. The value could be greater than 100 (and generally should be to avoid Insights containing single anomalies).

For Bucket Insights, this is the anomaly score calculated for the longest bucket span of the searches being monitored. The value must be in the range 25-100.

Configuration

Field Description
Insight description The description given to the Insight. For Influencer Insights, may contain placeholders for string substitution e.g. $clientip$
Maximum Insight span

This is the maximum interval between the earliest and latest anomalies within an Insight. If this interval is exceeded, a new Insight will be created.

For Bucket Insights a span of up to a few hours is typical. When the Insight is created, we look back for half of the maximum Insight span. Any anomalies found in this period are added to this Insight. New anomalies will be continually added to the Insight until the maximum Insight span is reached.

For Influencer Insights a span from a few hours to several days is more common. We look back for the whole of the maximum Insight span. If the sum of the influencer scores exceeds the Insight score threshold then an Insight will be created. New anomalies will be continually added to the Insight until the maximum Insight span is reached.

Default 4 hours.

Append window

Applicable only to Influencer Insights.

If new Insights are identified, we will attempt to merge them with an existing Insight, should one already exist with a status set to new. This is the maximum time gap within which to append.

Default 1 hour.

Base severity for Insights

Applicable only to Influencer Insights.

Sets the severity of Insights created by this rule. The Insight score will be calculated based on the anomalies contained within the Insight. The value will fall into the range defined by the chosen base severity.

Default warning.

Run Mode

Sets the mode in which to run automatic Insight creation searches.

Field Description
Continuously

Select this to continuously look for new Insights, analyzing new data as it becomes available.

Run LookBack for:
 Select the length of time, looking back from today, for which to search existing anomaly results. This happens once, prior to running continuously.
Start running continuously after LookBack:
 Select this to run ongoing Insight monitors.
Historical

Select this to run the analysis over a fixed time range. Unlike Anomaly Searches, you may re-run an analysis over the same or earlier time periods.

Earliest:Set the start time for analysis
Latest:Set the end time for analysis

Click Calculate time span to automatically populate Earliest and Latest based on the available anomaly results.

Click Estimate insight count to show the approximate number of Insights that are likely for the fixed time range. This estimate does not take content filters, merge rules or existing Insights into account.

Insights are created and scored using the initial anomaly score, which is the value that was calculated based on this data being the most recent seen. Insight scores are not re-normalized to account for behaviors seen in more recent data. Anomalies however, that are contained within each Insight, are subsequently be re-scored and may be higher or lower. This effect is most evident during start up or after significant changes in state, when the models have had less time to learn.

If re-running with historical data using the same configuration for the same time period, only deleted Insights will be re-created. The re-run will not attempt to duplicate Insights, and will only make changes to previously created Insights if you have significantly changed the configuration of the monitor.

In order to optimize the performance of historical Insight monitors, evaluating Insight creation is performed far less frequently than when running continuously in real-time. The effective Insight Monitor frequency is set to half the Maximum Insight span, instead of the default 1 minute. This may lead to Insight start and end points being slightly shifted in time.

Alerting

Splunk Server settings > Email settings are required to be correctly configured in order for email alerts to be sent.

Field Description
Enable email alerting

Sends an HTML email when the Insight Monitor creates or updates an Insight. Toggle email alerting on or off.

Default off.

Email to The email address(es) of the alert recipient(s). Separate multiple addresses with commas.
Email CC The email address(es) to be copied on the email. Separate multiple addresses with commas. Leave blank if nobody is to be copied.
Email subject The email subject line. May contain string substitution for “$score$”. Additionally for Influencer Insights, may contain influencer string substitution e.g. $user$ or $clientip$
Email body preamble Introductory text which will be included at the beginning of each email. If this is left blank the email body will contain just the Insight summary. Note: this does not accept string substitutions.
Alert throttle period

The minimum period between alert emails for the same Insight.

Default 1 hour.

Alert during lookback

Use with caution. Sends an email when Insights are created or updated during the lookback period. Only use if you are certain that only a small number of Insights will be created. Useful for test and demonstration purposes.

Default off.

View link locale The locale that will be used for embedded links contained within the email body, e.g. en-US.

Advanced Settings

It is unlikely that these advanced settings will need to be changed. If required, please contact support@prelert.com before changing.

Field Description
Content score threshold

Anomalies with an initial score that is lower than this threshold will not be included as Insight content.

Default 3.

Content filter

You may wish to add an optional Splunk search fragment that will restrict which results may be added to Insights.

This operates on the fields available in Prelert record results. This setting only affects which anomalies are added to the Insight; it does not affect the decision of whether to create an Insight.

For example, search NOT product="test"

Minimum number of content-contributing searches

Only create an Insight when at least n of the selected anomaly searches will contribute anomalies to it. This is applied after the content score threshold and content filter.

This setting is useful when you want to ensure that an Insight contains both a potential problem and a possible explanation.

Insight monitor frequency

Applicable only when running continuously in real-time.

Interval between Insight monitor scheduled searches when running continuously in real-time. Increasing this will reduce timeliness of Insight creation. Decreasing this will reduce the number of scheduled searches Splunk has to run.

Note: When running on historical data, the effective Insight monitor frequency is half of the Maximum Insight span.

Default 1 min.

Results index

The index that stores Prelert results.

Default is prelertresults.

Insight collection

The KV store collection in which editable Insight fields are stored.

Default is prelertinsights.

Worked Examples

1. Malicious user activity

User activity is being analyzed for unusual behaviors across several data sources.

  1. Unusual login locations for users
  2. Unusual logins by time of day by users
  3. Unusual data exfiltration amounts by users
  4. Unusual login failures rates for users

Four anomaly searches have been configred on Windows authentication logs and firewall proxy log data, which all share a common “username” field. The influencer for the above anomaly searches is set to the shared field “username”.

Insights: Create an Influencer Insight to alert if a user’s cumulative behavior is anomalous in multiple ways over an extended period. Allow for gaps in anomalous behavior of up to 1 week.

2. Application monitoring using a KPI

A company sells stationery products and office supplies over the web. The following activity is being monitored for unusual behaviors.

  1. Orders made per minute for each product
  2. Performance metrics for application servers e.g. network, CPU, disk activity
  3. Log data for application, database and web servers

Anomaly searches have been configured and are running in real-time, with the main key performance indicator (KPI) of the system defined by the orders made per min.

The performance metrics may exhibit anomalous behavior which is not neccessarily indicative of a system failure, for example a high load which is subsequently load balanced. You may be monitoring this sparately. In this case, the anomalous behavior in performance metrics becomes more interesting when it happens close in time to the KPI issue. Similarly, unusual log messages are useful troubleshooting contextual information in the time leading up to a significant drop in orders.

Insights: Create a Bucket Insight Monitor to alert if KPI orders has a critical anomaly in the low count of orders per minute. Include performance metrics and application log data for the preceding 2 hours as troubleshooting information.

3. Application monitoring without a KPI

A web hosting company is keen to provide good SLAs and uptime. The hosting company does not know anything about the applications that their customers are running, and is required to look for generic issues by monitoring generic counters.

  1. Performance metrics for servers
  2. System latency
  3. Unusual rates of HTTP error codes
  4. Rare URL access
  5. Unusual data volumes in/out

Generic anomaly detection searches have been created for the above. Data rates are high, and experince has shown that issues are fairly short lived.

Insights: Create a Bucket Insight Monitor to alert if anomalous behavior occurs close in time, across some or all of these measures. Include troubleshooting information for the preceding 1 hour.

Example Configuratuions

The following shows example configurations for each of the scenarios described above. Specfic settings and search names should be adapted for your environment.

Field Example 1 - malicious user Example 2 - KPI orders per min Example 3 - generic metrics
Monitor name users_monitor kpi_monitor platform_metrics
Display name Malicious users KPI Orders per min Hosting platform metrics
Monitor type Influencer Bucket Bucket
Influencer username n/a n/a
Anomaly search names login_locations (scoring), logins_time_of_week (scoring), data_ex (scoring), login_failures (scoring) low_orders_per_min (scoring), cloudwatch_metrics (context only), app_log_data (context only) perf_counters (scoring), latency (scoring), http_error_counters (scoring), rare_urls (scoring), data_volumes (scoring)
Number of scoring searches 1 1 2
Score threshold 110 75 70
Insight description Possible malicious user $username$ KPI low orders Hosting platform issue
Insight monitor frequency 10 mins 1 min 1 min
Max Insight span 4 weeks 4 hours 2 hours
Append window 1 week n/a n/a
Base severity for influencer Insights Major n/a n/a
Min # of content-contributing searches 3 1 1

See also