Evaluation Mode

Evaluation Mode allows you to quickly assess the effectiveness of anomaly detection configurations. It allows you to try out different configurations without making any permanent changes.

For example, you can test out your syntax or tune the alert rates using:

  • different values for bucketspan e.g. 5 mins compared to 30 mins
  • different functions e.g. count or non_zero_count
  • combinations of functions e.g. sum and max
  • different search filters e.g. where state==NY

Example: Event rates by log_level in Splunkd logs

This example requires privileges to read the Splunk _internal index.

  1. Select menu Tools / Evaluation Mode

  2. Paste in the following Splunk search, followed by the prelertautodetect command:

    index="_internal" sourcetype="splunkd"  log_level="ERROR" OR log_level="WARN" | prelertautodetect count by log_level
    

3. Select a suitable time range - consider the time to run and that enough data is present for analysis. In this example, we would recommend selecting the last 7 or the last 30 days.

  1. Run the search

This analysis will detect when unusual counts of log messages occur, it will:

  • Model the count of messages by log_level over time, starting with the most recent events
  • Detect when a count is unusual compared to its past behavior
Evaluation Mode results

By selecting a row in the bottom right hand panel, further charts and tables will be displayed below showing the details of the anomalies alongside source data.

prelertautodetect

For more infomation, please navigate to menu About / Command Reference

Time series analysis bucket interval

In Evaluation Mode, a good bucketspan is chosen based on the first 10,000 events. If less than 10,000 events are returned by the search, a popup will be displayed and results will not be returned. As a workaround, please define a known bucketspan e.g. 600 seconds.

Drilling through to source data

The bottom three rows of panels show anomaly details and source data for the time interval selected. Click on a row in the bottom table to drill through to further source data which will be displayed showing 10 time buckets either side of the anomaly. As this is a standard Splunk search page, it can be manually edited to help with further troubleshooting and analysis.

Limitations

Results are not stored - In Evaluation Mode, the results and configurations are temporary i.e. they are not saved. Therefore once you have closed the page the results are no longer available to view.

Data is analyzed in reverse time order - Evaluation Mode analyzes the results passed to it from the Splunk search. As these are presented in reverse time order, the data is modeled starting with the most recent events first. This will produce somewhat different results to using an Anomaly Search which goes forwards in time.

Raw and unnormalized scores - Evaluation Mode uses a raw anomaly score which enables assessment of relative anomalousness but does not extend to a severity definition i.e. critical, major, minor

See also