Timechart Mode

Timechart Mode is designed for timechart-based Splunk searches.

Simply take an existing timechart-based Splunk search (perhaps from one of your favorite dashboards) and use this for anomaly detection.

Once results have been analyzed, you can automatically create an Anomaly Search for continuous analysis.

Example: Event rates by log_level in Splunkd logs

This example requires privileges to read the Splunk _internal index.

  1. Select menu Tools / Timechart Mode

  2. Paste in the following Splunk timechart search:

    index="_internal" sourcetype="splunkd" log_level="ERROR" OR log_level="WARN" | timechart count by log_level
    
  3. Select a suitable time range - consider the time to run and that enough data is present for analysis. In this example, we would recommend selecting the last 7 or 30 days.

  4. Run the search

This analysis will detect when unusual counts of log messages occur, it will:

  • Model the count of messages by log_level over time
  • Detect when a count is unusual compared to its past behavior
Timechart Mode results

In the above example, a critical anomaly has been raised on 2015-12-09. Between 05:55 and 06:00 the typical count of ERROR log messages is 0.03, however during this period the actual count is 20. This is an unusual count of ERRORs.

The top panel, “Timechart data”, shows the results of the Splunk timechart command.

The second panel, “Anomalies by relative severity”, displays the anomaly results. The blue line is the Anomaly Score which identifies the most anomalous time period. Select this time range on the chart to filter the results for this particular interval.

The bottom left panel, “Anomaly list by field”, shows the anomalousness of each log_level. By selecting a row in this table, the individual anomaly results will be displayed in the bottom right panel. From here, expand the row for further details, or click on the row to drill through to the source data.

Time series analysis bucket interval

In Timechart Mode, the time series is analyzed in fixed 5 minute buckets, known as the bucketspan. Thus, typical and actual values are measured in 5 minute intervals and the anomalies are pinpointed to within 5 minutes of occuring.

Drilling through to source data

The bottom right panel, “Details on anomalies for …” shows the individual anomaly results for each 5 min time interval. Click on a row to drill through to the source data. This will be displayed showing 10 time buckets either side of the anomaly. As this is a standard Splunk search page, it can be manually edited to help with further troubleshooting and analysis.

Timechart Mode drill through

Limitations

Max limit of 50,000 events - If the number of results returned by Splunk’s timechart command is more that 50,000, an error will occur in timechart and Timechart Mode. This can happen when selecting “All Time” in the time picker. To avoid this, please select a shorter time range or filter your search command.

Supported functions only - Timechart Mode will only support the functions used for Anomaly Detection. For example count, sum, max, min, mean are all supported, however median is not.

See also