Count Functions

Detect anomalies where the count of events in a bucket is anomalous.

Use non_zero_count functions if your data is sparse and you wish to ignore cases where the bucket count is zero.

Use distinct_count functions to analyze when the number of distinct values in one field is unusual, as opposed to the total count.

Use high sided functions if you wish to monitor unusually high event rates. Use low sided functions if you wish to look at drops in event rate.

  • count, high_count, low_count
  • non_zero_count, high_non_zero_count, low_non_zero_count
  • distinct_count, high_distinct_count, low_distinct_count

count

Detect anomalies where the count of events in a bucket is anomalous. Use high sided functions if you wish to monitor unusually high event rates. Use low sided functions if you wish to look at drops in event rate.

  • fieldName: N/A
  • byFieldName: optional
  • overFieldName: optional

Example 1

Probably the simplest possible analysis! Flags up time buckets during which the overall count of events is higher or lower than usual.

{ "function" : "count" }
  • Models the event rate
  • Detects when the event rate is unusual compared to the past

Example 2

{ "function" : "high_count", "byFieldName" : "error_code", "overFieldName": "user" }
  • Models the event rate for each errorCode
  • Detects users that generate an unusually high count of errorCodes compared to other users

Example 3

In a data stream containing a field “status”, detect when the count of events for a given error is lower than usual. If the data stream consists of web server access log records, a drop in the count of events for a particular status code might be an indication that something isn’t working correctly.

{ "function" : "low_count", "byFieldName" : "status_code" }
  • Models the event rate for each status_code
  • Detects when a status_code has a unusually low count compared to its past behavior

non_zero_count

Using non_zero_count allows analysis to be performed on data which is known to have gaps or be sparse, and when the gaps are not considered important.

For example, if the number of events per bucketspan is seen as:

1,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,43,31,0,0,0,0,0,0,0,0,0,0,0,0,2,1

then non_zero_count will only model:

1,22,2,43,31,2,1

Use high sided functions if you wish to monitor unusually high event rates. Use low sided functions if you wish to look at drops in event rate.

  • fieldName: N/A
  • byFieldName: optional
  • overFieldName: N/A

Note: Population analysis (i.e. using an over field) is not applicable for this analysis function.

Example 1

{ "function" : "high_non_zero_count", "byFieldName" : "signaturename" }
  • Models the count of events for signaturenames
  • Ignores any buckets where the count is zero
  • Detects when a signaturename has an unusually high count of events compared to its past

distinct_count

Detect anomalies where the number of distinct values in one field is unusual. Available with both high and low sided functions for use if you wish to look at unusually high or unusually low distinct counts.

  • fieldName: required
  • byFieldName: optional
  • overFieldName: optional

Example 1

Detect when a system has an unusual number of logged in users.

{ "function" : "distinct_count", "fieldName" : "user" }
  • Models the distinct count of users
  • Detects when the distinct number of users is unusual compared to the past

Example 2

Detect instances of port scanning.

{ "function" : "high_distinct_count", "fieldName" : "dst_port", "overFieldName": "src_ip" }
  • Models the distinct count of ports
  • Detects src_ip’s that connect to an unusually high number of different dst_ports compared to other src_ip’s