Distributed Environments

Types of distributed environments

A Splunk deployment can be distributed in several ways:

  1. Multiple search peers; no dedicated search head
  2. Indexer layer with dedicated search head(s)
  3. Indexer layer with search head pooling (SHP) - now deprecated by Splunk
  4. Indexer layer with search head clustering (SHC)

Of these, Anomaly Detective supports options 2 and 4. (Search head pooling was supported in Anomaly Detective version 4.0, so do not upgrade beyond version 4.0 if you are using search head pooling.)

Forwarding of results

When you have search heads that are separate to indexers, a key configuration decision to make is whether to forward data that is indexed on the search head(s) to the indexers. This is achieved by setting up outputs.conf on the search head(s), and configuring the indexers to receive data on a specified port (usually 9997). Forwarding data to be indexed from search heads to indexers is not compulsory in distributed Splunk environments, but is strongly recommended by Splunk.

If you are using Anomaly Detective on a standalone search head then forwarding data to the indexer layer is also not compulsory but recommended. However, if you are using Anomaly Detective in a search head cluster then it is compulsory to set up data forwarding from your clustered search heads to your indexer layer; failure to do this will mean you only ever see an incomplete subset of Prelert results (and a different subset on each search head in the cluster).

Anomaly Detective ships with an outputs.conf file with forwarding settings commented out. If you would like to set up forwarding of results to your indexer layer and do not already have forwarding enabled, you can uncomment the contents of this file (being sure to change the dummy indexer names to the correct ones for your environment).

If forwarding is enabled then you must ensure that the prelertresults index is created on every indexer that your search head(s) might forward data to. See the following section for more information about this.

Where does Anomaly Detective need to be installed?

Anomaly Detective must obviously be installed on your search head(s). If you are using search head clustering Anomaly Detective will be installed on the search heads by pushing it from your deployer machine.

If you have a separate indexer layer, best practice is to install Anomaly Detective on all indexers as well. This is the easiest way to ensure that the prelertresults index exists on all your indexers, and also means that certain Prelert transforms can be distributed.

However, Prelert understands that you might not want to install an extra app on all your indexers, especially during an evaluation. If this is the case:

  • If you are forwarding data from your search head(s) to your indexer layer then you must somehow create the prelertresults index on each indexer
  • The prelertsplitdomainname custom search command must be preceeded by localop - see Highest Registered Domain

Where will Prelert anomaly searches/insight monitors run?

LookBacks for Prelert anomaly searches and insight monitors run on the machine they are initiated from, but are detached from Splunkweb by creating scripted inputs to do the work. (This means that if the user who initiated the LookBack logs out of Splunkweb the LookBack will not be terminated.)

Anomaly searches and insight monitors that are running continously are set up as scheduled saved searches. In a search head clustering environment, Splunk will choose which of the clustered search heads to run these on. This means that successive invocations of anomaly searches and insight monitors are likely to be on different search heads. Splunk search head clustering will automatically balance the search load across the machines in the cluster.

Where can Prelert anomaly searches/insight monitors be managed from?

In a search head clustering environment you can manage anomaly searches and insight monitors from any search head in the cluster. Splunk will replicate the configuration files that are altered across the cluster. It is important to realize that this replication process takes time (albeit a very small amount of time). Therefore it is recommended that at any given time only one search head should be used to manage a given anomaly search or insight monitor. Also, if switching between search heads, refresh your browser before performing management actions to ensure that the UI you will be managing searches from is accurately displaying the current state.

System clock synchronization

When setting up a distributed Splunk system, Splunk’s documentation emphasizes the importance of ensuring system clocks are synchronzed across all search peers and cluster members. This is best done by ensuring all machines are set up to use NTP with the same list of reliable NTP servers, and with firewall rules that allow NTP traffic.

Be aware that Prelert anomaly searches and insight monitors may unnecessarily run an extra bucket span behind real-time than necessary if clock skew is more than a second or two between the search head captain and the search head it delegates running the scheduled search to. (As an example, an anomaly search with bucket span 10 minutes may produce results 10 minutes later than it could if there is a clock skew of just 4 seconds between search heads in the search head cluster it is running on.)