Known Issues

Managing jobs and uploading data

Crash in Chrome when uploading a large file

There is a system 200MB limit when using the Kibana UI to upload a file for analysis via a browser. When the file size is large, a Chrome crash can occur which appears to be a documented out of memory error

To workaround, please use an alternative browser or upload the file using the Engine API data endpoint.

Back to top

500 HTTP response (no error message) occurs when header is invalid

If the header for CSV input data contains mismatched quotes and the values in the CSV data do not contain any quotes at all, then an HTTP 500 error message may occur. Upon reviewing the stderr.log file located in $PRELERT_HOME/logs/engine_api, this contains a java.lang.OutOfMemoryError error. The logging does not indicate that an error is present in the data header.

In this known issue, the header was:

"count,"_time","cs_host","sum_cs_bytes_"

Notice the missing quote after count. The correct header should have been:

"count","_time","cs_host","sum_cs_bytes_"

The problem is that the mismatched quote results in the majority of the CSV upload being treated as a single column value. If the CSV file is very large then this may cause all memory to be used in trying to read it.

To rectify this problem, please correct the header and re-send the data.

Back to top

Data upload temporarily pauses whilst job is persisting

The analytical model is stored in memory enabling anomaly detection in real-time across massive data sets. For resilience, the model is persisted every 3 hours. If uploading data whilst the job is periodically persisting, the data upload process will briefly pause. In very large datasets, this could take ~60 seconds before resuming. If timeouts have been set for the data upload process, then the timeout period should allow for this.

Back to top

Unable to analyze Elasticsearch fields in _source

There are a variety of different types of fields in Elasticsearch, for example, those in the original document that was indexed (stored by Elasticsearch as _source), fields created using a copy_to mapping, and script fields calculated at search time.

In order to analyze _source only fields, you must select the Retrieve whole _source document check box when configuring the Elasticsearch scheduler.

Back to top

Limited support for array fields from Elasticsearch

Elasticsearch arrays are converted to a comma separated string. This is treated as a single field by the Engine API, which is the supported format for the lat_long function. You may be able to use Transforms to extract a part of this string. You may be able to use this field in an analysis, however the whole string will be treated as the by, over or partition field e.g. “one,two,three” will be trated as a single entity.

Back to top

Job configuration picklists do not work in Safari

When creating an Anomaly Search Job there are several picklists available, for example when selecting the function for a detector. These picklists are not supported in Safari. To workaround, please manually enter the value required e.g. mean. The text boxes will accept keyboard input.

Back to top

Excessive messages reporting that “records are not in ascending chronological order”

An error message is reported when more than a significant percentage of input data is not in chronological date order or beyond the latency period (if specified):

A high proportion of records are not in ascending chronological order (86274 of 172550) and/or not within latency.

The out-of-date-order counters are cumulative for the lifetime of the job. Therefore this error message will continue to be reported even if the current batch of data is perfectly ordered.

As best practice, check the return status message and error code when uploading data. If signigicant out-of-order data has been uploaded, it will not have been analyzed and therefore the anomaly results are likely to be compromised. Re-creating the job and re-running with corrected data would be advisable.

Back to top

Analysis and reporting

Back to top

Early anomalies are detected in periodic data

The Engine API is able to detect daily periodic patterns and to model this behavior. In the early stages of an analysis, whilst the model is learning the periodic behavior, it is expected for anomalies to be raised which can be attributed to periodicity.

Daily patterns will start to be recognized after 2-3 days of data has been analyzed. If weekend patterns exist, then please allow for at least 2-3 weekends to have been seen. The model will then fine tune itself as more days are seen.

The rate and accuracy at which the Engine API can detect a daily periodic signal depends on the rate of input data and the features of the periodic pattern (i.e. smooth, noisy or spikey). In general, the more data that has been analyzed, the more accurate the modeling will be. If you are analyzing data that has a daily periodicity, then please allow for a learning period.

Back to top

A Job produces results with a timestamp in the future or an excessively large number of buckets are created

If your input data is timestamped with an Unix epoch value measured in milliseconds check that the timeFormat field of the Data Description is set to epoch_ms not plain epoch otherwise the Engine API will interpret the value as a high number of seconds since the epoch that is at some point in the future. Similarly, the job’s bucketSpan option is measured in seconds so if the difference between the timestamp on two records is greater than the bucketSpan the Engine assigns them to different buckets.

For example, the difference between epochs 1411653600000 and 1411653600600 is 600ms but if the Engine is reading the timestamps as seconds then it will see the difference as 600s (10 minutes), assuming a bucketSpan of 600s the Engine will assign the records to different buckets meaning new buckets are generated very quickly.

Back to top

Dashboard timecharts may not be drawn smoothly

Kibana charts use time aggregation when requesting data to plot. For example, for 1 month of data with a time aggregation of 1 day, 30 data points would be plotted. Sometimes this aggregation can cause undesirable effects, such as:

  • lines may not be drawn smoothly
  • lines may connect 2 adjacent data points, when zeros should have been plotted in between
  • anomalies may be hidden, for example plotting the average of 1 day would give different results to the average over 10 mins

For optimal display, timecharts should be viewed with the interval set as close to the bucketSpan as possible. Use the Interval picker if available and view results over a shorter time window for more granularity.

Back to top

Results are merged for duplicate detector descriptions

If two jobs are created, each containing a detector called “event count”, their combined maximum score will be displayed in the Explorer swimlane.

In order to work around, please ensure that detector descriptions are unique. These can be edited after the job has been created.

Back to top

Operational issues

You cannot use the ~ character or environment variables when specifying the installation directory

When you are prompted to specify the installation directory by the installer, you may enter a relative or absolute path. However, the path you enter may not contain the ~ character (tilde) used to indicate a home directory, nor may it contain environment variables such as $HOME.

If you use the ~ character when specifying your installation directory then the installation will fail.

If you include an environment variable when specifying your installation directory then it will not be expanded; you will end up installing to a directory named with the name of the environment variable rather than its value.

If you want to install to a sub-directory of your home directory, the easiest way is to have your home directory as the current directory when you run the installer, and then specify a relative installation directory. Alternatively, specify the absolute path to your home directory within the installation path rather than using the ~ character.

Back to top

ShardsException error message may indicate disk space is low

If there is very little free disk space on the partition where the Engine is installed, the datastore (which is Elasticsearch) may return error messages that are hard to interpret. These error messages contain the phrase “Primary shard is not active”. The reason why the primary shard is not active is likely to be that free disk space is low.

For example, when trying to create a new job you may receive a response that begins with the following:

Error in Elasticsearch: org.elasticsearch.action.UnavailableShardsException: [graphite][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: index

Errors similar to this will start to occur before the disk is completely full.

The solution is to free up more space on the partition where the Engine is installed.

Back to top

Running in an IPv6-only environment

By default, Java prefers to use the IPv6 capable network stack; however the majority of our users are still primarily IPv4 environments. To avoid problems with firewalls or VPNs preventing IPv6 sockets working, the two JVMs used by the Engine API are set to prefer IPv4 networking. This has been done by setting the Java system property java.net.preferIPv4Stack to true.

If you want to run the Engine API in an IPv6 environment you can reverse this preference for IPv4 as follows.

On Windows, edit the following two registry keys. In each case remove -Djava.net.preferIPv4Stack=true from the list of options:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\PrelertEngineApi\Parameters\Java\Options
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\PrelertElasticsearch\Parameters\Java\Options

On other platforms, edit $PRELERT_HOME/bin/prelert_startup.sh and make sure the following two lines are commented out:

#ES_USE_IPV4=true
#export ES_USE_IPV4

Note: You must comment them out rather than changing true to false. Any non-empty value for the environment variable ES_USE_IPV4 will cause the two JVMs to prefer the IPv4 networking stack.

Back to top

Unable to upgrade Jobs between major releases

With major new releases, changes are made to the model state and the analysis methods used. Because of this, it is generally not possible to upgrade jobs as you upgrade to a major new release of the Engine API.

Please re-install an instance of the Engine API and re-create your job configurations.

Back to top

Java licensing when embedding the Engine API on a “dedicated” device

Parts of the Engine API require a Java Runtime Environment (JRE). The free license for Oracle Java does not allow it to be used on “dedicated” devices, only “general purpose” machines. If you are distributing a pre-built appliance that embeds the Engine API and which the end user cannot easily repurpose to a completely different role then it is likely that you will need to purchase a commercial Java license if you redistribute the Oracle JRE shipped with the Engine API.

An alternative to obtaining a commercial Java license from Oracle for use on a “dedicated” device is to replace the Oracle JRE with one from OpenJDK. The replacement JRE must come from OpenJDK 8 update 20 or a later update of OpenJDK 8.

Prelert fully supports use of the Engine API on Linux using OpenJDK 8 update 40. A build of this can be downloaded from the Prelert website https://s3.amazonaws.com/prelert/builds/jre/prelert-openjre-8u40-linux-x64.tar.bz2.

To swap over the JRE that’s used by the Engine API the process is simply:

prelert_shutdown.sh
cd $PRELERT_HOME/cots
rm -rf jre
tar jxvf wherever-you-put-the-download/prelert-openjre-8u40-linux-x64.tar.bz2
prelert_startup.sh

OpenJDK is also available as a package for most Linux distributions. If you choose to use such a packaged version instead of downloading Prelert’s build, please ensure that it is OpenJDK 8 and not an earlier or later version, and that the update number is at least 20.

Back to top

Authentication not supported for the Elasticsearch instance used to store Prelert results (includes Shield)

Behavioral Analytics / Engine API 2.1 can query input data from a secured Elasticsearch cluster, using basic HTTP authentication, by specifying a username and password in the job configuration.

Additionally, information on Securing Behavioral Analytics or the Engine API the Kibana UI and/or the Engine REST API is available.

The Elasticsearch instance used to store Prelert Engine API results is accessed differently to the one(s) that input data is retrieved from, and we are currently unable to support using Elastic Shield or any other mechanism to add authentication to it. Instead, we recommend preventing unauthorised access by using a firewall to block access to the TCP ports of this Elasticsearch instance except via localhost.

If adding authentication to the Elasticsearch instance used to store Prelert results is a requirement for your implementation, please contact support@prelert.com.

Back to top

Elasticsearch and Kibana versions provided during install must be used

Behavioral Analytics / Engine API 2.1 will install a copy of Elasticsearch and Kibana. As Kibana and Elasticsearch version compatability is closely tied, the versions supplied during installation are required for a supported environment.

Back to top

Only fields available in Elasticsearch 5.0 _source document can be analyzed

At the time of release, Elasticsearch 5.0 is in Aplha and is not a supported platform for production use. For trail users, in order to analyze data from Elasticsearch 5.0, the Scheduler requires “Retrieve whole _source document” to be selected. At present, the fields must be available in the _source document in order to be analyzed. Future updates to Elasticsearch 5.0 are being reviewed on an on-going basis.

Back to top