Job Resource

An instance of an analysis job in the API is defined by the following properties:

id:

A numerical character string that uniquely identifies the job.

Type: String

description:

Optional brief description of the job.

Type: String

status:

The status of the job, containing one of the following values:

RUNNING:The job is actively receiving and processing data.
CLOSED:The job finished successfully with its model state persisted. The job is still available to accept further data. Note: If you are sending data on a periodic cycle and close the job at the end of each transaction, then the job will be marked as CLOSED in the intervals between data being sent. So if the data is being sent every minute say, and it takes 1 second to process, then the job will be marked as CLOSED for 59 seconds.
FAILED:The job did not finish successfully due to an error. Note: This can occur due to invalid input data. If so, then sending corrected data to a FAILED job will re-open the job and reset its status to RUNNING.

Type: String

timeout:

If data is not received within this timeout period, the job will automatically close. The default value is 600 seconds. Note: The job will be automatically re-opened as soon as new data is sent to it.

Type: Long

location:

The location of this resource e.g. http://localhost:8080/engine/v2/jobs/<jobId>.

Type: URL (String)

createTime:

The time the job was created, in ISO 8601 format.

Type: String

finishTime:

If the status field CLOSED or FAILED this is the time the job finished else it is null. Finished jobs either complete normally or fail. The datetime string is in ISO 8601 format.

Type: String

lastDataTime:

The time the last received data was processed, or null if no data has been processed. This is the time the job saw the data i.e. the time it was POSTed to the service.

Type: String

analysisConfig{}:
 

Analysis configuration, specifying how the data should be analyzed. See job configuration for further details.

Type: Object

analysisLimits{}:
 

Defines limits on the number of field values and time buckets to be analyzed. See Limits on the size of the analysis (analysisLimits).

Type: Object

dataDescription{}:
 

Describes the data format and how the API should parse the timestamp field. See Describing your data format (dataDescription).

Type: Object

transforms[]:

The list of data pre-processing transforms. See Data pre-processing transforms (transforms).

Type: Object

schedulerConfig[]:
 

Instructions on how to periodically retrieve input data from a data source. See Scheduled data extraction (schedulerConfig).

Type: Object

endpoints{}:

Links to job endpoints; alerting, bucket and record results, category definitions, data, logs. See Job Endpoints.

Type: Object

counts{}:

The number of records processed in the job and related error counts. See Job Counts.

Type: Object

modelSizeStats{}:
 

Information relating to the size and contents of the model. See Job Model Size Stats.

Type: Object

customSettings{}:
 

Custom URLs linking from anomaly results to source data.

Type: Object

modelDebugConfig{}:
 

Advanced configuration option. Stores model information along with results. Enabling this uses increased system resources and is not feasible for jobs with many entities. Model debug must be disabled if performance issues are experienced. See Model debug (modelDebugConfig).

Type: Object

ignoreDowntime:

Advanced configuration option. Controls the behavior after a job re-start where gaps in data may be treated as anomalous or as a maintenance window.

A job is re-started after a system start-up or after sending data to a CLOSED job. Sending data to a job that is already in a RUNNING state, will not re-start the job.

By default, this parameter is not specified, in which case the default bevahior is:

  • after a system re-start, a gap will be treated as downtime for maintenance
  • after sending data to a CLOSED job, a gap will be treated as anomalous

Contains one of the following values:

NEVER:For all job re-starts, gaps in data will be treated as anomalous.
ALWAYS:For all job re-starts, gaps in data will be assumed to be a maintenance window. i.e. not anomalous.
ONCE:For the next job re-start only, gaps in data will be assumed to be a maintenance window. i.e. not anomalous. After the next re-start, this parameter will be deleted.

Type: String

renormalizationWindowDays:
 

Advanced configuration option. Period over which scores are updated with new values due to re-normalization. Default is the longer of 30 days or 100 buckets.

Type: Integer

resultsRetentionDays:
 

Advanced configuration option. Time in days for which job results are retained. Once per day at 00:30 (server time), results older than this period will be deleted from the Elasticsearch datastore. If configured, model debug results are also deleted. Default is null, i.e. results are not deleted.

Type: Long

modelSnapshotRetentionDays:
 

Advanced configuration option. Time in days for which saved model snapshots are retained for this job. Snapshots older than this period will be deleted from the Elasticsearch datastore. Default is 1 day.

Type: Long

backgroundPersistInterval:
 

Advanced configuration option. For long running jobs, the model is periodically saved to disk. This is the number of seconds between each periodic persistence. Default is between 3 - 4 hours (slightly randomized to avoid all jobs persisting at exactly the same time).

For very large models (several GB), persistence could take 10-20 minutes. Please do not set this value too low. The smallest allowed value is 3600 (1 hour).

Type: Integer

averageBucketProcessingTimeMs:
 

The average time taken to process a bucket and produce results. If no bucket results have been emitted this value will be null. The average is an exponential moving average of each bucket’s processing time.

Type: Number

Job Counts

Information provided regarding job operational progress. Describes the number of records processed and any related error counts.

Important

Job count values are cumulative for the lifetime of the job and are not reset if a model snapshot is reverted or if old results are deleted from the data store.

bucketCount:

The number of bucket results produced by the job.

Type: Long

inputRecordCount:
 

The number data records read by the job.

Type: Long

inputFieldCount:
 

The total number of record fields read including those that aren’t used in the analysis.

Type: Long

inputBytes:

The number of raw bytes read by the job.

Type: Long

processedFieldCount:
 

The total number of fields in all the records that have been processed by the job. Only the fields set in the jobs Detector Configuration contribute to this count. The time stamp is not included.

Type: Long

processedRecordCount:
 

The number of records that have been processed by the job, including records with missing fields as these are still analyzed. Records not in chronological order and outside of the latency window or with invalid timestamps and those filtered by a exclude transform are not processed. The total number of records sent to the Engine (i.e. the inputRecordCount) is equal to processedRecordCount + invalidDateCount + outOfOrderTimeStampCount + excludedRecordCount .

Type: Long

invalidDateCount:
 

The number of records with either a missing date field or a date that could not be parsed. These records are not processed by the Engine.

Type: Long

missingFieldCount:
 

Jobs are configured to analyze particular fields. This value is the number of records that are missing a configured field. Records with missing fields are still processed by the Engine because not all fields may be missing. The value of processedRecordCount includes this count.

Type: Long

outOfOrderTimeStampCount:
 

The Engine API prefers time series data to be in ascending chronological order. Records that are out of time sequence and outside of the latency window are discarded. This value is the count of such records.

Type: Long

failedTransformCount:
 

The number of transform operations that failed. In theory this value could be greater than the number of records if multiple transforms are applied to each record.

Type: Long

excludedRecordCount:
 

The number of records filtered out by exclude transforms. These records are removed before being sent to analysis.

Type: Long

latestRecordTimeStamp:
 

The timestamp of the last chronologically ordered record. This may not be the same as the timestamp of the last record if the records are not in strict chronological order. The datetime string is in ISO 8601 format.

Type: String

Job Model Size Stats

Information provided relating to the size and contents of the model.

memoryStatus:

The status of the internal mathematical models. See Limits on the size of the analysis (analysisLimits). Contains one of the following values:

OK:The internal models stayed below the configured value.
SOFT_LIMIT:The internal models used more than 60% of the configured memory limit and the Engine has started to prune older unused models to free up space.
HARD_LIMIT:The internal models used more space that the configured memory limit and the Engine was unable to process all incoming data.

Type: String

modelBytes:

The number of bytes of the memory used by the internal mathematical models. This is the maximum value since the last time the model was persisted. If the job is CLOSED, this displays the latest size.

Type: Long

totalByFieldCount:
 

The number of by field values analyzed by the models. Note that by field values are counted separately for each detector and partition.

Type: Long

totalOverFieldCount:
 

The number of over field values analyzed by the models. Note that over field values are counted separately for each detector and partition.

Type: Long

totalPartitionFieldCount:
 

The number of partition field values analyzed by the models.

Type: Long

Job Endpoints

Information provided to describe URL links to other job endpoints.

alertsLongPoll:

A link to the job’s low latency alerting endpoint e.g. http://localhost:8080/engine/v2/alerts_longpoll/<jobId>.

Type: URL (String)

buckets:

A link to the job’s result buckets e.g. http://localhost:8080/engine/v2/results/<jobId>/buckets.

Type: URL (String)

records:

A link to the job’s anonaly records e.g. http://localhost:8080/engine/v2/results/<jobId>/records.

Type: URL (String)

data:

A link to the job’s data endpoint e.g. http://localhost:8080/engine/v2/data/<jobId>.

Type: URL (String)

logs:

A link to the job’s log files e.g. http://localhost:8080/engine/v2/logs/<jobId>.

Type: URL (String)

categoryDefinitions:
 

A link to the job’s category definitions e.g. http://localhost:8080/engine/v2/results/<jobId>/categorydefinitions.

Type: URL (String)