concat

The concat command simply concatenates all the input fields together. This may be useful if you wish to create a unique key for use as a ‘by’ field from by related fields together.

  • multiple inputs
  • single output
  • default output fieldname: “concat”
  • optional delimiter argument to join the inputs with

Example 1

In this CSV log file the date and time values are in 2 separate fields but the Prelert Engine requires both date and time to be in a single field.

date,       time,     event_type, ...
2015-03-31, 08:00:00, login_event, ...
2015-03-31, 08:00:27, login_event, ...
...

The solution is to concatenate the 2 fields together

"transforms":[
    {
        "transform" : "concat",
        "inputs" : ["date", "time"],
        "outputs" : ["datetime"]
    }
]

which transforms the input data to

date,       time,     event_type,   datetime,  ...
2015-03-31, 08:00:00, login_event,  2015-03-3108:00:00,   ...
2015-03-31, 08:00:27, login_event,  2015-03-3108:00:27,   ...
...

then use the new datetime field in the job’s data description

"dataDescription" : {
     "fieldDelimiter":",",
     "timeField":"datetime",
     "timeFormat":"yyyy-MM-ddHH:mm:ss"
}

Example 2

Append performance_metric to host and write to the default field concat

"transforms":[
    {
        "transform" : "concat",
        "inputs" : ["host",  "performance_metric"]
    }
]

In Example 2, the following input data

time,host,performance_metric,value
2015-03-01 01:00:00Z,host1,CPU,63
2015-03-01 01:00:00Z,host1,Network,2534
2015-03-01 01:00:00Z,host2,CPU,77
2015-03-01 01:00:00Z,host2,Network,9836

would transform to look like this

time,host,performance_metric,value,concat
2015-03-01 01:00:00Z,host1,CPU,63,host1CPU
2015-03-01 01:00:00Z,host1,Network,2534,host1Network
2015-03-01 01:00:00Z,host2,CPU,77,host2CPU
2015-03-01 01:00:00Z,host2,Network,9836,host2Network

Example 3

In this data set the host and port are in separate fields we wish to join them with a colon to make a valid host port pair

datetime,            host,      port, ...
2015-03-31T08:00:00, localhost, 80, ...
2015-03-31T08:00:27, localhost, 443, ...
...

Define a concat transform with : as the optional delimiter argument

"transforms":[
    {
        "transform" : "concat",
        "arguments" : [":"],
        "inputs" : ["host",  "port"],
        "outputs" : ["host_port"]
    }
]

after the transform the data will look like this

datetime,            host_port, ...
2015-03-31T08:00:00, localhost:80, ...
2015-03-31T08:00:27, localhost:443, ...