- Overview
- Comparison between expression languages
- Configuring expression language used
- Measurement transformation
- JEXL Based Transformations
- Legacy Expression Language Transformations
The IoTAgent Library provides an expression language for measurement transformation, that can be used to adapt the information coming from the South Bound APIs to the information reported to the Context Broker. This is really useful when you need to adapt measure.
There are available two differen expression languages jexl
and legacy
. The recommended language to use is jexl
,
which is newer and most powerful.
JEXL overpasses the legacy language in several aspects:
- JEXL supports multiples types: Boolean, String, Number, Object, Array, NULL.
- JEXL also supports the management and creation of JSON structures (including GeoJSON Objects)
- JEXL allows to navigate and filter objects and arrays.
- JEXL supports if..then...else... via ternary operator.
- JEXL additionally supports the following operations: Divide and floor
//
, Modulus%
, Logical AND&&
and Logical OR||
. Negation operator is!
- JEXL supports comparisons.
- JEXL supports defining custom transformations (JEXL functions are not currently supported in Perseo).
For more details, check JEXL language details here.
By default, in order to maintain backward compatibility, legacy
language is applied. There are different levels to
configure the expression language used:
- At global level.
- At service group level.
- At device level.
Setting the expression language at global level
It is possible to set the default language at global level by setting the
global configuration parameter defaultExpressionLanguage
or the
environment variable
IOTA_DEFAULT_EXPRESSION_LANGUAGE
. This option configures the default expression language used to compute expressions,
possible values are: legacy
or jexl
. When not set or wrongly set, legacy
is used as default value.
Setting the expression language at service group level
It is possible to define the expression language at service group by adding the expressionLanguage
parameter. It is
optional and the possible values are: legacy
or jexl
. When not set or wrongly set, legacy
is used as default. You
can check the following example:
{
"services": [
{
"apikey": "801230BJKL23Y9090DSFL123HJK09H324HV8732",
"cbHost": "http://orion:1026",
"entity_type": "Thing",
"resource": "/iot/d"
"expressionLanguage": "jexl",
"attributes": [...]
}
]
}
Setting the expression language at device level
Expression language can be configured at device level by adding the expressionLanguage
parameter to the device
provisioning payload. It is optional and the possible values are: legacy
or jexl
. When not set or wrongly set,
legacy
is used as default. The following example shows how to provision a device using jexl
language.
{
"devices":[
{
"device_id":"45",
"protocol":"GENERIC_PROTO",
"entity_name":"WasteContainer:WC45",
"entity_type":"WasteContainer",
"expressionLanguage": "jexl",
"attributes":[...]
}
]
}
Expressions can be defined for Active attributes, either in the Device provisioning or in the Configuration provisioning. The following example shows a device provisioning payload with defined expressions (using the JEXL Expression language):
{
"devices": [
{
"device_id": "45",
"protocol": "GENERIC_PROTO",
"entity_name": "WasteContainer:WC45",
"entity_type": "WasteContainer",
"expressionLanguage": "jexl",
"attributes": [
{
"name": "location",
"type": "geo:json",
"expression": "{coordinates: [longitude,latitude], type: 'Point'}"
},
{
"name": "fillingLevel",
"type": "Number",
"expression": "level / 100"
},
{
"name": "level",
"type": "Number"
},
{
"name": "latitude",
"type": "Number"
},
{
"name": "longitude",
"type": "Number"
}
]
}
]
}
Interactive expression {coordinates: [longitude,latitude], type: 'Point'}
Interactive expression level / 100
The value of the expression
attribute is a string that can contain any number of expression patterns. In order to
complete expression to be evaluated, all the expression patterns must be evaluable (there must be a value in the
measurement for all the variables of all the expression patterns).
Note that you need to include in the provision operation all the attributes required as inputs for the expressions. In
this example, they are level
, latitude
and longitude
. Otherwise the device sending the measures will get
{"name":"ATTRIBUTE_NOT_FOUND","message":"Some of the attributes does not exist"}
when it sends some of these and the
expression will not be calculated.
The exact same syntax works for Configuration and Device provisioning.
Whenever a new measurement arrives to the IoT Agent for a device with declared expressions, all of the expressions for the device will be checked for execution: for all the defined active attributes containing expressions, the IoT Agent will check which ones contain expressions whose variables are present in the received measurement. For all of those whose variables are covered, their expressions will be executed with the received values, and their values updated in the Context Broker.
E.g.: if a device with the following provisioning information is provisioned in the IoT Agent:
{
"name":"location",
"type":"geo:point",
"expression": "longitude+', '+latitude"
},
{
"name":"fillingLevel",
"type":"Number",
"expression": "level / 100",
},
Interactive expression longitude+', '+latitude
Interactive expression level / 100
and a measurement with the following values arrive to the IoT Agent:
latitude: 1.9
level: 85.3
The only expression rule that will be executed will be that of the fillingLevel
attribute. It will produce the value
0.853
that will be sent to the Context Broker.
Note that expressions are only applied if the attribute name (as received by the IoT Agent in the southbound interface) matches the expression variable. Otherwise, the southbound value is used directly. Let's illustrate with the following example:
"consumption": {
"type": "String",
"value": "spaces | trim"
}
- Case 1: the following measure is received at the southbound interface:
consumption: "0.44"
As spaces
attribute is not included, then the expression is not applied and the consumption
measure value is
directly used, so the following is sent to CB:
"consumption": {
"type": "String",
"value": "0.44"
}
- Case 2: the following measure is received at the southbound interface:
consumption: "0.44"
spaces: " foobar "
As spaces
attribute is included, then the expression is evaluated, so overriding the 0.44 value and sending the
following to CB:
"consumption": {
"type": "String",
"value": "foobar"
}
Interactive expression spaces | trim
To allow support for expressions in combination with multi entity plugin, where the same attribute is generated for
different entities out of different incoming attribute values (i.e. object_id
), we introduced support for object_id
in the expression context.
For example, the following device:
"WeatherStation": {
"commands": [],
"type": "WeatherStation",
"lazy": [],
"active": [
{
"object_id": "v1",
"name": "vol",
"expression" : "v1*100",
"type": "Number",
"entity_name": "WeatherStation1"
},
{
"object_id": "v2",
"name": "vol",
"expression" : "v2*100",
"type": "Number",
"entity_name": "WeatherStation2"
},
{
"object_id": "v",
"name": "vol",
"expression" : "v*100",
"type": "Number"
}
]
}
When receiving the attributes v
, v1
and v2
in a payload from a message received in the southbound:
({
"name": "v",
"type": "Number",
"value": 0
},
{
"name": "v1",
"type": "Number",
"value": 1
},
{
"name": "v2",
"type": "Number",
"value": 2
})
Will now generate the following NGSI v2 payload:
{
"actionType": "append",
"entities": [
{
"id": "ws9",
"type": "WeatherStation",
"vol": {
"type": "Number",
"value": 0
}
},
{
"vol": {
"type": "Number",
"value": 100
},
"type": "WeatherStation",
"id": "WeatherStation1"
},
{
"vol": {
"type": "Number",
"value": 200
},
"type": "WeatherStation",
"id": "WeatherStation2"
}
]
}
Apart from measurement transformations (including multientity cases), expressions can be used in the following cases:
- Default EntityName defined at group level and used for provision and autoprovision of devices
- Commands (push and pull)
- Endpoint of push http commands
In all of them the following device data is available to all expressions
id
: device IDentity_name
: NGSI entity Name (principal)type
: NGSI entity type (principal)service
: device servicesubservice
: device subservice
The recommended expression language for the IoTAgent Library is JEXL. To use JEXL,
you will need to either configure it as default language using the defaultExpressionLanguage
field to jexl
(see
configuration documentation) or configuring the usage of JEXL as expression language for a given
group or device (as shown above).
{
"devices": [
{
"device_id": "45",
"protocol": "GENERIC_PROTO",
"entity_name": "WasteContainer:WC45",
"entity_type": "WasteContainer",
"expressionLanguage": "jexl",
"attributes": [
{
"name": "location",
"type": "geo:json",
"expression": "{coordinates: [longitude,latitude], type: 'Point'}"
},
{
"name": "fillingLevel",
"type": "Number",
"expression": "level / 100"
},
{
"name": "level",
"type": "Number"
},
{
"name": "latitude",
"type": "Number"
},
{
"name": "longitude",
"type": "Number"
}
]
}
]
}
In the following we provide examples of using JEXL to apply transformations.
The following table shows expressions and their expected outcomes taking into account the following measures at southbound interface:
value
with value 6 (number)ts
with value 1637245214901 (unix timestamp)name
with value"DevId629"
(string)object
with value{name: "John", surname: "Doe"}
(JSON object)array
with value[1, 3]
(JSON Array)
Expression | Expected outcome | Format | Playground |
---|---|---|---|
5 * value |
30 |
Integer | Example |
(6 + value) * 3 |
36 |
Integer | Example |
value / 12 + 1 |
1.5 |
Float | Example |
(5 + 2) * (value + 7) |
91 |
Integer | Example |
value * 5.2 |
31.2 |
Float | Example |
"Pruebas " + "De Strings" |
"Pruebas De Strings" |
String | Example |
name + "value is " +value |
"DevId629 value is 6" |
String | Example |
{coordinates: [value,value], type: 'Point'} |
{"coordinates": [6,6], "type": "Point"} |
GeoJSON Object |
Example |
ts|toisodate |
2021-11-18T14:20:14.901Z |
ISO 8601 DateTime |
Example |
Support for trim
, length
, substr
and indexOf
transformations was added.
Expression | Expected outcome | Playground |
---|---|---|
" a "| trim |
a |
Example |
name|length |
8 |
Example |
name|indexOf("e") |
1 |
Example |
name|substr(0,name|indexOf("e")+1) |
"De" |
Example |
The following are some examples of JEXL expressions not supported by the legacy expression language:
Expression | Expected outcome | Format | Playground |
---|---|---|---|
value == 6? true : false |
true |
Boolean | Example |
value == 6 && name|indexOf("e")>0 |
true |
Boolean | Example |
array[1]+1 |
3 |
Number | Example |
object.name |
"John" |
String | Example |
{type:"Point",coordinates: [value,value]} |
{type:"Point",coordinates: [6,6]} |
JSON Object | Example |
There are several predefined JEXL transformations available to be used at any JEXL expression. The definition of those transformations and their JavaScript implementation can be found at jexlTransformsMap.js.
The library module also exports a method iotAgentLib.dataPlugins.expressionTransformation.setJEXLTransforms(Map)
to be
used by specific IoT Agent implementations in order to incorporate extra transformations to this set. It is important to
remark that the lib jexlTransformsMap
cannot be overwritten by the API additions. The idea behind this is to be able
to incorporate new transformations from the IoT Agent configuration file in a fast and tactical way.
Current common transformation set:
JEXL Transformation | Equivalent JavaScript Function |
---|---|
jsonparse: (str) | JSON.parse(str); |
jsonstringify: (obj) | JSON.stringify(obj); |
indexOf: (val, char) | String(val).indexOf(char); |
length: (val) | String(val).length; |
trim: (val) | String(val).trim(); |
substr: (val, int1, int2) | String(val).substr(int1, int2); |
addreduce: (arr) | arr.reduce((i, v) | i + v)); |
lengtharray: (arr) | arr.length; |
typeof: (val) | typeof val; |
isarray: (arr) | Array.isArray(arr); |
isnan: (val) | isNaN(val); |
parseint: (val) | parseInt(val); |
parsefloat: (val) | parseFloat(val); |
toisodate: (val) | new Date(val).toISOString(); |
timeoffset:(isostr) | new Date(isostr).getTimezoneOffset(); |
tostring: (val) | val.toString(); |
urlencode: (val) | encodeURI(val); |
urldecode: (val) | decodeURI(val); |
replacestr: (str, from, to) | str.replace(from, to); |
replaceregexp: (str, reg, to) | str.replace(new RegExp(reg), to); |
replaceallstr: (str, from, to) | str.replaceAll(from, to); |
replaceallregexp: (str, reg, to) | str.replaceAll(new RegExp(reg,"g"), to); |
split: (str, ch) | str.split(ch); |
joinarrtostr: (arr, ch) | arr.join(ch); |
concatarr: (arr, arr2) | arr.concat(arr2); |
mapper: (val, values, choices) | choices[values.findIndex((target) | target == val)]); |
thmapper: (val, values, choices) | choices[values.reduce((acc,curr,i,arr) | (acc==0)||acc?acc:val<=curr?acc=i:acc=null,null)]; |
bitwisemask: (i,mask,op,shf) | (op==="&"?parseInt(i)&mask: op==="|"?parseInt(i)|mask: op==="^"?parseInt(i)^mask:i)>>shf; |
slice: (arr, init, end) | arr.slice(init,end); |
addset: (arr, x) | { return Array.from((new Set(arr)).add(x)) } |
removeset: (arr, x) | { let s = new Set(arr); s.delete(x); return Array.from(s) } |
touppercase: (val) | String(val).toUpperCase() |
tolowercase: (val) | String(val).toLowerCase() |
You have available this JEXL interactive playground with all the transformations already loaded, in which you can test all the functions described above.
As described in previous sections, this is the default language just for backward compatibility reasons, but we strongly encourage you not to use this language for defining expression in order to transform measures, since it is not capable to handle JSON native types, it is not extensible and many other reasons as described on the section Comparison between expression languages. The following example shows a device provisioning payload with defined expressions using the Legacy Expression Language:
{
"devices": [
{
"device_id": "45",
"protocol": "GENERIC_PROTO",
"entity_name": "WasteContainer:WC45",
"entity_type": "WasteContainer",
"attributes": [
{
"name": "location",
"type": "geo:point",
"expression": "${@latitude}, ${@longitude}"
},
{
"name": "fillingLevel",
"type": "Number",
"expression": "${@level / 100}"
},
{
"name": "level",
"type": "Number"
},
{
"name": "latitude",
"type": "Number"
},
{
"name": "longitude",
"type": "Number"
}
]
}
]
}
Each expression pattern is marked with the following secuence: ${<expression>}
where <expression>
is a construction
with a valid syntax.
The way the parse()
function works (at expressionParser.js
) is as follows:
- Expressions can have two return types:
String
orNumber
. This return type must be configured for each attribute that is going to be converted. Default value type isString
. - Whenever an expression is executed without error, its result will be cast to the configured type. If the conversion fails (e.g.: if the expression is null or a String and is cast to Number), the measurement update will fail, and an error will be reported to the device.
However, the usage that the Expression Translation plugin does of that function is using always String
type. That
means that at the end, the result of the expression will be always cast to String
. However, in NGSI v2 that String
result could be re-cast to the right type (i.e. the one defined for the attribute in the provision operation). Have a
look at the NGSI v2 support for more information on this.
All the information reported in the measurement received by the IoT Agent is available for the expression to use. For
every attribute coming from the South Bound, a variable with the syntax @<object_id>
will be created for its use in
the expression language.
Attribute expressions can contain values taken from the value of other attributes. Those values have, by default, the
String type. For most arithmetic operations (*
, /
, etc...) if a variable is involved, its value will be cast to
Number, regardless of the original type. For, example, if a variable @humidity
has the value '50'
(a String value),
the following expression:
${@humidity * 10}
will give 500
as the result (i.e.: the value '50'
is cast to number, to get 50
, that is then multiplied by 10). If
this cast fails (because the value of the variable is not a number, e.g.: 'Fifty'
), the overall result will be NaN
.
The expression language allows for two kinds of constants:
- Numbers (integer or float)
- Strings (marked with double quotes)
Current allowed characters are:
- All the alphanumerical characters
- Whitespaces
The following operations are currently available, divided by attribute type
- multiplication ('*')
- division ('/')
- addition ('+')
- subtraction ('-' binary)
- negation ('-' unary)
- power ('^')
- concatenation ('#'): returns the concatenation of the two values separated by
#
. - substring location (
indexOf(<variable>, <substring>)
): returns the index where the first occurrence of the substring<substring>
can be found in the string value of<variable>
. - substring (
substr(<variable>, <start> <end>)
): returns a substring of the string variable passed as a parameter, starting at posisitonstart
and ending at position<end>
. - whitespace removal (
trim(<string>)
): removes all the spaces surrounding the string passed as a parameter.
Parenthesis can be used to define precedence in the operations. Whitespaces between tokens are generally ignored.
The following table shows expressions and their expected outcomes for a measure with two attributes: @value
with value
6
and @name
with value DevId629
.
Expression | Expected outcome | Format |
---|---|---|
5 \* @value |
30 |
Number |
(6 + @value) \* 3 |
36 |
Number |
@value / 12 + 1 |
1.5 |
Number |
(5 + 2) \* (@value + 7) |
91 |
Number |
@value \* 5.2 |
31.2 |
Number |
"Pruebas " + "De Strings" |
Pruebas De Strings |
String |
@name value is @value |
DevId629 value is 6 |
String |
As it is explained in previous sections, expressions can have two return types: String
or Number
, being the former
one the default. Whenever an expression is executed without error, its result will be cast to the configured type.
NGSI v2 and NGSI-LD fully supports all the types described in the JSON specification (string
, number
, boolean
,
object
, array
and null
). Therefore, the result of an expression must be cast to the appropriate type (the type
used to define the attribute) in order to avoid inconsistencies between the type field for an attribute and the type of
the value that is being sent.
The expression parser for legacy does not support JSON Arrays and JSON document. For the rest of types the workflow will be the following:
- Variables will be cast to String no matter the expression type (see comments above regarding this)
- The expression will be applied
- The output type will be cast again to the original attribute type.
- If attribute type is
Number
and the value is anInteger
, then the value is casted to integer (JSON number) - If attribute type is
Number
and the value is aFloat
, then the value is casted to float (JSON number) - If attribute type is
Boolean
then the value is cast to boolean (JSON boolean). In order to do this conversion, onlytrue
or1
are cast to true. - If attribute type is
None
then the value is cast tonull
(JSON null)
Note. All the operations and castings described above are not performed using JELX, because the user has full control on the final value of the attributes, so there is no need of adding a layer of autocast that would interfere and makes things more complicated.
E.g.: if a device with the following provisioning information is provisioned in the IoT Agent:
{
"name": "status",
"type": "Boolean",
"expression": "${@status * 20}"
}
and a measurement with the following values arrive to the IoT Agent:
status: true
- The expression
*
is a multiplication, so the expression type makesstatus
to be casted to Number. The cast oftrue
to number is 1. - Expression is evaluated, resulting in 20
- 20 is cast to
20
since Expression Plugin always use String as Expression type. - The attribute type is
Boolean
so the result is casted to Boolean before sending it to CB. The cast of20
to boolean is false (onlytrue
or1
are cast to true).
More examples of this workflow are presented below for the different types of attributes supported in NGSI v2 and the two possible types of expressions: Integer (arithmetic operations) or Strings.
pressure
with value 52 (integer)consumption
with value 0.44 (float)active
with valuenull
(None type)
Expression | Expected outcome | Format |
---|---|---|
${@pressure * 20} |
1040 |
Integer |
${trim(@pressure)} |
52 |
Integer |
${@consumption * 20} |
8.8 |
Float |
${trim(@consumption)} |
0.44 |
Float |
${@pressure * 20} |
1040 |
Integer |
${@active * 20} |
null |
None |
${trim(@active) |
null |
None |
${trim(@consumption)} |
0.44 |
Float |