From 0c61fcc779f439930ff7ed9ace70157a0f307c5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ferm=C3=ADn=20Gal=C3=A1n=20M=C3=A1rquez?= Date: Thu, 24 Aug 2023 11:40:13 +0200 Subject: [PATCH 1/5] Update deprecated.md --- doc/deprecated.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/deprecated.md b/doc/deprecated.md index c95324c4e..185786b34 100644 --- a/doc/deprecated.md +++ b/doc/deprecated.md @@ -22,6 +22,8 @@ A list of deprecated features and the version in which they were deprecated foll - Support groups (provision) statically defined by configuration - Support to in-memory registry (i.e.`deviceRegistry.type=memory`) - Support to legacy expressions (finally removed in 3.2.0) +- Bidirectinal pluging (to be removed in 3.4.0) +- appendMode (to be removed in 3.4.0) The use of Node.js v14 is highly recommended. @@ -51,3 +53,5 @@ The following table provides information about the last iotagent-node-lib versio | Support to Node.js v12 | 2.24.0 | September 2nd, 2022 | | Support to NGSI-LD 1.3 | 2.25.0 | January 24th, 2023 | | Support to Legacy Expressions | 3.1.0 | April 25th, 2023 | +| bidirrectional plugin | 3.3.0 | To be defined | +| appendMode | 3.3.0 | To be defined | From 6535b0935ba8a5307f300447876525272bff6aa0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ferm=C3=ADn=20Gal=C3=A1n=20M=C3=A1rquez?= Date: Thu, 24 Aug 2023 11:40:47 +0200 Subject: [PATCH 2/5] Update CHANGES_NEXT_RELEASE --- CHANGES_NEXT_RELEASE | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGES_NEXT_RELEASE b/CHANGES_NEXT_RELEASE index be943341f..3568c4ec8 100644 --- a/CHANGES_NEXT_RELEASE +++ b/CHANGES_NEXT_RELEASE @@ -9,3 +9,5 @@ - Fix: do not propagate group config (timestamp and explicitAttrs) to autoprovisioned devices (at database level) (#1377) - Fix: appendMode at general level (config.js / env var) changes its default from false to true - Fix: remove sensitive MongoDB connection parameters from log traces (remove 'option' object from logs) +- Deprecate: bidirectional plugin +- Deprecate: appendMode From 5d885777010ef56bd2de815949c44cfcc932c436 Mon Sep 17 00:00:00 2001 From: mapedraza Date: Thu, 24 Aug 2023 12:28:22 +0200 Subject: [PATCH 3/5] remove reverse & appendMode doc --- doc/api.md | 98 ++------------------------------ doc/howto.md | 120 +++++++++++++++++++-------------------- doc/installationguide.md | 5 -- 3 files changed, 64 insertions(+), 159 deletions(-) diff --git a/doc/api.md b/doc/api.md index f3425d79a..98bde807f 100644 --- a/doc/api.md +++ b/doc/api.md @@ -19,8 +19,7 @@ - [Measurement persistence options](#measurement-persistence-options) - [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision) - [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs) - - [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode) - - [Differences between `autoprovision`, `explicitAttrs` and `appendMode`](#differences-between-autoprovision-explicitattrs-and-appendmode) + - [Differences between `autoprovision`, `explicitAttrs`.](#differences-between-autoprovision-explicitattrs) - [Expression language support](#expression-language-support) - [Examples of JEXL expressions](#examples-of-jexl-expressions) - [Available functions](#available-functions) @@ -31,7 +30,6 @@ - [Multientity measurement transformation support (`object_id`)](#multientity-measurement-transformation-support-object_id) - [Timestamp Compression](#timestamp-compression) - [Timestamp Processing](#timestamp-processing) - - [Bidirectionality plugin (bidirectional)](#bidirectionality-plugin-bidirectional) - [Overriding global Context Broker host](#overriding-global-context-broker-host) - [Multitenancy, FIWARE Service and FIWARE ServicePath](#multitenancy-fiware-service-and-fiware-servicepath) - [Secured access to the Context Broker](#secured-access-to-the-context-broker) @@ -143,9 +141,9 @@ described in the [Device datamodel](#device-datamodel) section. If devices are not pre-registered, they will be automatically created when a measure arrives to the IoT Agent - this process is known as autoprovisioning. The IoT Agent will create an empty device with the group `apiKey` and `type` - the -associated document created in database doesn't include config group parameters (in particular, `timestamp`, `explicitAttrs`, -`active` or `attributes`, `static` and `lazy` attributes and commands). The IoT Agent will also create the entity in the -Context Broker if it does not exist yet. +associated document created in database doesn't include config group parameters (in particular, `timestamp`, +`explicitAttrs`, `active` or `attributes`, `static` and `lazy` attributes and commands). The IoT Agent will also create +the entity in the Context Broker if it does not exist yet. This behavior allows that autoprovisioned parameters can freely established modifying the device information after creation using the provisioning API. However, note that if a device (autoprovisioned or not) doesn't have these @@ -200,8 +198,6 @@ Some transformation plugins also allow the use of the following optional fields: with the `entity_type` attribute. If no type is configured, the device entity type is used instead. Entity names can be defined as expressions, using the [Expression Language definition](#expression-language-support). - **entity_type**: configures the type of an alternative entity. -- **reverse**: add bidirectionality expressions to the attribute. See the **bidirectionality** transformation plugin - in the [Data Mapping Plugins section](development.md#bidirectionality-plugin-bidirectional) for details. Additionally for commands (which are attributes of type `command`) the following fields are optional: @@ -371,15 +367,13 @@ used should be taken from those defined by ## Measurement persistence options -There are 3 different options to configure how the IoTAgent stores the measures received from the devices, depending on +There are 2 different options to configure how the IoTAgent stores the measures received from the devices, depending on the following parameters: - `autoprovision`: If the device is not provisioned, the IoTAgent will create a new device and entity for it. - `explicitAttrs`: If the measure element (object_id) is not defined in the mappings of the device or config group provision, the measure is stored in the Context Broker by adding a new attribute to the entity with the same name of the undefined measure element. -- `appendMode`: It configures the request to the Context Broker to update the entity every time a new measure arrives. - It have implications depending if the entity is already created or not in the Context Broker. ### Autoprovision configuration (autoprovision) @@ -452,29 +446,7 @@ depending on the JEXL expression evaluation: - If it evaluates to an array just measures defined in the array (identified by their attribute names, not by their object_id) will be will be propagated to NGSI interface (as in case 3) -### Configuring operation to persist the data in Context Broker (appendMode) - -This is a flag that can be enabled by activating the parameter `appendMode` in the configuration file or by using the -`IOTA_APPEND_MODE` environment variable (more info -[here](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/installationguide.md)). If this flag is -activated (its default behaviour), the update requests to the Context Broker will be performed always with APPEND type, -instead of the default UPDATE. This have implications in the use of attributes with Context Providers, so this flag -should be used with care. - -When running the agent using `appendMode=false`, if sending measures that are not included in the config group (as -active measures), the IoT Agent returns a `422 Unprocessable Entity` code with the following message: - -```json -{ - "name": "ENTITY_GENERIC_ERROR", - "message": "Error accesing entity data for device: deviceType:dev of type: deviceType" -} -``` - -Additionally, the agent creates the device and the corresponding entity in the broker if `autoprovision==true` (default -behaviour). - -### Differences between `autoprovision`, `explicitAttrs` and `appendMode` +### Differences between `autoprovision`, `explicitAttrs`. Since those configuration parameters are quite similar, this section is intended to clarify the relation between them. @@ -486,16 +458,6 @@ related to the **southbound**. What `explicitAttrs` does is to filter from the southbound the parameters that are not explicitly defined in the device provision or config group. That also would avoid propagating the measures to the Context Broker. -The default way the agent updates the information into the Context Broker is by using an update request. If -`appendMode=true` (the default behaviour), the IoTA will use an append request instead of an update one. This means it -will store the attributes even if they are not present in the entity. This seems the same functionality that the one -provided by `autoprovision`, but it is a different concept since the scope of this config is to setup how the IoT -interacts with the context broker, this is something related to the **northbound**. - -Note that, even creating a config group with `autoprovision=true` and `explicitAttrs=true`, if you do not provision -previously the entity in the Context Broker (having all attributes to be updated), it would fail if `appendMode=false`. -For further information check the issue [#1301](https://github.com/telefonicaid/iotagent-node-lib/issues/1301). - ## Expression language support The IoTAgent Library provides an expression language for measurement transformation and other purposes. This expression @@ -887,54 +849,6 @@ The IOTA processes the entity attributes looking for a `TimeInstant` attribute. adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the Standard `observedAt` property-of-a-property is used instead. -## Bidirectionality plugin (bidirectional) - -This plugin allows the devices with composite values an expression to update the original values in the devices when the -composite expressions are updated in the Context Broker. This behavior is achieved through the use of subscriptions. - -IoTAs using this plugins should also define a notification handler to handle incoming values. This handler will be -intercepted by the plugin, so the mapped values are included in the updated notification. - -When a device is provisioned with bidirectional attributes, the IoTAgent subscribes to changes in that attribute. When a -change notification for that attribute arrives to the IoTA, it applies the transformation defined in the device -provisioning payload to the notification, and calls the underlying notification handler with the transformed entity -including the `value` along with any `metadata`, and in the case of an NGSI-LD bidirectional attribute a `datasetId` if -provided. - -The following `attributes` section shows an example of the plugin configuration (using `IOTA_AUTOCAST=false` to avoid -translation from geo:point to geo:json) - -```json - "attributes": [ - { - "name":"location", - "type":"geo:point", - "expression": "latitude, longitude", - "reverse": [ - { - "object_id":"longitude", - "type": "Number", - "expression": "location | split(', ')[0] | parsefloat()" - }, - { - "object_id":"latitude", - "type": "Number", - "expression": "location | split(', ')[1] | parsefloat()" - } - ] - } - ], -``` - -For each attribute that would have bidirectionality, a new field `reverse` must be configured. This field will contain -an array of fields that will be created based on the notifications content. The expression notification can contain any -attribute of the same entity as the bidirectional attribute; declaring them in the expressions will add them to the -subscription payload. - -For each attribute in the `reverse` array, an expression must be defined to calculate its value based on the -notification attributes. This value will be passed to the underlying protocol with the `object_id` name. Details about -how the value is then progressed to the device are protocol-specific. - ## Overriding global Context Broker host **cbHost**: Context Broker host URL. This option can be used to override the global CB configuration for specific types diff --git a/doc/howto.md b/doc/howto.md index 55e1fdace..081bf55f7 100644 --- a/doc/howto.md +++ b/doc/howto.md @@ -75,12 +75,12 @@ folder of your project. Remember to change the Context Broker IP to your local C Now we can begin with the code of our IoT Agent. The very minimum code we need to start an IoT Agent is the following: ```javascript -var iotAgentLib = require("iotagent-node-lib"), - config = require("./config"); +var iotAgentLib = require('iotagent-node-lib'), + config = require('./config'); iotAgentLib.activate(config, function (error) { if (error) { - console.log("There was an error activating the IOTA"); + console.log('There was an error activating the IOTA'); process.exit(1); } }); @@ -112,10 +112,10 @@ In order to add the Express dependency to your project, add the following line t The require section would end up like this (the standard `http` module is also needed): ```javascript -var iotAgentLib = require("iotagent-node-lib"), - http = require("http"), - express = require("express"), - config = require("./config"); +var iotAgentLib = require('iotagent-node-lib'), + http = require('http'), + express = require('express'), + config = require('./config'); ``` And install the dependencies as usual with `npm install`. You will have to require both `express` and `http` in your @@ -129,16 +129,16 @@ function initSouthbound(callback) { southboundServer = { server: null, app: express(), - router: express.Router(), + router: express.Router() }; - southboundServer.app.set("port", 8080); - southboundServer.app.set("host", "0.0.0.0"); + southboundServer.app.set('port', 8080); + southboundServer.app.set('host', '0.0.0.0'); - southboundServer.router.get("/iot/d", manageULRequest); + southboundServer.router.get('/iot/d', manageULRequest); southboundServer.server = http.createServer(southboundServer.app); - southboundServer.app.use("/", southboundServer.router); - southboundServer.server.listen(southboundServer.app.get("port"), southboundServer.app.get("host"), callback); + southboundServer.app.use('/', southboundServer.router); + southboundServer.server.listen(southboundServer.app.get('port'), southboundServer.app.get('host'), callback); } ``` @@ -154,18 +154,18 @@ function manageULRequest(req, res, next) { iotAgentLib.retrieveDevice(req.query.i, req.query.k, function (error, device) { if (error) { res.status(404).send({ - message: "Couldn't find the device: " + JSON.stringify(error), + message: "Couldn't find the device: " + JSON.stringify(error) }); } else { values = parseUl(req.query.d, device); - iotAgentLib.update(device.name, device.type, "", values, device, function (error) { + iotAgentLib.update(device.name, device.type, '', values, device, function (error) { if (error) { res.status(500).send({ - message: "Error updating the device", + message: 'Error updating the device' }); } else { res.status(200).send({ - message: "Device successfully updated", + message: 'Device successfully updated' }); } }); @@ -190,17 +190,17 @@ function parseUl(data, device) { } function createAttribute(element) { - var pair = element.split("|"), + var pair = element.split('|'), attribute = { name: pair[0], value: pair[1], - type: findType(pair[0]), + type: findType(pair[0]) }; return attribute; } - return data.split(",").map(createAttribute); + return data.split(',').map(createAttribute); } ``` @@ -227,14 +227,14 @@ show the modifications in the `activate()` function: ```javascript iotAgentLib.activate(config, function (error) { if (error) { - console.log("There was an error activating the IOTA"); + console.log('There was an error activating the IOTA'); process.exit(1); } else { initSouthbound(function (error) { if (error) { - console.log("Could not initialize South bound API due to the following error: %s", error); + console.log('Could not initialize South bound API due to the following error: %s', error); } else { - console.log("Both APIs started successfully"); + console.log('Both APIs started successfully'); } }); } @@ -286,7 +286,7 @@ A HTTP request library will be needed in order to make those calls. To this exte used. In order to do so, add the following require statement to the initialization code: ```javascript -request = require("request"); +request = require('request'); ``` and add the `request` dependency to the `package.json` file: @@ -303,11 +303,11 @@ and add the `request` dependency to the `package.json` file: The require section should now look like this: ```javascript -var iotAgentLib = require("iotagent-node-lib"), - http = require("http"), - express = require("express"), - request = require("request"), - config = require("./config"); +var iotAgentLib = require('iotagent-node-lib'), + http = require('http'), + express = require('express'), + request = require('request'), + config = require('./config'); ``` ### Implementation @@ -321,11 +321,11 @@ for the context provisioning requests. At this point, we should provide two hand ```javascript function queryContextHandler(id, type, service, subservice, attributes, callback) { var options = { - url: "http://127.0.0.1:9999/iot/d", - method: "GET", + url: 'http://127.0.0.1:9999/iot/d', + method: 'GET', qs: { - q: attributes.join(), - }, + q: attributes.join() + } }; request(options, function (error, response, body) { @@ -350,21 +350,21 @@ attributes). Here is the code for the `createResponse()` function: ```javascript function createResponse(id, type, attributes, body) { - var values = body.split(","), + var values = body.split(','), responses = []; for (var i = 0; i < attributes.length; i++) { responses.push({ name: attributes[i], - type: "string", - value: values[i], + type: 'string', + value: values[i] }); } return { id: id, type: type, - attributes: responses, + attributes: responses }; } ``` @@ -374,11 +374,11 @@ function createResponse(id, type, attributes, body) { ```javascript function updateContextHandler(id, type, service, subservice, attributes, callback) { var options = { - url: "http://127.0.0.1:9999/iot/d", - method: "GET", + url: 'http://127.0.0.1:9999/iot/d', + method: 'GET', qs: { - d: createQueryFromAttributes(attributes), - }, + d: createQueryFromAttributes(attributes) + } }; request(options, function (error, response, body) { @@ -388,31 +388,31 @@ function updateContextHandler(id, type, service, subservice, attributes, callbac callback(null, { id: id, type: type, - attributes: attributes, + attributes: attributes }); } }); } ``` -The updateContext handler deals with the modification requests that arrive at the North Port of the IoT Agent via `/v2/op/update`. It is -invoked once for each entity requested (note that a single request can contain multiple entity updates), with the same -parameters used in the queryContext handler. The only difference is the value of the attributes array, now containing a -list of attribute objects, each containing name, type and value. The handler must also make use of the callback to -return a list of updated attributes. +The updateContext handler deals with the modification requests that arrive at the North Port of the IoT Agent via +`/v2/op/update`. It is invoked once for each entity requested (note that a single request can contain multiple entity +updates), with the same parameters used in the queryContext handler. The only difference is the value of the attributes +array, now containing a list of attribute objects, each containing name, type and value. The handler must also make use +of the callback to return a list of updated attributes. For this handler we have used a helper function called `createQueryFromAttributes()`, that transforms the NGSI representation of the attributes to the UL type expected by the device: ```javascript function createQueryFromAttributes(attributes) { - var query = ""; + var query = ''; for (var i in attributes) { - query += attributes[i].name + "|" + attributes[i].value; + query += attributes[i].name + '|' + attributes[i].value; if (i != attributes.length - 1) { - query += ","; + query += ','; } } @@ -544,10 +544,6 @@ request/seconds that can be manage by the server. It's important to remark that included in the IoT Agent Node Lib but it is not mandatory that you activate this functionality. In this example, we will see how to use this functionality to deploy an IoT Agent in multi-thread environment. -**WARNING:** it has been observed in Orion-IOTA integration tests some fails in bidirectional plugin usage scenarios in -multi-thread mode. The fail has not been confirmed yet (it could be a glitch of the testing environment). However, take -this into account if you use multi-thread in combination with bidirectional plugin. - In order to activate the functionality, you have two options, configure the `config.js` file to add the following line: ```javascript @@ -563,9 +559,9 @@ variable and afterward the value of the multiCore in the `config.js` file. The r (the standard `http` module is also needed): ```javascript -var iotAgent = require("../lib/iotagent-implementation"), - iotAgentLib = require("iotagent-node-lib"), - config = require("./config"); +var iotAgent = require('../lib/iotagent-implementation'), + iotAgentLib = require('iotagent-node-lib'), + config = require('./config'); ``` It is important to mention the purpose of the `iotAgent` variable. It is the proper implementation of the IoT Agent @@ -581,9 +577,9 @@ about starting the IoTAgent: ```javascript iotAgentLib.startServer(config, iotAgent, function (error) { if (error) { - console.log(context, "Error starting IoT Agent: [%s] Exiting process", error); + console.log(context, 'Error starting IoT Agent: [%s] Exiting process', error); } else { - console.log(context, "IoT Agent started"); + console.log(context, 'IoT Agent started'); } }); ``` @@ -610,7 +606,7 @@ handlers themselves. Here we can see the definition of the configuration handler ```javascript function configurationHandler(configuration, callback) { - console.log("\n\n* REGISTERING A NEW CONFIGURATION:\n%s\n\n", JSON.stringify(configuration, null, 4)); + console.log('\n\n* REGISTERING A NEW CONFIGURATION:\n%s\n\n', JSON.stringify(configuration, null, 4)); callback(null, configuration); } ``` @@ -627,8 +623,8 @@ feature, let's use the provisioning handler to change the value of the type of t ```javascript function provisioningHandler(device, callback) { - console.log("\n\n* REGISTERING A NEW DEVICE:\n%s\n\n", JSON.stringify(device, null, 4)); - device.type = "CertifiedType"; + console.log('\n\n* REGISTERING A NEW DEVICE:\n%s\n\n', JSON.stringify(device, null, 4)); + device.type = 'CertifiedType'; callback(null, device); } ``` diff --git a/doc/installationguide.md b/doc/installationguide.md index 987a9425f..5d054f999 100644 --- a/doc/installationguide.md +++ b/doc/installationguide.md @@ -249,10 +249,6 @@ used for the same purpose. For instance: 'http://192.168.56.1:4041'. - **iotaVersion**: indicates the version of the IoTA that will be displayed in the about method (it should be filled automatically by each IoTA). -- **appendMode**: if this flag is activated (its default behaviour), the update requests to the Context Broker will be - performed always with APPEND type, instead of the default UPDATE. This have implications in the use of attributes - with Context Providers, so this flag should be used with care. This flag is overwritten by `autoprovision` flag in - group or device provision. - **dieOnUnexpectedError**: if this flag is activated, the IoTAgent will not capture global exception, thus dying upon any unexpected error. - **singleConfigurationMode**: enables the Single Configuration mode for backwards compatibility (see description in @@ -352,7 +348,6 @@ overrides. | IOTA_MONGO_SSL | `mongodb.ssl` | | IOTA_MONGO_EXTRAARGS | `mongodb.extraArgs` | | IOTA_SINGLE_MODE | `singleConfigurationMode` | -| IOTA_APPEND_MODE | `appendMode` | | IOTA_POLLING_EXPIRATION | `pollingExpiration` | | IOTA_POLLING_DAEMON_FREQ | `pollingDaemonFrequency` | | IOTA_AUTOCAST | `autocast` | From 7053ce0aa9d5e381946089dc30769907bc3cec8c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ferm=C3=ADn=20Gal=C3=A1n=20M=C3=A1rquez?= Date: Thu, 24 Aug 2023 12:41:10 +0200 Subject: [PATCH 4/5] REMOVE appendMode from config.js example --- config.js | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/config.js b/config.js index 9b93b86ad..3542fc7e2 100644 --- a/config.js +++ b/config.js @@ -76,8 +76,7 @@ var config = { subservice: '/gardens', providerUrl: 'http://192.168.56.1:4041', deviceRegistrationDuration: 'P1M', - defaultType: 'Thing', - appendMode: true + defaultType: 'Thing' }; module.exports = config; From 18e31f95a8c9cf6e45acb44c6aa4a1f741a05abc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ferm=C3=ADn=20Gal=C3=A1n=20M=C3=A1rquez?= Date: Thu, 24 Aug 2023 12:42:04 +0200 Subject: [PATCH 5/5] Apply suggestions from code review --- doc/api.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/api.md b/doc/api.md index 98bde807f..25c7cdebe 100644 --- a/doc/api.md +++ b/doc/api.md @@ -19,7 +19,7 @@ - [Measurement persistence options](#measurement-persistence-options) - [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision) - [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs) - - [Differences between `autoprovision`, `explicitAttrs`.](#differences-between-autoprovision-explicitattrs) + - [Differences between `autoprovision`, `explicitAttrs`](#differences-between-autoprovision-explicitattrs) - [Expression language support](#expression-language-support) - [Examples of JEXL expressions](#examples-of-jexl-expressions) - [Available functions](#available-functions) @@ -446,7 +446,7 @@ depending on the JEXL expression evaluation: - If it evaluates to an array just measures defined in the array (identified by their attribute names, not by their object_id) will be will be propagated to NGSI interface (as in case 3) -### Differences between `autoprovision`, `explicitAttrs`. +### Differences between `autoprovision`, `explicitAttrs` Since those configuration parameters are quite similar, this section is intended to clarify the relation between them.