FIWARE Persisting Context Data using NIFI.postman_collection.json

{
	"info": {
		"_postman_id": "33990d83-5855-48cb-b4ed-48fb14fb0a6b",
		"name": "FIWARE Persisting Context Data using NIFI",
		"description": "This tutorial is an introduction to [FIWARE Draco](http://fiware-draco.readthedocs.io/en/latest/) - a generic enabler which is used to persist context data into third-party databases creating a historical view of the context. The tutorial activates the IoT sensors connected in the [previous tutorial](https://github.com/Fiware/tutorials.IoT-Agent) and persists measurements\nfrom those sensors into a database for further analysis.\n\nThe `docker-compose` files for this tutorial can be found on GitHub: \n\n![GitHub](https://fiware.github.io/tutorials.Historic-Context-NIFI/icon/GitHub-Mark-32px.png) [FIWARE 302: Persisting Context Data NIFI](https://github.com/Fiware/tutorials.Historic-Context-NIFI)\n\n# Data Persistence\n\n> \"Plots within plots, but all roads lead down the dragon’s gullet.\"\n>\n> — George R.R. Martin (A Dance With Dragons)\n\n\nPrevious tutorials have introduced a set of IoT Sensors (providing measurements of the\nstate of the real world), and two FIWARE Components - the **Orion Context Broker** and an **IoT Agent**. \nThis tutorial will introduct a new data persistance component - FIWARE **Draco**.\n\nThe system so far has been built up to handle the current context, in other words it holds the data entities\ndefining the state of the real-world objects at a given moment in time.\n\nFrom this definition you can see - context is only interested in the **current** state of the system\nIt is not the responsibility of any of the existing components to report on the historical state of the system,\nthe context is based on the last measurement each sensor has sent data to the context broker.\n\nIn order to do this, we will need to extend the existing architecture to persist changes of state into a database whenever \nthe context is updated.\n\nPersisting historical context data is useful for big data analysis - it can be used to discover trends, or data \ncan be sampled and aggregated to remove the influence of outlying data measurements. However within each Smart Solution,\nthe significance of each entity type will differ and entities and attributes may need to be sampled at different rates.\n\nSince the business requirements for using context data differ from application to appliation, there is no one standard use \ncase  for historical data persistence - each situation is unique - it is not the case that one size fits all.\nTherefore rather than overloading the context broker with the job of historical context data persistence, this role has been\nseparated out into a separate, highly configurable component - **Draco**.\n\nAs you would expect, **Draco**, as part of an Open Source platform, is technology agnostic regarding the database \nto be used for data persistance. The database you choose to use will depend upon your own business needs. \n\nHowever there is a cost to offering this flexibility - each part of the system must be separately configured and\nnotifications must be set up to only pass the minimal data required as necessary.\n\n\n\n\n#### Device Monitor\n\nFor the purpose of this tutorial, a series of dummy IoT devices have been created, which will be attached to the context broker.\nThe state of each device can be seen on the UltraLight device monitor web-page found at: `http://localhost:3000/device/monitor`\n\n![FIWARE Monitor](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/device-monitor.png)\n\n\n\n\n# Architecture\n\nThis application builds on the components and dummy IoT devices created in \n[previous tutorials](https://github.com/Fiware/tutorials.IoT-Agent/). It will make use of three FIWARE components - \nthe [Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/), the\n[IoT Agent for UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/) and introduce the\n[Draco Generic Enabler](http://fiware-draco.readthedocs.io/en/latest/) for persisting context data to a database.\nAdditional databases are now involved - both the Orion Context Broker and the IoT Agent rely on [MongoDB](https://www.mongodb.com/) technology to keep persistence of the information they hold, and we will be persisting our historical context data another database - either **MySQL** , **PostgreSQL**  or **Mongo-DB** database.\n\n\nTherefore the overall architecture will consist of the following elements:\n\n* Three **FIWARE Generic Enablers**:\n  * The FIWARE [Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/) which will receive requests using [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2)\n  * The FIWARE [IoT Agent for UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/) which will receive southbound requests\n    using [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2) and convert them to \n    [UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual) commands for \n    the devices\n  * FIWARE [Draco](http://fiware-draco.readthedocs.io/en/latest/) which will subscribe to context changes and persist them into a database (**MySQL** , **PostgreSQL**  or **Mongo-DB**)\n* One, two or three of the following **Databases**:\n  * The underlying [MongoDB](https://www.mongodb.com/) database :\n    + Used by the **Orion Context Broker** to hold context data information such as data entities, subscriptions and registrations\n    + Used by the **IoT Agent** to hold device information such as device URLs and Keys\n    + Potentially used as a data sink to hold historical context data.\n  * An additional [PostgreSQL](https://www.postgresql.org/) database :\n    + Potentially used as a data sink to hold historical context data.\n  * An additional [MySQL](https://www.mysql.com/) database :\n     + Potentially used as a data sink to hold historical context data.\n* Three **Context Providers**:\n  * The **Stock Management Frontend** is not used in this tutorial. It does the following:\n    + Display store information and allow users to interact with the dummy IoT devices\n    + Show which products can be bought at each store\n    + Allow users to \"buy\" products and reduce the stock count.\n  * A webserver acting as set of [dummy IoT devices](https://github.com/Fiware/tutorials.IoT-Sensors) using the [UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual) protocol running over HTTP.\n  * The **Context Provider NGSI** proxy is not used in this tutorial. It does the following:\n    + receive requests using [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2)\n    + makes requests to publicly available data sources using their own APIs in a proprietory format \n    + returns context data back to the Orion Context Broker in [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2) format.\n\nSince all interactions between the elements are initiated by HTTP requests, the entities can be containerized and run from exposed ports. \n\nThe specific architecture of each section of the tutorial is discussed below.\n\n\n# Start Up\n\nBefore you start you should ensure that you have obtained or built the necessary Docker images locally. Please run\n\n```console\n./services create\n``` \n\n>**Note** The `context-provider` image has not yet been pushed to Docker hub.\n> Failing to build the Docker sources before proceeding will result in the following error:\n>\n>```\n>Pulling context-provider (fiware/cp-web-app:latest)...\n>ERROR: The image for the service you're trying to recreate has been removed.\n>```\n\n\nThereafter, all services can be initialised from the command line by running the [services](https://github.com/Fiware/tutorials.Historic-Context/blob/master/services) Bash script provided within the repository:\n\n```console\n./services <command>\n``` \n\nWhere `<command>` will vary depending upon the databases we wish to activate.\nThis command will also import seed data from the previous tutorials and provision the dummy IoT sensors on startup.\n\n>:information_source: **Note:** If you want to clean up and start over again you can do so with the following command:\n>\n>```console\n>./services stop\n>``` \n>\n",
		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
	},
	"item": [
		{
			"name": "Mongo DB - Persisting Context",
			"item": [
				{
					"name": "Draco - Check System Health",
					"request": {
						"method": "GET",
						"header": [],
						"url": {
							"raw": "http://{{draco}}/nifi-api/system-diagnostics",
							"protocol": "http",
							"host": [
								"{{draco}}"
							],
							"path": [
								"nifi-api",
								"system-diagnostics"
							]
						},
						"description": "Once Draco is running, you can check the status by making an HTTP request to the exposed draco port to `/nifi-api/system-diagnostics`. If the response is blank, this is usually because Draco is not running or is listening on another port."
					},
					"response": []
				},
				{
					"name": "Orion - Subscribe to Context Changes",
					"request": {
						"method": "POST",
						"header": [
							{
								"key": "Content-Type",
								"value": "application/json"
							},
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n  \"description\": \"Notify Draco of all context changes\",\n  \"subject\": {\n    \"entities\": [\n      {\n        \"idPattern\": \".*\"\n      }\n    ]\n  },\n  \"notification\": {\n    \"http\": {\n      \"url\": \"http://draco:5050/v2/notify\"\n    }\n  },\n  \"throttling\": 5\n}"
						},
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "Once a dynamic context system is up and running, we need to inform **Draco** of changes in context.\n\nThis is done by making a POST request to the `/v2/subscription` endpoint of the Orion Context Broker.\n\n* The `fiware-service` and `fiware-servicepath` headers are used to filter the subscription to only listen to measurements from the attached IoT Sensors\n* The `idPattern` in the request body ensures that Draco will be informed of all context data changes.\n* The notification `url` must match the configured `NIFI_WEB_HTTP_PORT` and the URL the sync is listening on (usually  `/v2/notify`)\n* The `throttling` value defines the rate that changes are sampled.\n\nAs you can see, the database used to persist context data has no impact on the details of the subscription. It is the same for each database."
					},
					"response": []
				},
				{
					"name": "Orion - Check Subscription is working",
					"request": {
						"method": "GET",
						"header": [
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "If a subscription has been created, you can check to see if it is firing by making a GET \nrequest to the `/v2/subscriptions` endpoint.\n\nWithin the `notification` section of the response, you can see several additional `attributes` which describe the health of the subscription\n\nIf the criteria of the subscription have been met, `timesSent` should be greater than `0`.\nA zero value would indicate that the `subject` of the subscription is incorrect or the subscription \nhas created with the wrong `fiware-service-path` or `fiware-service` header\n\nThe `lastNotification` should be a recent timestamp - if this is not the case, then the devices\nare not regularly sending data. Remember to unlock the **Smart Door** and switch on the **Smart Lamp**\n\nThe `lastSuccess` should match the `lastNotification` date - if this is not the case \nthen **Draco** is not receiving the subscription properly. Check that the host name\nand port are correct. \n\nFinally, check that the `status` of the subscription is `active` - an expired subscription\nwill not fire."
					},
					"response": []
				}
			],
			"description": "Persisting historic context data using MongoDB technology is relatively simple to configure since we are already using a\nMongoDB instance to hold data related to the Orion Context Broker and the IoT Agent. The MongoDB instance is listening\non the standard `27017` port and the overall architecture can be seen below:\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/mongo-draco-tutorial.png)\n\n## MongoDB - Database Server Configuration\n\n```yaml\nmongo-db:\n    image: mongo:3.6\n    hostname: mongo-db\n    container_name: db-mongo\n    ports:\n        - \"27017:27017\"\n    networks:\n        - default\n\n```\n\n## MongoDB - Draco Configuration\n\n```yaml\ndraco:\n    image: ging/fiware-draco:1.1.0\n    container_name: draco\n    depends_on:\n        - mongo-db\n    environment:\n        - NIFI_WEB_HTTP_PORT=9090\n    ports:\n        - \"9090:9090\"\n        - \"5050:5050\"\n    healthcheck:\n        test: curl --fail -s http://localhost:9090/nifi-api/system-diagnostics || exit 1\n```\n\nThe `draco` container is listening on two ports:\n\n-   The Subscription Port for Draco - `5050` is where the service will be listening for notifications from the Orion\n    context broker\n-   The Web interface for Draco - `9090` is exposed purely for configuring the processors\n\n\n## MongoDB - Start up\n\nTo start the system with a **MongoDB** database only, run the following command:\n\n```console\n./services mongodb\n```\n\nThen go to your browser and open Draco using this URL `http://localhost:9090/nifi`\n\nNow go to the Components toolbar which is placed in the upper section of the NiFi GUI, find the template icon and drag\nand drop it inside the Draco user space. At this point, a popup should be displayed with a list of all the templates\navailable. Please select the template MONGO-TUTORIAL.\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/mongo-tutorial-template.png)\n\nSelect all the processors (press shift and click on every processor) and start them by clicking on the start button.\nNow, you can see that the status icon of each processor turned from red to green.\n\n### Generating Context Data\n\nFor the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated.\nThe dummy IoT Sensors can be used to do this. Open the device monitor page at `http://localhost:3000/device/monitor`\nand unlock a **Smart Door** and switch on a **Smart Lamp**. This can be done by selecting an appropriate the command \nfrom the drop down list and pressing the `send` button. The stream of measurements coming from the devices can then\nbe seen on the same page:\n\n![](https://fiware.github.io/tutorials.Historic-Context/img/door-open.gif)",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "a63e990e-c30a-419c-a54c-5d24baa792f2",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "d2710f35-1e52-4ac5-87ad-d7a93c640386",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				}
			]
		},
		{
			"name": "Mongo DB - Reading Data",
			"item": [],
			"description": "To read mongo-db data from the command line, we will need access to the `mongo` tool run an interactive instance\nof the `mongo` image as shown to obtain a command line prompt:\n\n```console\ndocker run -it --network fiware_default  --entrypoint /bin/bash mongo\n```\n\nYou can then log into to the running `mongo-db` database by using the command line as shown:\n\n```bash\nmongo --host mongo-db\n```\n\n### Show Available Databases on the MongoDB server\n\nTo show the list of available databases, run the statement as shown:\n\n#### Query:\n\n```\nshow dbs\n```\n\n#### Result:\n\n```\nadmin          0.000GB\niotagentul     0.000GB\nlocal          0.000GB\norion          0.000GB\norion-openiot  0.000GB\nsth_openiot    0.000GB\n```\n\nThe result include two databases `admin` and `local` which are set up by default by **MongoDB**, along with four databases\ncreated by the FIWARE platform. The Orion Context Broker has created two separate database instance for each `fiware-service`\n- the Store enitities were created without defining a `fiware-service` and therefore are held within the `orion` database,\nwhereas the IoT device entities were created using the `openiot` `fiware-service` header and are held separately. The IoT Agent was intialized to hold the IoT sensor data in a separate  **MongoDB** database called `iotagentul`. \n\nAs a result of the subscription of Draco to Orion Context Broker, a new database has been created called `sth_openiot`. The default value for a **Mongo DB** database holding historic context consists of the `sth_` prefix followed by the `fiware-service` header - therefore `sth_openiot` holds the historic context of the IoT devices.\n\n### Read Historical Context from the MongoDB server\n\n#### Query:\n\n```\nuse sth_openiot\nshow collections\n```\n\n#### Result:\n\n```\nswitched to db sth_openiot\n\nsth_/_Door:001_Door\nsth_/_Door:001_Door.aggr\nsth_/_Lamp:001_Lamp\nsth_/_Lamp:001_Lamp.aggr\nsth_/_Motion:001_Motion\nsth_/_Motion:001_Motion.aggr\n```\n\nLooking within the `sth_openiot` you will see that a series of tables have been created. The names of each table consist\nof the `sth_` prefix followed by the `fiware-servicepath` header followed by the entity id. Two table are created for\neach entity - the `.aggr` table holds some aggregated data which will be accessed in a later tutorial. The raw data\ncan be seen in the tables without the `.aggr` suffix. \n\nThe historical data can be seen by looking at the data within each table, by default each row will contain the sampled value of a single attribute. \n\n#### Query:\n\n```\ndb[\"sth_/_Door:001_Door\"].find().limit(10)\n```\n\n#### Result:\n\n```\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f7635d\"), \"recvTime\" : ISODate(\"2018-06-12T10:46:30.897Z\"), \"attrName\" : \"TimeInstant\", \"attrType\" : \"ISO8601\", \"attrValue\" : \"2018-06-12T10:46:30.836Z\" }\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f7635e\"), \"recvTime\" : ISODate(\"2018-06-12T10:46:30.897Z\"), \"attrName\" : \"close_status\", \"attrType\" : \"commandStatus\", \"attrValue\" : \"UNKNOWN\" }\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f7635f\"), \"recvTime\" : ISODate(\"2018-06-12T10:46:30.897Z\"), \"attrName\" : \"lock_status\", \"attrType\" : \"commandStatus\", \"attrValue\" : \"UNKNOWN\" }\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f76360\"), \"recvTime\" : ISODate(\"2018-06-12T10:46:30.897Z\"), \"attrName\" : \"open_status\", \"attrType\" : \"commandStatus\", \"attrValue\" : \"UNKNOWN\" }\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f76361\"), \"recvTime\" : ISODate(\"2018-06-12T10:46:30.836Z\"), \"attrName\" : \"refStore\", \"attrType\" : \"Relationship\", \"attrValue\" : \"Store:001\" }\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f76362\"), \"recvTime\" : ISODate(\"2018-06-12T10:46:30.836Z\"), \"attrName\" : \"state\", \"attrType\" : \"Text\", \"attrValue\" : \"CLOSED\" }\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f76363\"), \"recvTime\" : ISODate(\"2018-06-12T10:45:26.368Z\"), \"attrName\" : \"unlock_info\", \"attrType\" : \"commandResult\", \"attrValue\" : \" unlock OK\" }\n{ \"_id\" : ObjectId(\"5b1fa48630c49e0012f76364\"), \"recvTime\" : ISODate(\"2018-06-12T10:45:26.368Z\"), \"attrName\" : \"unlock_status\", \"attrType\" : \"commandStatus\", \"attrValue\" : \"OK\" }\n{ \"_id\" : ObjectId(\"5b1fa4c030c49e0012f76385\"), \"recvTime\" : ISODate(\"2018-06-12T10:47:28.081Z\"), \"attrName\" : \"TimeInstant\", \"attrType\" : \"ISO8601\", \"attrValue\" : \"2018-06-12T10:47:28.038Z\" }\n{ \"_id\" : ObjectId(\"5b1fa4c030c49e0012f76386\"), \"recvTime\" : ISODate(\"2018-06-12T10:47:28.081Z\"), \"attrName\" : \"close_status\", \"attrType\" : \"commandStatus\", \"attrValue\" : \"UNKNOWN\" }\n```\n\nThe usual **Mongo-DB** query syntax can be used to filter appropriate fields and values. For example to read the rate at which the **Motion Sensor** with the `id=Motion:001_Motion` is accumulating, you would make a query as follows:\n\n#### Query:\n\n```\ndb[\"sth_/_Motion:001_Motion\"].find({attrName: \"count\"},{_id: 0, attrType: 0, attrName: 0 } ).limit(10)\n```\n\n#### Result:\n\n```\n{ \"recvTime\" : ISODate(\"2018-06-12T10:46:18.756Z\"), \"attrValue\" : \"8\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:46:36.881Z\"), \"attrValue\" : \"10\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:46:42.947Z\"), \"attrValue\" : \"11\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:46:54.893Z\"), \"attrValue\" : \"13\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:47:00.929Z\"), \"attrValue\" : \"15\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:47:06.954Z\"), \"attrValue\" : \"17\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:47:15.983Z\"), \"attrValue\" : \"19\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:47:49.090Z\"), \"attrValue\" : \"23\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:47:58.112Z\"), \"attrValue\" : \"25\" }\n{ \"recvTime\" : ISODate(\"2018-06-12T10:48:28.218Z\"), \"attrValue\" : \"29\" }\n```\n\n\nTo leave the MongoDB client and leave interactive mode, run the following:\n\n```console\nexit\n```\n\n```console\nexit\n```\n",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "9e70a754-3893-4617-adc7-c0471f7372f7",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "c43d1162-ac29-4852-9094-b1b86604352b",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				}
			]
		},
		{
			"name": "PostgreSQL - Persisting Context",
			"item": [
				{
					"name": "Draco - Check System Health",
					"request": {
						"method": "GET",
						"header": [],
						"url": {
							"raw": "http://{{draco}}/nifi-api/system-diagnostics",
							"protocol": "http",
							"host": [
								"{{draco}}"
							],
							"path": [
								"nifi-api",
								"system-diagnostics"
							]
						},
						"description": "Once Draco is running, you can check the status by making an HTTP request to the exposed draco port to `/nifi-api/system-diagnostics`. If the response is blank, this is usually because Draco is not running or is listening on another port."
					},
					"response": []
				},
				{
					"name": "Orion - Subscribe to Context Changes",
					"request": {
						"method": "POST",
						"header": [
							{
								"key": "Content-Type",
								"value": "application/json"
							},
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n  \"description\": \"Notify Draco of all context changes\",\n  \"subject\": {\n    \"entities\": [\n      {\n        \"idPattern\": \".*\"\n      }\n    ]\n  },\n  \"notification\": {\n    \"http\": {\n      \"url\": \"http://draco:5050/v2/notify\"\n    }\n  },\n  \"throttling\": 5\n}"
						},
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "Once a dynamic context system is up and running, we need to inform **Draco** of changes in context.\n\nThis is done by making a POST request to the `/v2/subscription` endpoint of the Orion Context Broker.\n\n* The `fiware-service` and `fiware-servicepath` headers are used to filter the subscription to only listen to measurements from the attached IoT Sensors\n* The `idPattern` in the request body ensures that Draco will be informed of all context data changes.\n* The notification `url` must match the configured `NIFI_WEB_HTTP_PORT` and the URL the sync is listening on (usually  `/v2/notify`)\n* The `throttling` value defines the rate that changes are sampled.\n\nAs you can see, the database used to persist context data has no impact on the details of the subscription. It is the same for each database."
					},
					"response": []
				},
				{
					"name": "Orion - Check Subscription is working",
					"request": {
						"method": "GET",
						"header": [
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "If a subscription has been created, you can check to see if it is firing by making a GET \nrequest to the `/v2/subscriptions` endpoint.\n\nWithin the `notification` section of the response, you can see several additional `attributes` which describe the health of the subscription\n\nIf the criteria of the subscription have been met, `timesSent` should be greater than `0`.\nA zero value would indicate that the `subject` of the subscription is incorrect or the subscription \nhas created with the wrong `fiware-service-path` or `fiware-service` header\n\nThe `lastNotification` should be a recent timestamp - if this is not the case, then the devices\nare not regularly sending data. Remember to unlock the **Smart Door** and switch on the **Smart Lamp**\n\nThe `lastSuccess` should match the `lastNotification` date - if this is not the case \nthen **Draco** is not receiving the subscription properly. Check that the host name\nand port are correct. \n\nFinally, check that the `status` of the subscription is `active` - an expired subscription\nwill not fire."
					},
					"response": []
				}
			],
			"description": "To persist historic context data into an alternative database such as **PostgreSQL**, we will need an additional\ncontainer which hosts the PostgreSQL server - the default Docker image for this data can be used. The PostgreSQL\ninstance is listening on the standard `5432` port and the overall architecture can be seen below:\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/postgres-draco-tutorial.png)\n\nWe now have a system with two databases, since the MongoDB container is still required to hold data related to the Orion\nContext Broker and the IoT Agent.\n\n## PostgreSQL - Database Server Configuration\n\n```yaml\npostgres-db:\n    image: postgres:latest\n    hostname: postgres-db\n    container_name: db-postgres\n    expose:\n        - \"5432\"\n    ports:\n        - \"5432:5432\"\n    networks:\n        - default\n    environment:\n        - \"POSTGRES_PASSWORD=password\"\n        - \"POSTGRES_USER=postgres\"\n        - \"POSTGRES_DB=postgres\"\n```\n\nThe `postgres-db` container is listening on a single port:\n\n-   Port `5432` is the default port for a PostgreSQL server. It has been exposed so you can also run the `pgAdmin4` tool\n    to display database data if you wish\n\nThe `postgres-db` container is driven by environment variables as shown:\n\n| Key               | Value.     | Description                               |\n| ----------------- | ---------- | ----------------------------------------- |\n| POSTGRES_PASSWORD | `password` | Password for the PostgreSQL database user |\n| POSTGRES_USER     | `postgres` | Username for the PostgreSQL database user |\n| POSTGRES_DB       | `postgres` | The name of the PostgreSQL database       |\n\n> :information_source: **Note:** Passing the Username and Password in plain text environment variables like this is a\n> security risk. Whereas this is acceptable practice in a tutorial, for a production environment, you can avoid this\n> risk by applying [Docker Secrets](https://blog.docker.com/2017/02/docker-secrets-management/)\n\n## PostgreSQL - Draco Configuration\n\n```yaml\ndraco:\n    image: ging/fiware-draco:1.1.0\n    container_name: draco\n    depends_on:\n        - postgres-db\n    environment:\n        - NIFI_WEB_HTTP_PORT=9090\n    ports:\n        - \"9090:9090\"\n        - \"5050:5050\"\n    healthcheck:\n        test: curl --fail -s http://localhost:9090/nifi-api/system-diagnostics || exit 1\n```\n\nThe `draco` container is listening on two ports:\n\n-   The Subscription Port for Draco - `5050` is where the service will be listening for notifications from the Orion\n    context broker\n-   The Web interface for Draco - `9090` is exposed purely for configuring the processors.\n\n## PostgreSQL - Start up\n\nTo start the system with a **PostgreSQL** database run the following command:\n\n```console\n./services postgres\n```\n\nThen go to your browser and open Draco using this URL `http://localhost:9090/nifi`\n\nNow go to the Components toolbar which is placed in the upper section of the NiFi GUI, find the template icon and drag\nand drop it inside the Draco user space. At this point, a popup should be displayed with a list of all the templates\navailable. Please select the template POSTGRESQL-TUTORIAL.\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/postgres-tutorial-template.png)\n\nBefore starting the processors, you need to set your PostgreSQL password and enable the DBCConnectionPool controller.\nFor doing that please follow the instructions:\n\n1.  Do right click on any part of the Draco GUI user space, and then click on configure.\n    ![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step1.png)\n\n2.  Go to the Controller Services Tab, at this point a list of controllers should be displayed, locate the\n    DBCConnectionPool controller.\n\n3.  Click on the configuration button of the \"DBCPConnectionPool\"\n    ![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step2.png)\n\n4.  Go to the controller Properties tab and put \"password\" in the password field, then apply the changes.\n    ![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/controller-postgresql.png)\n\n5.  Enable the processor by clicking on the thunder icon and then click on enable, then close the controller\n    configuration page.\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step4.png)\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step5.png)\n\n6.  Select all the processors (press shift and click on every processor) and start them by clicking on the start button.\n    Now, you can see that the status icon of each processor turned from red to green.\n\n\n\n\n## PostgreSQL - Start up\n\nTo start the system with a **PostgreSQL** database run the following command:\n\n```console\n./services postgres\n``` \n\n## PostgreSQL- Database Server Configuration\n\n```yaml\npostgres-db:\n      image: postgres:latest\n      hostname: postgres-db\n      container_name: db-postgres\n      expose:\n        - \"5432\"\n      ports:\n        - \"5432:5432\"\n      networks:\n        - default\n      environment:\n        - \"POSTGRES_PASSWORD=password\"\n        - \"POSTGRES_USER=postgres\"\n        - \"POSTGRES_DB=postgres\"\n\n```\n\nThe `postgres-db` container is listening on a single port: \n\n* Port `5432` is the default port for a PostgreSQL server. It has been exposed so you can also run the `pgAdmin4` tool to display database data if you wish\n\nThe `postgres-db` container is driven by environment variables as shown:\n\n| Key             |Value.    |Description                    |\n|-----------------|----------|-------------------------------|\n|POSTGRES_PASSWORD|`password`| Password for the PostgreSQL database user|\n|POSTGRES_USER    |`postgres`| Username for the PostgreSQL database user|\n|POSTGRES_DB      |`postgres`| The name of the PostgreSQL database      | \n\n\n\n## PostgreSQL - Draco Configuration\n\n```yaml\ndraco:\n    image: fiware/draco-ngsi:latest\n    hostname: draco\n    container_name: fiware-draco\n    networks:\n        - default\n    depends_on:\n        - postgres-db\n    expose:\n        - \"5080\"\n    ports:\n        - \"5050:5050\"\n        - \"5080:5080\"\n    environment:\n        - \"CYGNUS_POSTGRESQL_HOST=postgres-db\"\n        - \"CYGNUS_POSTGRESQL_PORT=5432\"\n        - \"CYGNUS_POSTGRESQL_USER=postgres\" \n        - \"CYGNUS_POSTGRESQL_PASS=password\" \n        - \"CYGNUS_LOG_LEVEL=DEBUG\"\n        - \"CYGNUS_SERVICE_PORT=5050\"\n        - \"CYGNUS_API_PORT=5080\"\n        - \"CYGNUS_POSTGRESQL_ENABLE_CACHE=true\"\n```\n\nThe `draco` container is listening on two ports: \n\n* The service will be listening on port `5050` for notifications from the Orion context broker\n* Port `5080` is exposed purely for tutorial access - so that cUrl or Postman can make provisioning commands\n  without being part of the same network.\n\n\n### Generating Context Data\n\nFor the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated.\nThe dummy IoT Sensors can be used to do this. Open the device monitor page at `http://localhost:3000/device/monitor`\nand unlock a **Smart Door** and switch on a **Smart Lamp**. This can be done by selecting an appropriate the command \nfrom the drop down list and pressing the `send` button. The stream of measurements coming from the devices can then\nbe seen on the same page:\n\n![](https://fiware.github.io/tutorials.Historic-Context/img/door-open.gif)",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "f8ca5d71-4eb4-42b9-95e0-c3ef11567514",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "43d701ab-eb41-472e-875e-4ac51e27579a",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				}
			]
		},
		{
			"name": "PostgreSQL - Reading Data",
			"item": [],
			"description": "To read PostgreSQL data from the command line, we will need access to the `postgres` client, to do this, run an\ninteractive instance of the `postgresql-client` image supplying the connection string as shown to obtain a command \nline prompt:\n\n```console\ndocker run -it --rm  --network fiware_default jbergknoff/postgresql-client \\\n   postgresql://postgres:password@postgres-db:5432/postgres\n```\n\n### Show Available Databases on the PostgreSQL server\n\nTo show the list of available databases, run the statement as shown:\n\n#### Query:\n\n```\n\\list\n```\n\n#### Result:\n\n```\n   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   \n-----------+----------+----------+------------+------------+-----------------------\n postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | \n template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +\n           |          |          |            |            | postgres=CTc/postgres\n template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +\n           |          |          |            |            | postgres=CTc/postgres\n(3 rows)\n```\n\nThe result include two template databases `template0` and `template1` as well as the `postgres` database setup when the docker container was started.\n\n\nTo show the list of available schemas, run the statement as shown:\n\n#### Query:\n\n```\n\\dn\n```\n\n#### Result:\n\n```\n  List of schemas\n  Name   |  Owner   \n---------+----------\n openiot | postgres\n public  | postgres\n(2 rows)\n```\n\n\nAs a result of the subscription of Draco to Orion Context Broker, a new schema has been created called `openiot`. \nThe name of the schema matches the `fiware-service` header - therefore `openiot` holds the historic context of the\nIoT devices.\n\n\n\n### Read Historical Context from the PostgreSQL server\n\nOnce running a docker container within the network, it is possible to obtain information about the running\ndatabase.\n\n\n#### Query:\n\n```sql\nSELECT table_schema,table_name\nFROM information_schema.tables\nWHERE table_schema ='openiot'\nORDER BY table_schema,table_name;\n```\n\n#### Result:\n\n```\n table_schema |    table_name     \n--------------+-------------------\n openiot      | door_001_door\n openiot      | lamp_001_lamp\n openiot      | motion_001_motion\n(3 rows)\n```\n\nThe `table_schema` matches the `fiware-service` header supplied with the context data:\n\nTo read the data within a table, run a select statement as shown:\n\n#### Query:\n\n```sql\nSELECT * FROM openiot.motion_001_motion limit 10;\n```\n\n#### Result:\n\n```\n  recvtimets   |         recvtime         | fiwareservicepath |  entityid  | entitytype |  attrname   |   attrtype   |        attrvalue         |                                    attrmd                                    \n---------------+--------------------------+-------------------+------------+------------+-------------+--------------+--------------------------+------------------------------------------------------------------------------\n 1528803005491 | 2018-06-12T11:30:05.491Z | /                 | Motion:001 | Motion     | TimeInstant | ISO8601      | 2018-06-12T11:30:05.423Z | []\n 1528803005491 | 2018-06-12T11:30:05.491Z | /                 | Motion:001 | Motion     | count       | Integer      | 7                        | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:30:05.423Z\"}]\n 1528803005491 | 2018-06-12T11:30:05.491Z | /                 | Motion:001 | Motion     | refStore    | Relationship | Store:001                | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:30:05.423Z\"}]\n 1528803035501 | 2018-06-12T11:30:35.501Z | /                 | Motion:001 | Motion     | TimeInstant | ISO8601      | 2018-06-12T11:30:35.480Z | []\n 1528803035501 | 2018-06-12T11:30:35.501Z | /                 | Motion:001 | Motion     | count       | Integer      | 10                       | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:30:35.480Z\"}]\n 1528803035501 | 2018-06-12T11:30:35.501Z | /                 | Motion:001 | Motion     | refStore    | Relationship | Store:001                | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:30:35.480Z\"}]\n 1528803041563 | 2018-06-12T11:30:41.563Z | /                 | Motion:001 | Motion     | TimeInstant | ISO8601      | 2018-06-12T11:30:41.520Z | []\n 1528803041563 | 2018-06-12T11:30:41.563Z | /                 | Motion:001 | Motion     | count       | Integer      | 12                       | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:30:41.520Z\"}]\n 1528803041563 | 2018-06-12T11:30:41.563Z | /                 | Motion:001 | Motion     | refStore    | Relationship | Store:001                | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:30:41.520Z\"}]\n 1528803047545 | 2018-06-12T11:30:47.545Z | /     \n```\n\nThe usual **PostgreSQL** query syntax can be used to filter appropriate fields and values. For example to read the rate at which the **Motion Sensor** with the `id=Motion:001_Motion` is accumulating, you would make a query as follows:\n\n#### Query:\n\n```sql\nSELECT recvtime, attrvalue FROM openiot.motion_001_motion WHERE attrname ='count'  limit 10;\n```\n\n#### Result:\n\n```\n         recvtime         | attrvalue \n--------------------------+-----------\n 2018-06-12T11:30:05.491Z | 7\n 2018-06-12T11:30:35.501Z | 10\n 2018-06-12T11:30:41.563Z | 12\n 2018-06-12T11:30:47.545Z | 13\n 2018-06-12T11:31:02.617Z | 15\n 2018-06-12T11:31:32.718Z | 20\n 2018-06-12T11:31:38.733Z | 22\n 2018-06-12T11:31:50.780Z | 24\n 2018-06-12T11:31:56.825Z | 25\n 2018-06-12T11:31:59.790Z | 26\n(10 rows)\n```\n\nTo leave the Postgres client and leave interactive mode, run the following:\n\n```console\n\\q\n```\n You will then return to the commmand line.\n\n",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "9547319e-f5df-4a3d-9a69-86221f7aeb2f",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "eaf3c847-e1eb-48e2-aadd-35caf0463942",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				}
			]
		},
		{
			"name": "MySQL - Persisting Context",
			"item": [
				{
					"name": "Draco - Check System Health",
					"request": {
						"method": "GET",
						"header": [],
						"url": {
							"raw": "http://{{draco}}/nifi-api/system-diagnostics",
							"protocol": "http",
							"host": [
								"{{draco}}"
							],
							"path": [
								"nifi-api",
								"system-diagnostics"
							]
						},
						"description": "Once Draco is running, you can check the status by making an HTTP request to the exposed draco port to `/nifi-api/system-diagnostics`. If the response is blank, this is usually because Draco is not running or is listening on another port."
					},
					"response": []
				},
				{
					"name": "Orion - Subscribe to Context Changes",
					"request": {
						"method": "POST",
						"header": [
							{
								"key": "Content-Type",
								"value": "application/json"
							},
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n  \"description\": \"Notify Draco of all context changes\",\n  \"subject\": {\n    \"entities\": [\n      {\n        \"idPattern\": \".*\"\n      }\n    ]\n  },\n  \"notification\": {\n    \"http\": {\n      \"url\": \"http://draco:5050/v2/notify\"\n    }\n  },\n  \"throttling\": 5\n}"
						},
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "Once a dynamic context system is up and running, we need to inform **Draco** of changes in context.\n\nThis is done by making a POST request to the `/v2/subscription` endpoint of the Orion Context Broker.\n\n* The `fiware-service` and `fiware-servicepath` headers are used to filter the subscription to only listen to measurements from the attached IoT Sensors\n* The `idPattern` in the request body ensures that Draco will be informed of all context data changes.\n* The notification `url` must match the configured `NIFI_WEB_HTTP_PORT` and the URL the sync is listening on (usually  `/v2/notify`)\n* The `throttling` value defines the rate that changes are sampled.\n\nAs you can see, the database used to persist context data has no impact on the details of the subscription. It is the same for each database."
					},
					"response": []
				},
				{
					"name": "Orion - Check Subscription is working",
					"request": {
						"method": "GET",
						"header": [
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "If a subscription has been created, you can check to see if it is firing by making a GET \nrequest to the `/v2/subscriptions` endpoint.\n\nWithin the `notification` section of the response, you can see several additional `attributes` which describe the health of the subscription\n\nIf the criteria of the subscription have been met, `timesSent` should be greater than `0`.\nA zero value would indicate that the `subject` of the subscription is incorrect or the subscription \nhas created with the wrong `fiware-service-path` or `fiware-service` header\n\nThe `lastNotification` should be a recent timestamp - if this is not the case, then the devices\nare not regularly sending data. Remember to unlock the **Smart Door** and switch on the **Smart Lamp**\n\nThe `lastSuccess` should match the `lastNotification` date - if this is not the case \nthen **Draco** is not receiving the subscription properly. Check that the host name\nand port are correct. \n\nFinally, check that the `status` of the subscription is `active` - an expired subscription\nwill not fire."
					},
					"response": []
				}
			],
			"description": "Similarly, to persisting historic context data into **MySQL**, we will again need an additional container which hosts\nthe MySQL server, once again the default Docker image for this data can be used. The MySQL instance is listening on the\nstandard `3306` port and the overall architecture can be seen below:\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/mysql-draco-tutorial.png)\n\nOnce again we have a system with two databases, since the MongoDB container is still required to hold data related to\nthe Orion Context Broker and the IoT Agent.\n\n## MySQL - Database Server Configuration\n\n```yaml\nmysql-db:\n    restart: always\n    image: mysql:5.7\n    hostname: mysql-db\n    container_name: db-mysql\n    expose:\n        - \"3306\"\n    ports:\n        - \"3306:3306\"\n    networks:\n        - default\n    environment:\n        - \"MYSQL_ROOT_PASSWORD=123\"\n        - \"MYSQL_ROOT_HOST=%\"\n```\n\n> :information_source: **Note:** Using the default `root` user and displaying the password in an environment variables\n> like this is a security risk. Whereas this is acceptable practice in a tutorial, for a production environment, you can\n> avoid this risk by setting up another user and applying\n> [Docker Secrets](https://blog.docker.com/2017/02/docker-secrets-management/)\n\nThe `mysql-db` container is listening on a single port:\n\n-   Port `3306` is the default port for a MySQL server. It has been exposed so you can also run other database tools to\n    display data if you wish\n\nThe `mysql-db` container is driven by environment variables as shown:\n\n| Key                 | Value.     | Description                                                                                                                                                                                           |\n| ------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| MYSQL_ROOT_PASSWORD | `123`.     | specifies a password that is set for the MySQL `root` account.                                                                                                                                        |\n| MYSQL_ROOT_HOST     | `postgres` | By default, MySQL creates the `root'@'localhost` account. This account can only be connected to from inside the container. Setting this environment variable allows root connections from other hosts |\n\n## MySQL - Draco Configuration\n\n```yaml\ndraco:\n    image: ging/fiware-draco:1.1.0\n    container_name: draco\n    depends_on:\n        - mysql-db\n    environment:\n        - NIFI_WEB_HTTP_PORT=9090\n    ports:\n        - \"9090:9090\"\n        - \"5050:5050\"\n    healthcheck:\n        test: curl --fail -s http://localhost:9090/nifi-api/system-diagnostics || exit 1\n```\n\nThe `draco` container is listening on two ports:\n\n-   The Subscription Port for Draco - `5050` is where the service will be listening for notifications from the Orion\n    context broker\n-   The Web interface for Draco - `9090` is exposed purely for configuring the processors\n\n\n\n## MySQL - Start up\n\nTo start the system with a **MySQL** database run the following command:\n\n```console\n./services mysql\n```\n\nThen go to your browser and open Draco using this URL `http://localhost:9090/nifi`\n\nNow go to the Components toolbar which is placed in the upper section of the NiFi GUI, find the template icon and drag\nand drop it inside the Draco user space. At this point, a popup should be displayed with a list of all the templates\navailable. Please select the template MYSQL-TUTORIAL.\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/draco-template1.png)\n\nBefore starting the processors, you need to set your MySQL password and enable the DBCConnectionPool controller. For\ndoing that please follow the instructions:\n\n1.  Do right click on any part of the Draco GUI user space, and then click on configure.\n    ![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step1.png)\n\n2.  Go to the Controller Services Tab, at this point a list of controllers should be displayed, locate the\n    DBCConnectionPool controller.\n\n3.  Click on the configuration button of the \"DBCPConnectionPool\"\n    ![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step2.png)\n\n4.  Go to the controller Properties tab and put \"123\" in the password field, then apply the changes.\n    ![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step3.png)\n\n5.  Enable the processor by clicking on the thunder icon and then click on enable, then close the controller\n    configuration page. ![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step4.png)\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/step5.png)\n\n6.  Select all the processors (press shift and click on every processor) and start them by clicking on the start button.\n    Now, you can see that the status icon of each processor turned from red to green.\n\n### Generating Context Data\n\nFor the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated.\nThe dummy IoT Sensors can be used to do this. Open the device monitor page at `http://localhost:3000/device/monitor`\nand unlock a **Smart Door** and switch on a **Smart Lamp**. This can be done by selecting an appropriate the command \nfrom the drop down list and pressing the `send` button. The stream of measurements coming from the devices can then\nbe seen on the same page:\n\n![](https://fiware.github.io/tutorials.Historic-Context/img/door-open.gif)",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "1c3c984c-886c-464c-b936-a5b355263c89",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "cafefa9a-6026-45d2-8877-2298a955ba2d",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				}
			]
		},
		{
			"name": "MySQL - Reading Data",
			"item": [],
			"description": "To read MySQL data from the command line, we will need access to the `mysql` client, to do this, run an\ninteractive instance of the `mysql` image supplying the connection string as shown to obtain a command \nline prompt:\n\n```console\ndocker exec -it  db-mysql mysql -h mysql-db -P 3306  -u root -p123\n```\n\n### Show Available Databases on the MySQL server\n\nTo show the list of available databases, run the statement as shown:\n\n#### Query:\n\n```sql\nSHOW DATABASES;\n```\n\n#### Result:\n\n```\n+--------------------+\n| Database           |\n+--------------------+\n| information_schema |\n| mysql              |\n| openiot            |\n| performance_schema |\n| sys                |\n+--------------------+\n5 rows in set (0.00 sec)\n```\n\n\nTo show the list of available schemas, run the statement as shown:\n\n#### Query:\n\n```sql\nSHOW SCHEMAS;\n```\n\n#### Result:\n\n```\n+--------------------+\n| Database           |\n+--------------------+\n| information_schema |\n| mysql              |\n| openiot            |\n| performance_schema |\n| sys                |\n+--------------------+\n5 rows in set (0.00 sec)\n```\n\n\nAs a result of the subscription of Draco to Orion Context Broker, a new schema has been created called `openiot`. \nThe name of the schema matches the `fiware-service` header - therefore `openiot` holds the historic context of the\nIoT devices.\n\n\n\n### Read Historical Context from the PostgreSQL server\n\nOnce running a docker container within the network, it is possible to obtain information about the running\ndatabase.\n\n\n#### Query:\n\n```sql\nSHOW tables FROM openiot;\n```\n\n#### Result:\n\n```\n table_schema |    table_name     \n--------------+-------------------\n openiot      | door_001_door\n openiot      | lamp_001_lamp\n openiot      | motion_001_motion\n(3 rows)\n```\n\nThe `table_schema` matches the `fiware-service` header supplied with the context data:\n\nTo read the data within a table, run a select statement as shown:\n\n#### Query:\n\n```sql\nSELECT * FROM openiot.Motion_001_Motion limit 10;\n```\n\n#### Result:\n\n```\n+---------------+-------------------------+-------------------+------------+------------+-------------+--------------+--------------------------+------------------------------------------------------------------------------+\n| recvTimeTs    | recvTime                | fiwareServicePath | entityId   | entityType | attrName    | attrType     | attrValue                | attrMd                                                                       |\n+---------------+-------------------------+-------------------+------------+------------+-------------+--------------+--------------------------+------------------------------------------------------------------------------+\n| 1528804397955 | 2018-06-12T11:53:17.955 | /                 | Motion:001 | Motion     | TimeInstant | ISO8601      | 2018-06-12T11:53:17.923Z | []                                                                           |\n| 1528804397955 | 2018-06-12T11:53:17.955 | /                 | Motion:001 | Motion     | count       | Integer      | 3                        | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:53:17.923Z\"}] |\n| 1528804397955 | 2018-06-12T11:53:17.955 | /                 | Motion:001 | Motion     | refStore    | Relationship | Store:001                | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:53:17.923Z\"}] |\n| 1528804403954 | 2018-06-12T11:53:23.954 | /                 | Motion:001 | Motion     | TimeInstant | ISO8601      | 2018-06-12T11:53:23.928Z | []                                                                           |\n| 1528804403954 | 2018-06-12T11:53:23.954 | /                 | Motion:001 | Motion     | count       | Integer      | 5                        | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:53:23.928Z\"}] |\n| 1528804403954 | 2018-06-12T11:53:23.954 | /                 | Motion:001 | Motion     | refStore    | Relationship | Store:001                | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:53:23.928Z\"}] |\n| 1528804409970 | 2018-06-12T11:53:29.970 | /                 | Motion:001 | Motion     | TimeInstant | ISO8601      | 2018-06-12T11:53:29.948Z | []                                                                           |\n| 1528804409970 | 2018-06-12T11:53:29.970 | /                 | Motion:001 | Motion     | count       | Integer      | 7                        | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:53:29.948Z\"}] |\n| 1528804409970 | 2018-06-12T11:53:29.970 | /                 | Motion:001 | Motion     | refStore    | Relationship | Store:001                | [{\"name\":\"TimeInstant\",\"type\":\"ISO8601\",\"value\":\"2018-06-12T11:53:29.948Z\"}] |\n| 1528804446083 | 2018-06-12T11:54:06.83  | /                 | Motion:001 | Motion     | TimeInstant | ISO8601      | 2018-06-12T11:54:06.062Z | []                                                                           |\n+---------------+-------------------------+-------------------+------------+------------+-------------+--------------+--------------------------+------------------------------------------------------------------------------+    \n```\n\nThe usual **MySQL** query syntax can be used to filter appropriate fields and values. For example to read the rate at which the **Motion Sensor** with the `id=Motion:001_Motion` is accumulating, you would make a query as follows:\n\n#### Query:\n\n```sql\nSELECT recvtime, attrvalue FROM openiot.Motion_001_Motion WHERE attrname ='count' LIMIT 10;\n```\n\n#### Result:\n\n```\n+-------------------------+-----------+\n| recvtime                | attrvalue |\n+-------------------------+-----------+\n| 2018-06-12T11:53:17.955 | 3         |\n| 2018-06-12T11:53:23.954 | 5         |\n| 2018-06-12T11:53:29.970 | 7         |\n| 2018-06-12T11:54:06.83  | 12        |\n| 2018-06-12T11:54:12.132 | 13        |\n| 2018-06-12T11:54:24.177 | 14        |\n| 2018-06-12T11:54:36.196 | 16        |\n| 2018-06-12T11:54:42.195 | 18        |\n| 2018-06-12T11:55:24.300 | 23        |\n| 2018-06-12T11:55:30.350 | 25        |\n+-------------------------+-----------+\n10 rows in set (0.00 sec)\n```\n\nTo leave the MySQL client and leave interactive mode, run the following:\n\n```console\n\\q\n```\n You will then return to the commmand line.\n",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "0a433918-9a68-4cbe-9bdc-031270322304",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "db11ec7d-1bde-4317-8318-db7b0ebf5c61",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				}
			]
		},
		{
			"name": "Multi-Agent Persisting to Multiple Databases",
			"item": [
				{
					"name": "Draco - Obtain version information",
					"request": {
						"method": "GET",
						"header": [],
						"url": {
							"raw": "http://{{draco}}/v1/version",
							"protocol": "http",
							"host": [
								"{{draco}}"
							],
							"path": [
								"v1",
								"version"
							]
						},
						"description": "Once Draco is running, You can check the status by making an HTTP request to the exposed `CYGNUS_API_PORT` port. \nIf the response is blank, this is usually because Draco is not running or is listening on another port.\n\n>**Troubleshooting:** What if the response is blank ?\n>\n> * To check that a docker container is running try\n>\n>```bash\n>docker ps\n>```\n>\n>You should see several containers running. If `draco` is not running, you can restart the containers as necessary."
					},
					"response": []
				},
				{
					"name": "Orion - Subscribe to Context Changes",
					"request": {
						"method": "POST",
						"header": [
							{
								"key": "Content-Type",
								"value": "application/json"
							},
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n  \"description\": \"Notify Draco of all context changes for MySQL on port 5050\",\n  \"subject\": {\n    \"entities\": [\n      {\n        \"idPattern\": \".*\"\n      }\n    ]\n  },\n  \"notification\": {\n    \"http\": {\n      \"url\": \"http://draco:5050/notify\"\n    },\n    \"attrsFormat\": \"legacy\"\n  },\n  \"throttling\": 5\n}"
						},
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "Once a dynamic context system is up and running, we need to inform **Draco** of changes in context.\n\nThis is done by making a POST request to the `/v2/subscription` endpoint of the Orion Context Broker.\n\n* The `fiware-service` and `fiware-servicepath` headers are used to filter the subscription to only listen to measurements from the attached IoT Sensors\n* The `idPattern` in the request body ensures that Draco will be informed of all context data changes.\n* The `attrsFormat=legacy` is required since Draco currently only accepts notifications in the older NGSI v1 format.\n* The `throttling` value defines the rate that changes are sampled.\n\nWhen running in **multi-agent** mode, the  notification `url` for each subscription must match the defaults for the given database.\n\nThe default port mapping can be seen below:\n\n| sink       | port |\n|-----------:|-----:|\n| mysql      | 5050 |\n| mongo      | 5051 |\n| ckan       | 5052 |\n| hdfs       | 5053 |\n| postgresql | 5054 | \n| cartodb    | 5055 | \n\nSince this subscription is using port `5050` the context data will eventually be persisted to the *MySQL* database."
					},
					"response": []
				},
				{
					"name": "Orion - Check Subscription is working",
					"request": {
						"method": "GET",
						"header": [
							{
								"key": "fiware-service",
								"value": "openiot"
							},
							{
								"key": "fiware-servicepath",
								"value": "/"
							}
						],
						"url": {
							"raw": "http://{{orion}}/v2/subscriptions/",
							"protocol": "http",
							"host": [
								"{{orion}}"
							],
							"path": [
								"v2",
								"subscriptions",
								""
							]
						},
						"description": "If a subscription has been created, you can check to see if it is firing by making a GET \nrequest to the `/v2/subscriptions` endpoint.\n\nWithin the `notification` section of the response, you can see several additional `attributes` which describe the health of the subscription\n\nIf the criteria of the subscription have been met, `timesSent` should be greater than `0`.\nA zero value would indicate that the `subject` of the subscription is incorrect or the subscription \nhas created with the wrong `fiware-service-path` or `fiware-service` header\n\nThe `lastNotification` should be a recent timestamp - if this is not the case, then the devices\nare not regularly sending data. Remember to unlock the **Smart Door** and switch on the **Smart Lamp**\n\nThe `lastSuccess` should match the `lastNotification` date - if this is not the case \nthen **Draco** is not receiving the subscription properly. Check that the host name\nand port are correct. \n\nFinally, check that the `status` of the subscription is `active` - an expired subscription\nwill not fire."
					},
					"response": []
				}
			],
			"description": "It is also possible to configure Draco to populate multiple databases simultaneously. We can combine the architecture\nfrom the three previous examples and configure Draco to store data in multiple sinks.\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/multiple-draco-tutorial.png)\n\nWe now have a system with three databases, PostgreSQL and MySQL for data persistence and MongoDB for both data\npersistence and holding data related to the Orion Context Broker and the IoT Agent.\n\n## Multi-Agent - Draco Configuration for Multiple Databases\n\n```yaml\ndraco:\n    image: ging/fiware-draco:1.1.0\n    container_name: draco\n    depends_on:\n        - mysql-db\n        - mongo-db\n        - postgres-db\n    environment:\n        - NIFI_WEB_HTTP_PORT=9090\n    ports:\n        - \"9090:9090\"\n        - \"5050:5050\"\n    healthcheck:\n        test: curl --fail -s http://localhost:9090/nifi-api/system-diagnostics || exit 1\n```\n\nThe `draco` container is listening on two ports:\n\n-   The Subscription Port for Draco - `5050` is where the service will be listening for notifications from the Orion\n    context broker\n-   The Web interface for Draco - `9090` is exposed purely for configuring the processors\n\n\n## Multi-Agent - Start up\n\nTo start the system with **multiple** databases run the following command:\n\n```console\n./services multiple\n```\n\nThen go to your browser and open Draco using this URL `http://localhost:9090/nifi`\n\nNow go to the Components toolbar which is placed in the upper section of the NiFi GUI, find the template icon and drag\nand drop it inside the Draco user space. At this point, a popup should be displayed with a list of all the templates\navailable. Please select the template MULTIPLE-SINKS-TUTORIAL.\n\n![](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/multiple-tutorial-template.png)\n\nNow repeat the process for setting the password in the controller \"DBCPConnectionPool\" each connection MySQL and\nPostgreSQL\n\nSelect all the processors (press shift and click on every processor) and start them by clicking on the start button.\nNow, you can see that the status icon of each processor turned from red to green.\n\n### Generating Context Data\n\nFor the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated.\nThe dummy IoT Sensors can be used to do this. Open the device monitor page at `http://localhost:3000/device/monitor`\nand unlock a **Smart Door** and switch on a **Smart Lamp**. This can be done by selecting an appropriate the command \nfrom the drop down list and pressing the `send` button. The stream of measurements coming from the devices can then\nbe seen on the same page:\n\n![](https://fiware.github.io/tutorials.Historic-Context/img/door-open.gif)",
			"event": [
				{
					"listen": "prerequest",
					"script": {
						"id": "1c3c984c-886c-464c-b936-a5b355263c89",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				},
				{
					"listen": "test",
					"script": {
						"id": "cafefa9a-6026-45d2-8877-2298a955ba2d",
						"type": "text/javascript",
						"exec": [
							""
						]
					}
				}
			]
		}
	],
	"event": [
		{
			"listen": "prerequest",
			"script": {
				"id": "d1cff9ac-8932-417c-b6e2-582ba712ff79",
				"type": "text/javascript",
				"exec": [
					""
				]
			}
		},
		{
			"listen": "test",
			"script": {
				"id": "cc3f098c-a786-4112-a15e-b1c8c83cc657",
				"type": "text/javascript",
				"exec": [
					""
				]
			}
		}
	],
	"variable": [
		{
			"id": "ae13f117-12ed-42fa-babd-d465dbfdf57f",
			"key": "orion",
			"value": "localhost:1026",
			"type": "string"
		},
		{
			"id": "209da7ff-3c70-4e87-a881-6165824232a8",
			"key": "draco",
			"value": "localhost:9090",
			"type": "string"
		}
	],
	"protocolProfileBehavior": {}
}