{"activeVersionTag":"latest","latestAvailableVersionTag":"latest","collection":{"info":{"_postman_id":"33990d83-5855-48cb-b4ed-48fb14fb0a6b","name":"FIWARE Persisting Context Data using NIFI","description":"This tutorial is an introduction to [FIWARE Draco](http://fiware-draco.readthedocs.io/en/latest/) - a generic enabler which is used to persist context data into third-party databases creating a historical view of the context. The tutorial activates the IoT sensors connected in the [previous tutorial](https://github.com/Fiware/tutorials.IoT-Agent) and persists measurements\nfrom those sensors into a database for further analysis.\n\nThe `docker-compose` files for this tutorial can be found on GitHub: \n\n![GitHub](https://fiware.github.io/tutorials.Historic-Context-NIFI/icon/GitHub-Mark-32px.png) [FIWARE 302: Persisting Context Data NIFI](https://github.com/Fiware/tutorials.Historic-Context-NIFI)\n\n# Data Persistence\n\n> \"Plots within plots, but all roads lead down the dragon’s gullet.\"\n>\n> — George R.R. Martin (A Dance With Dragons)\n\n\nPrevious tutorials have introduced a set of IoT Sensors (providing measurements of the\nstate of the real world), and two FIWARE Components - the **Orion Context Broker** and an **IoT Agent**. \nThis tutorial will introduct a new data persistance component - FIWARE **Draco**.\n\nThe system so far has been built up to handle the current context, in other words it holds the data entities\ndefining the state of the real-world objects at a given moment in time.\n\nFrom this definition you can see - context is only interested in the **current** state of the system\nIt is not the responsibility of any of the existing components to report on the historical state of the system,\nthe context is based on the last measurement each sensor has sent data to the context broker.\n\nIn order to do this, we will need to extend the existing architecture to persist changes of state into a database whenever \nthe context is updated.\n\nPersisting historical context data is useful for big data analysis - it can be used to discover trends, or data \ncan be sampled and aggregated to remove the influence of outlying data measurements. However within each Smart Solution,\nthe significance of each entity type will differ and entities and attributes may need to be sampled at different rates.\n\nSince the business requirements for using context data differ from application to appliation, there is no one standard use \ncase  for historical data persistence - each situation is unique - it is not the case that one size fits all.\nTherefore rather than overloading the context broker with the job of historical context data persistence, this role has been\nseparated out into a separate, highly configurable component - **Draco**.\n\nAs you would expect, **Draco**, as part of an Open Source platform, is technology agnostic regarding the database \nto be used for data persistance. The database you choose to use will depend upon your own business needs. \n\nHowever there is a cost to offering this flexibility - each part of the system must be separately configured and\nnotifications must be set up to only pass the minimal data required as necessary.\n\n\n\n\n#### Device Monitor\n\nFor the purpose of this tutorial, a series of dummy IoT devices have been created, which will be attached to the context broker.\nThe state of each device can be seen on the UltraLight device monitor web-page found at: `http://localhost:3000/device/monitor`\n\n![FIWARE Monitor](https://fiware.github.io/tutorials.Historic-Context-NIFI/img/device-monitor.png)\n\n\n\n\n# Architecture\n\nThis application builds on the components and dummy IoT devices created in \n[previous tutorials](https://github.com/Fiware/tutorials.IoT-Agent/). It will make use of three FIWARE components - \nthe [Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/), the\n[IoT Agent for UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/) and introduce the\n[Draco Generic Enabler](http://fiware-draco.readthedocs.io/en/latest/) for persisting context data to a database.\nAdditional databases are now involved - both the Orion Context Broker and the IoT Agent rely on [MongoDB](https://www.mongodb.com/) technology to keep persistence of the information they hold, and we will be persisting our historical context data another database - either **MySQL** , **PostgreSQL**  or **Mongo-DB** database.\n\n\nTherefore the overall architecture will consist of the following elements:\n\n* Three **FIWARE Generic Enablers**:\n  * The FIWARE [Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/) which will receive requests using [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2)\n  * The FIWARE [IoT Agent for UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/) which will receive southbound requests\n    using [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2) and convert them to \n    [UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual) commands for \n    the devices\n  * FIWARE [Draco](http://fiware-draco.readthedocs.io/en/latest/) which will subscribe to context changes and persist them into a database (**MySQL** , **PostgreSQL**  or **Mongo-DB**)\n* One, two or three of the following **Databases**:\n  * The underlying [MongoDB](https://www.mongodb.com/) database :\n    + Used by the **Orion Context Broker** to hold context data information such as data entities, subscriptions and registrations\n    + Used by the **IoT Agent** to hold device information such as device URLs and Keys\n    + Potentially used as a data sink to hold historical context data.\n  * An additional [PostgreSQL](https://www.postgresql.org/) database :\n    + Potentially used as a data sink to hold historical context data.\n  * An additional [MySQL](https://www.mysql.com/) database :\n     + Potentially used as a data sink to hold historical context data.\n* Three **Context Providers**:\n  * The **Stock Management Frontend** is not used in this tutorial. It does the following:\n    + Display store information and allow users to interact with the dummy IoT devices\n    + Show which products can be bought at each store\n    + Allow users to \"buy\" products and reduce the stock count.\n  * A webserver acting as set of [dummy IoT devices](https://github.com/Fiware/tutorials.IoT-Sensors) using the [UltraLight 2.0](http://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual) protocol running over HTTP.\n  * The **Context Provider NGSI** proxy is not used in this tutorial. It does the following:\n    + receive requests using [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2)\n    + makes requests to publicly available data sources using their own APIs in a proprietory format \n    + returns context data back to the Orion Context Broker in [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2) format.\n\nSince all interactions between the elements are initiated by HTTP requests, the entities can be containerized and run from exposed ports. \n\nThe specific architecture of each section of the tutorial is discussed below.\n\n\n# Start Up\n\nBefore you start you should ensure that you have obtained or built the necessary Docker images locally. Please run\n\n```console\n./services create\n``` \n\n>**Note** The `context-provider` image has not yet been pushed to Docker hub.\n> Failing to build the Docker sources before proceeding will result in the following error:\n>\n>```\n>Pulling context-provider (fiware/cp-web-app:latest)...\n>ERROR: The image for the service you're trying to recreate has been removed.\n>```\n\n\nThereafter, all services can be initialised from the command line by running the [services](https://github.com/Fiware/tutorials.Historic-Context/blob/master/services) Bash script provided within the repository:\n\n```console\n./services <command>\n``` \n\nWhere `<command>` will vary depending upon the databases we wish to activate.\nThis command will also import seed data from the previous tutorials and provision the dummy IoT sensors on startup.\n\n>:information_source: **Note:** If you want to clean up and start over again you can do so with the following command:\n>\n>```console\n>./services stop\n>``` \n>\n","schema":"https://schema.getpostman.com/json/collection/v2.0.0/collection.json","isPublicCollection":true,"owner":"513743","team":157450,"collectionId":"33990d83-5855-48cb-b4ed-48fb14fb0a6b","publishedId":"S17qTpnL","public":true,"publicUrl":"https://documenter-api.postman.tech/view/513743/S17qTpnL","privateUrl":"https://go.postman.co/documentation/513743-33990d83-5855-48cb-b4ed-48fb14fb0a6b","customColor":{"top-bar":"FFFFFF","right-sidebar":"303030","highlight":"233C68"},"documentationLayout":"classic-double-column","version":"8.10.1","publishDate":"2020-01-02T11:06:41.000Z","activeVersionTag":"latest","documentationTheme":"light","metaTags":{},"logos":{}},"statusCode":200},"environments":[],"user":{"authenticated":false,"permissions":{"publish":false}},"run":{"button":{"js":"https://run.pstmn.io/button.js","css":"https://run.pstmn.io/button.css"}},"web":"https://www.getpostman.com/","team":{"logo":"https://res.cloudinary.com/postman/image/upload/t_team_logo_pubdoc/v1/team/d7085d490b9144732c65203aa6e3b68b31884d1c33a86b8a00d15da75147ae33","favicon":""},"isEnvFetchError":false,"languages":"[{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"HttpClient\"},{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"RestSharp\"},{\"key\":\"curl\",\"label\":\"cURL\",\"variant\":\"cURL\"},{\"key\":\"dart\",\"label\":\"Dart\",\"variant\":\"http\"},{\"key\":\"go\",\"label\":\"Go\",\"variant\":\"Native\"},{\"key\":\"http\",\"label\":\"HTTP\",\"variant\":\"HTTP\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"OkHttp\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"Unirest\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"Fetch\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"jQuery\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"XHR\"},{\"key\":\"c\",\"label\":\"C\",\"variant\":\"libcurl\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Axios\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Native\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Request\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Unirest\"},{\"key\":\"objective-c\",\"label\":\"Objective-C\",\"variant\":\"NSURLSession\"},{\"key\":\"ocaml\",\"label\":\"OCaml\",\"variant\":\"Cohttp\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"cURL\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"Guzzle\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"HTTP_Request2\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"pecl_http\"},{\"key\":\"powershell\",\"label\":\"PowerShell\",\"variant\":\"RestMethod\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"http.client\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"Requests\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"httr\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"RCurl\"},{\"key\":\"ruby\",\"label\":\"Ruby\",\"variant\":\"Net::HTTP\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"Httpie\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"wget\"},{\"key\":\"swift\",\"label\":\"Swift\",\"variant\":\"URLSession\"}]","languageSettings":[{"key":"csharp","label":"C#","variant":"HttpClient"},{"key":"csharp","label":"C#","variant":"RestSharp"},{"key":"curl","label":"cURL","variant":"cURL"},{"key":"dart","label":"Dart","variant":"http"},{"key":"go","label":"Go","variant":"Native"},{"key":"http","label":"HTTP","variant":"HTTP"},{"key":"java","label":"Java","variant":"OkHttp"},{"key":"java","label":"Java","variant":"Unirest"},{"key":"javascript","label":"JavaScript","variant":"Fetch"},{"key":"javascript","label":"JavaScript","variant":"jQuery"},{"key":"javascript","label":"JavaScript","variant":"XHR"},{"key":"c","label":"C","variant":"libcurl"},{"key":"nodejs","label":"NodeJs","variant":"Axios"},{"key":"nodejs","label":"NodeJs","variant":"Native"},{"key":"nodejs","label":"NodeJs","variant":"Request"},{"key":"nodejs","label":"NodeJs","variant":"Unirest"},{"key":"objective-c","label":"Objective-C","variant":"NSURLSession"},{"key":"ocaml","label":"OCaml","variant":"Cohttp"},{"key":"php","label":"PHP","variant":"cURL"},{"key":"php","label":"PHP","variant":"Guzzle"},{"key":"php","label":"PHP","variant":"HTTP_Request2"},{"key":"php","label":"PHP","variant":"pecl_http"},{"key":"powershell","label":"PowerShell","variant":"RestMethod"},{"key":"python","label":"Python","variant":"http.client"},{"key":"python","label":"Python","variant":"Requests"},{"key":"r","label":"R","variant":"httr"},{"key":"r","label":"R","variant":"RCurl"},{"key":"ruby","label":"Ruby","variant":"Net::HTTP"},{"key":"shell","label":"Shell","variant":"Httpie"},{"key":"shell","label":"Shell","variant":"wget"},{"key":"swift","label":"Swift","variant":"URLSession"}],"languageOptions":[{"label":"C# - HttpClient","value":"csharp - HttpClient - C#"},{"label":"C# - RestSharp","value":"csharp - RestSharp - C#"},{"label":"cURL - cURL","value":"curl - cURL - cURL"},{"label":"Dart - http","value":"dart - http - Dart"},{"label":"Go - Native","value":"go - Native - Go"},{"label":"HTTP - HTTP","value":"http - HTTP - HTTP"},{"label":"Java - OkHttp","value":"java - OkHttp - Java"},{"label":"Java - Unirest","value":"java - Unirest - Java"},{"label":"JavaScript - Fetch","value":"javascript - Fetch - JavaScript"},{"label":"JavaScript - jQuery","value":"javascript - jQuery - JavaScript"},{"label":"JavaScript - XHR","value":"javascript - XHR - JavaScript"},{"label":"C - libcurl","value":"c - libcurl - C"},{"label":"NodeJs - Axios","value":"nodejs - Axios - NodeJs"},{"label":"NodeJs - Native","value":"nodejs - Native - NodeJs"},{"label":"NodeJs - Request","value":"nodejs - Request - NodeJs"},{"label":"NodeJs - Unirest","value":"nodejs - Unirest - NodeJs"},{"label":"Objective-C - NSURLSession","value":"objective-c - NSURLSession - Objective-C"},{"label":"OCaml - Cohttp","value":"ocaml - Cohttp - OCaml"},{"label":"PHP - cURL","value":"php - cURL - PHP"},{"label":"PHP - Guzzle","value":"php - Guzzle - PHP"},{"label":"PHP - HTTP_Request2","value":"php - HTTP_Request2 - PHP"},{"label":"PHP - pecl_http","value":"php - pecl_http - PHP"},{"label":"PowerShell - RestMethod","value":"powershell - RestMethod - PowerShell"},{"label":"Python - http.client","value":"python - http.client - Python"},{"label":"Python - Requests","value":"python - Requests - Python"},{"label":"R - httr","value":"r - httr - R"},{"label":"R - RCurl","value":"r - RCurl - R"},{"label":"Ruby - Net::HTTP","value":"ruby - Net::HTTP - Ruby"},{"label":"Shell - Httpie","value":"shell - Httpie - Shell"},{"label":"Shell - wget","value":"shell - wget - Shell"},{"label":"Swift - URLSession","value":"swift - URLSession - Swift"}],"layoutOptions":[{"value":"classic-single-column","label":"Single Column"},{"value":"classic-double-column","label":"Double Column"}],"versionOptions":[],"environmentOptions":[{"value":"0","label":"No Environment"}],"canonicalUrl":"https://documenter.gw.postman.com/view/metadata/S17qTpnL"}