{"activeVersionTag":"latest","latestAvailableVersionTag":"latest","collection":{"info":{"_postman_id":"17fa5c4e-3836-49ce-8769-df5e9bba674e","name":"NGSI-LD QuantumLeap","description":"This tutorial is an introduction to [FIWARE QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/) - a generic\nenabler which is used to persist context data into a **CrateDB** database. The tutorial activates the IoT sensors\nconnected in the [previous tutorial](https://github.com/FIWARE/tutorials.IoT-Agent) and persists measurements from those\nsensors into the database. To retrieve time-based aggregations of such data, users can either use **QuantumLeap** query\nAPI or connect directly to the **CrateDB** HTTP endpoint. Results are visualised on a graph or via the **Grafana** time\nseries analytics tool.\n\nThe `docker-compose` files for this tutorial can be found on GitHub: \n\n![GitHub](https://fiware.github.io/tutorials.Time-Series-Data/icon/GitHub-Mark-32px.png) [FIWARE 301: Persisting Context Data](https://github.com/Fiware/tutorials.Time-Series-Data)\n\n> \"Forever is composed of nows.\"\n>\n> — Emily Dickinson\n\nFIWARE [QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/) is an time-based data-persistence generic enabler\ncreated specifically to persist and query time-series database (currently CrateDB and TimescaleDB). The component can\nrespond to NGSI-v2 or NGSI-LD subscriptions.\n\n[CrateDB](https://crate.io/) is a distributed SQL DBMS designed for use with the internet of Things. It is capable of\ningesting a large number of data points per second and can be queried in real-time. The database is designed for the\nexecution of complex queries such as geospatial and time series data. Retrieval of this historic context data allows for\nthe creation of graphs and dashboards displaying trends over time.\n\n[TimescaleDB](https://www.timescale.com/) scales PostgreSQL for time-series data via automatic partitioning across time\nand space (partitioning key), yet retains the standard PostgreSQL interface. In other words, TimescaleDB exposes what\nlook like regular tables, but are actually only an abstraction (or a virtual view) of many individual tables comprising\nthe actual data. In combination with [PostGIS](https://postgis.net/) extension can support geo-timeseries.\n\n## Analyzing time series data\n\nThe appropriate use of time series data analysis will depend on your use case and the reliability of the data\nmeasurements you receive. Time series data analysis can be used to answer questions such as:\n\n-   What was the maximum measurement of a device within a given time period?\n-   What was the average measurement of a device within a given time period?\n-   What was the sum of the measurements sent by a device within a given time period?\n\nIt can also be used to reduce the significance of each individual data point to exclude outliers by smoothing.\n\n#### Grafana\n\n[Grafana](https://grafana.com/) is an open source software for time series analytics tool which will be used during this\ntutorial. It integrates with a variety of time-series databases including **CrateDB** and **TimescaleDB**. It is\navailable licensed under the Apache License 2.0. More information can be found at `https://grafana.com/`.\n\n#### Device Monitor\n\nFor the purpose of this tutorial, a series of dummy agricultural IoT devices have been created, which will be attached\nto the context broker. Details of the architecture and protocol used can be found in the\n[IoT Sensors tutorial](https://github.com/FIWARE/tutorials.IoT-Sensors/tree/NGSI-LD) The state of each device can be\nseen on the UltraLight device monitor web page found at: `http://localhost:3000/device/monitor`\n\n![FIWARE Monitor](https://fiware.github.io/tutorials.Time-Series-Data/img/farm-devices.png)\n\n#### Device History\n\nOnce **QuantumLeap** has started aggregating data, the historical state of each device can be seen on the device history\nweb page found at: `http://localhost:3000/device/history/urn:ngsi-ld:Farm:001`\n\n![](https://fiware.github.io/tutorials.Time-Series-Data/img/history-graphs.png)\n\n# Architecture\n\nThis application builds on the components and dummy IoT devices created in\n[previous tutorials](https://github.com/FIWARE/tutorials.IoT-Agent/). It will use three FIWARE components: the\n[Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/), the\n[IoT Agent for Ultralight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/), and\n[QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/) .\n\nTherefore the overall architecture will consist of the following elements:\n\n-   The **FIWARE Generic Enablers**:\n\n    -   The [Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/) which will receive requests using\n        [NGSI-LD](https://forge.etsi.org/swagger/ui/?url=https://forge.etsi.org/gitlab/NGSI-LD/NGSI-LD/raw/master/spec/updated/full_api.json)\n    -   The FIWARE [IoT Agent for UltraLight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/) which will\n        receive southbound requests using\n        [NGSI-LD](https://forge.etsi.org/swagger/ui/?url=https://forge.etsi.org/gitlab/NGSI-LD/NGSI-LD/raw/master/spec/updated/full_api.json)\n        and convert them to\n        [UltraLight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual)\n        commands for the devices\n    -   FIWARE [QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/) subscribed to context changes and\n        persisting them into a **CrateDB** database\n\n-   A [MongoDB](https://www.mongodb.com/) database:\n\n    -   Used by the **Orion Context Broker** to hold context data information such as data entities, subscriptions and\n        registrations\n    -   Used by the **IoT Agent** to hold device information such as device URLs and Keys\n\n-   A [CrateDB](https://crate.io/) database:\n\n    -   Used as a data sink to hold time-based historical context data\n    -   offers an HTTP endpoint to interpret time-based data queries\n\n-   The **Tutorial Application** does the following:\n    -   Offers static `@context` files defining the context entities within the system.\n    -   Acts as set of dummy [agricultural IoT devices](https://github.com/FIWARE/tutorials.IoT-Sensors/tree/NGSI-LD)\n        using the\n        [UltraLight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual)\n        protocol running over HTTP.\n\nSince all interactions between the elements are initiated by HTTP requests, the entities can be containerized and run\nfrom exposed ports.\n\nThe overall architecture can be seen below:\n\n![](https://fiware.github.io/tutorials.Time-Series-Data/img/architecture.png)\n\n# Prerequisites\n\n## Docker and Docker Compose\n\nTo keep things simple all components will be run using [Docker](https://www.docker.com). **Docker** is a container\ntechnology which allows to different components isolated into their respective environments.\n\n-   To install Docker on Windows follow the instructions [here](https://docs.docker.com/docker-for-windows/)\n-   To install Docker on Mac follow the instructions [here](https://docs.docker.com/docker-for-mac/)\n-   To install Docker on Linux follow the instructions [here](https://docs.docker.com/install/)\n\n**Docker Compose** is a tool for defining and running multi-container Docker applications. A series of\n[YAML files](https://raw.githubusercontent.com/Fiware/tutorials.Time-Series-Data/master/docker-compose.yml) are used to\nconfigure the required services for the application. This means all container services can be brought up in a single\ncommand. Docker Compose is installed by default as part of Docker for Windows and Docker for Mac, however Linux users\nwill need to follow the instructions found [here](https://docs.docker.com/compose/install/)\n\nYou can check your current **Docker** and **Docker Compose** versions using the following commands:\n\n```console\ndocker-compose -v\ndocker version\n```\n\nPlease ensure that you are using Docker version 18.03 or higher and Docker Compose 1.21 or higher and upgrade if\nnecessary.\n\n## Cygwin for Windows\n\nWe will start up our services using a simple Bash script. Windows users should download [cygwin](http://www.cygwin.com/)\nto provide a command-line functionality similar to a Linux distribution on Windows.\n\n# Start Up\n\nBefore you start, you should ensure that you have obtained or built the necessary Docker images locally. Please clone\nthe repository and create the necessary images by running the commands as shown:\n\n```console\ngit clone https://github.com/FIWARE/tutorials.Time-Series-Data.git\ncd tutorials.Time-Series-Data\ngit checkout NGSI-LD\n\n./services create\n```\n\nThereafter, all services can be initialized from the command-line by running the\n[services](https://github.com/FIWARE/tutorials.Time-Series-Data/blob/NGSI-v2/services) Bash script provided within the\nrepository:\n\n```console\n./services start\n```\n\n> :information_source: **Note:** If you want to clean up and start over again you can do so with the following command:\n>\n> ```console\n> ./services stop\n> ```\n\n# Connecting FIWARE to a CrateDB Database via QuantumLeap\n\nIn the configuration, **QuantumLeap** listens to NGSI LD notifications on port `8868` and persists historic context data\nto the **CrateDB**. **CrateDB** is accessible using port `4200` and can either be queried directly or attached to the\nGrafana analytics tool. The rest of the system providing the context data has been described in previous tutorials\n\n## CrateDB Database Server Configuration\n\n```yaml\ncrate-db:\n    image: crate:4.1.4\n    hostname: crate-db\n    ports:\n        - \"4200:4200\"\n        - \"4300:4300\"\n    command:\n        crate -Clicense.enterprise=false -Cauth.host_based.enabled=false  -Ccluster.name=democluster\n        -Chttp.cors.enabled=true -Chttp.cors.allow-origin=\"*\"\n    environment:\n        - CRATE_HEAP_SIZE=2g\n```\n\nIf CrateDB exits immediately with a\n`max virtual memory areas vm.max_map_filling [65530] is too low, increase to at least [262144]` error, this can be fixed\nby running the `sudo sysctl -w vm.max_map_filling=262144` command on the host machine. For further information look\nwithin the CrateDB\n[documentation](https://crate.io/docs/crate/howtos/en/latest/admin/bootstrap-checks.html#bootstrap-checks) and Docker\n[troubleshooting guide](https://crate.io/docs/crate/howtos/en/latest/deployment/containers/docker.html#troubleshooting)\n\n## QuantumLeap Configuration\n\n```yaml\nquantumleap:\n    image: smartsdk/quantumleap\n    hostname: quantumleap\n    ports:\n        - \"8668:8668\"\n    depends_on:\n        - crate-db\n    environment:\n        - CRATE_HOST=crate-db\n```\n\n## Grafana Configuration\n\n```yaml\ngrafana:\n    image: grafana/grafana\n    depends_on:\n        - cratedb\n    ports:\n        - \"3003:3000\"\n    environment:\n        - GF_INSTALL_PLUGINS=https://github.com/orchestracities/grafana-map-plugin/archive/master.zip;grafana-map-plugin,grafana-clock-panel,grafana-worldmap-panel\n```\n\nThe `quantumleap` container is listening on one port:\n\n-   The Operations for port for QuantumLeap - `8668` is where the service will be listening for notifications from the\n    Orion context broker and where users can query data from.\n\nThe `CRATE_HOST` environment variable defines the location where the data will be persisted.\n\nThe `cratedb` container is listening on two ports:\n\n-   The Admin UI is available on port `4200`\n-   The transport protocol is available on `port 4300`\n\nThe `grafana` container has connected up port `3000` internally with port `3003` externally. This is because the Grafana\nUI is usually available on port `3000`, but this port has already been taken by the dummy devices UI so it has been\nshifted to another port. The Grafana Environment variables are described within their own\n[documentation](https://grafana.com/docs/installation/configuration/). The configuration ensures we will be able to\nconnect to the **CrateDB** database later on in the tutorial. The configuration also imports a custom map plugin that\nhelps you in displaying NGSI v2 entities over a map.\n\n### Generating Context Data\n\nFor the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated. The\ndummy IoT Sensors can be used to do this.\n\nDetails of various buildings around the farm can be found in the tutorial application. Open\n`http://localhost:3000/app/farm/urn:ngsi-ld:Building:farm001` to display a building with an associated filling sensor\nand thermostat.\n\n![](https://fiware.github.io/tutorials.Subscriptions/img/fmis.png)\n\nRemove some hay from the barn, update the thermostat and open the device monitor page at\n`http://localhost:3000/device/monitor` and start a **Tractor** and switch on a **Smart Lamp**. This can be done by\nselecting an appropriate command from the drop down list and pressing the `send` button. The stream of measurements\ncoming from the devices can then be seen on the same page.\n","schema":"https://schema.getpostman.com/json/collection/v2.0.0/collection.json","isPublicCollection":false,"owner":"513743","team":157450,"collectionId":"17fa5c4e-3836-49ce-8769-df5e9bba674e","publishedId":"TWDUpxxx","public":true,"publicUrl":"https://documenter-api.postman.tech/view/513743/TWDUpxxx","privateUrl":"https://go.postman.co/documentation/513743-17fa5c4e-3836-49ce-8769-df5e9bba674e","customColor":{"top-bar":"FFFFFF","right-sidebar":"303030","highlight":"233c68"},"documentationLayout":"classic-double-column","customisation":null,"version":"8.10.0","publishDate":"2021-02-17T10:17:37.000Z","activeVersionTag":"latest","documentationTheme":"light","metaTags":{},"logos":{}},"statusCode":200},"environments":[],"user":{"authenticated":false,"permissions":{"publish":false}},"run":{"button":{"js":"https://run.pstmn.io/button.js","css":"https://run.pstmn.io/button.css"}},"web":"https://www.getpostman.com/","team":{"logo":"https://res.cloudinary.com/postman/image/upload/t_team_logo_pubdoc/v1/team/d7085d490b9144732c65203aa6e3b68b31884d1c33a86b8a00d15da75147ae33","favicon":""},"isEnvFetchError":false,"languages":"[{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"HttpClient\"},{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"RestSharp\"},{\"key\":\"curl\",\"label\":\"cURL\",\"variant\":\"cURL\"},{\"key\":\"dart\",\"label\":\"Dart\",\"variant\":\"http\"},{\"key\":\"go\",\"label\":\"Go\",\"variant\":\"Native\"},{\"key\":\"http\",\"label\":\"HTTP\",\"variant\":\"HTTP\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"OkHttp\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"Unirest\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"Fetch\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"jQuery\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"XHR\"},{\"key\":\"c\",\"label\":\"C\",\"variant\":\"libcurl\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Axios\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Native\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Request\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Unirest\"},{\"key\":\"objective-c\",\"label\":\"Objective-C\",\"variant\":\"NSURLSession\"},{\"key\":\"ocaml\",\"label\":\"OCaml\",\"variant\":\"Cohttp\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"cURL\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"Guzzle\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"HTTP_Request2\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"pecl_http\"},{\"key\":\"powershell\",\"label\":\"PowerShell\",\"variant\":\"RestMethod\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"http.client\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"Requests\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"httr\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"RCurl\"},{\"key\":\"ruby\",\"label\":\"Ruby\",\"variant\":\"Net::HTTP\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"Httpie\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"wget\"},{\"key\":\"swift\",\"label\":\"Swift\",\"variant\":\"URLSession\"}]","languageSettings":[{"key":"csharp","label":"C#","variant":"HttpClient"},{"key":"csharp","label":"C#","variant":"RestSharp"},{"key":"curl","label":"cURL","variant":"cURL"},{"key":"dart","label":"Dart","variant":"http"},{"key":"go","label":"Go","variant":"Native"},{"key":"http","label":"HTTP","variant":"HTTP"},{"key":"java","label":"Java","variant":"OkHttp"},{"key":"java","label":"Java","variant":"Unirest"},{"key":"javascript","label":"JavaScript","variant":"Fetch"},{"key":"javascript","label":"JavaScript","variant":"jQuery"},{"key":"javascript","label":"JavaScript","variant":"XHR"},{"key":"c","label":"C","variant":"libcurl"},{"key":"nodejs","label":"NodeJs","variant":"Axios"},{"key":"nodejs","label":"NodeJs","variant":"Native"},{"key":"nodejs","label":"NodeJs","variant":"Request"},{"key":"nodejs","label":"NodeJs","variant":"Unirest"},{"key":"objective-c","label":"Objective-C","variant":"NSURLSession"},{"key":"ocaml","label":"OCaml","variant":"Cohttp"},{"key":"php","label":"PHP","variant":"cURL"},{"key":"php","label":"PHP","variant":"Guzzle"},{"key":"php","label":"PHP","variant":"HTTP_Request2"},{"key":"php","label":"PHP","variant":"pecl_http"},{"key":"powershell","label":"PowerShell","variant":"RestMethod"},{"key":"python","label":"Python","variant":"http.client"},{"key":"python","label":"Python","variant":"Requests"},{"key":"r","label":"R","variant":"httr"},{"key":"r","label":"R","variant":"RCurl"},{"key":"ruby","label":"Ruby","variant":"Net::HTTP"},{"key":"shell","label":"Shell","variant":"Httpie"},{"key":"shell","label":"Shell","variant":"wget"},{"key":"swift","label":"Swift","variant":"URLSession"}],"languageOptions":[{"label":"C# - HttpClient","value":"csharp - HttpClient - C#"},{"label":"C# - RestSharp","value":"csharp - RestSharp - C#"},{"label":"cURL - cURL","value":"curl - cURL - cURL"},{"label":"Dart - http","value":"dart - http - Dart"},{"label":"Go - Native","value":"go - Native - Go"},{"label":"HTTP - HTTP","value":"http - HTTP - HTTP"},{"label":"Java - OkHttp","value":"java - OkHttp - Java"},{"label":"Java - Unirest","value":"java - Unirest - Java"},{"label":"JavaScript - Fetch","value":"javascript - Fetch - JavaScript"},{"label":"JavaScript - jQuery","value":"javascript - jQuery - JavaScript"},{"label":"JavaScript - XHR","value":"javascript - XHR - JavaScript"},{"label":"C - libcurl","value":"c - libcurl - C"},{"label":"NodeJs - Axios","value":"nodejs - Axios - NodeJs"},{"label":"NodeJs - Native","value":"nodejs - Native - NodeJs"},{"label":"NodeJs - Request","value":"nodejs - Request - NodeJs"},{"label":"NodeJs - Unirest","value":"nodejs - Unirest - NodeJs"},{"label":"Objective-C - NSURLSession","value":"objective-c - NSURLSession - Objective-C"},{"label":"OCaml - Cohttp","value":"ocaml - Cohttp - OCaml"},{"label":"PHP - cURL","value":"php - cURL - PHP"},{"label":"PHP - Guzzle","value":"php - Guzzle - PHP"},{"label":"PHP - HTTP_Request2","value":"php - HTTP_Request2 - PHP"},{"label":"PHP - pecl_http","value":"php - pecl_http - PHP"},{"label":"PowerShell - RestMethod","value":"powershell - RestMethod - PowerShell"},{"label":"Python - http.client","value":"python - http.client - Python"},{"label":"Python - Requests","value":"python - Requests - Python"},{"label":"R - httr","value":"r - httr - R"},{"label":"R - RCurl","value":"r - RCurl - R"},{"label":"Ruby - Net::HTTP","value":"ruby - Net::HTTP - Ruby"},{"label":"Shell - Httpie","value":"shell - Httpie - Shell"},{"label":"Shell - wget","value":"shell - wget - Shell"},{"label":"Swift - URLSession","value":"swift - URLSession - Swift"}],"layoutOptions":[{"value":"classic-single-column","label":"Single Column"},{"value":"classic-double-column","label":"Double Column"}],"versionOptions":[],"environmentOptions":[{"value":"0","label":"No Environment"}],"canonicalUrl":"https://documenter.gw.postman.com/view/metadata/TWDUpxxx"}