{"activeVersionTag":"latest","latestAvailableVersionTag":"latest","collection":{"info":{"_postman_id":"dceacf1d-ab06-4cdb-a217-9956d295e6ab","name":"FIWARE Querying Time Series Data (QuantumLeap)","description":"This tutorial is an introduction to\n[FIWARE QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/) - a\ngeneric enabler which is used to persist context data into a **CrateDB**\ndatabase. The tutorial activates the IoT sensors connected in the\n[previous tutorial](https://github.com/Fiware/tutorials.IoT-Agent) and persists\nmeasurements from those sensors into the database. To retrieve time-based\naggregations of such data, users can either use **QuantumLeap** query API or\nconnect directly to the **CrateDB** HTTP endpoint. Results are visualised on a\ngraph or via the **Grafana** time series analytics tool.\n\nThe tutorial uses [cUrl](https://ec.haxx.se/) commands throughout, but is also\navailable as\n[Postman documentation](https://fiware.github.io/tutorials.Time-Series-Data/)\n\nThe `docker-compose` files for this tutorial can be found on GitHub: \n\n![GitHub](https://fiware.github.io/tutorials.Time-Series-Data/icon/GitHub-Mark-32px.png) [FIWARE 304: Time-Series Data](https://github.com/Fiware/tutorials.Time-Series-Data)\n\n# Persisting and Querying Time Series Data (CrateDB)\n\n> \"Forever is composed of nows.\"\n>\n> — Emily Dickinson\n\n[Previous tutorials](https://github.com/Fiware/tutorials.Historic-Context) have\nshown how to persist historic context data into a range of databases such as\n**MySQL** and **PostgreSQL**. Furthermore, the\n[Short Term Historic](https://github.com/Fiware/tutorials.Short-Term-History)\ntutorial has introduced the\n[STH-Comet](https://fiware-sth-comet.readthedocs.io/) generic enabler for\npersisting and querying historic context data using a **MongoDB** database.\n\nFIWARE [QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/) is an\nalternative generic enabler created specifically for data persistence into the\n**CrateDB** time-series database, and therefore offers an alternative to the\n[STH-Comet](https://fiware-sth-comet.readthedocs.io/).\n\n[CrateDB](https://crate.io/) is a distributed SQL DBMS designed for use with the\ninternet of Things. It is capable of ingesting a large number of data points per\nsecond and can be queried in real-time. The database is designed for the\nexecution of complex queries such as geospatial and time series data. Retrieval\nof this historic context data allows for the creation of graphs and dashboards\ndisplaying trends over time.\n\nA summary of the differences can be seen below:\n\n| QuantumLeap                                                                                            | STH-Comet                                                                                          |\n| ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- |\n| Offers an NGSI v2 interface for notifications                                                          | Offers an NGSI v1 interface for notifications                                                      |\n| Persists Data to a CrateDB database                                                                    | Persists Data to MongoDB database                                                                  |\n| Offers its own HTTP endpoint for queries, but you can also query CrateDB                               | Offers its own HTTP endpoint for queries - MongoDB database cannot be accessed directly            |\n| QuantumLeap supports complex data queries (thanks to CrateDB)                                          | STH-Comet offers a limited set of queries                                                          |\n| CrateDB is a distributed and scalable SQL DBMS built atop NoSQL storage                                | MongoDB is a document based NoSQL database                                                         |\n| QuantumLeap's API is docummented in OpenAPI [here](https://app.swaggerhub.com/apis/smartsdk/ngsi-tsdb) | STH-Comet's API is explained in its docs [here](https://fiware-sth-comet.readthedocs.io/en/latest) |\n\nFurther details about the differences between the underlying database engines\ncan be found [here](https://db-engines.com/en/system/CrateDB%3BMongoDB).\n\n## Analyzing time series data\n\nThe appropriate use of time series data analysis will depend on your use case\nand the reliability of the data measurements you receive. Time series data\nanalysis can be used to answer questions such as:\n\n-   What was the maximum measurement of a device within a given time period?\n-   What was the average measurement of a device within a given time period?\n-   What was the sum of the measurements sent by a device within a given time\n    period?\n\nIt can also be used to reduce the significance of each individual data point to\nexclude outliers by smoothing.\n\n#### Grafana\n\n[Grafana](https://grafana.com/) is an open source software for time series\nanalytics tool which will be used during this tutorial. It integrates with a\nvariety of time-series databases including **CrateDB**. It is available licensed\nunder the Apache License 2.0. More information can be found at\nhttps://grafana.com/.\n\n#### Device Monitor\n\nFor the purpose of this tutorial, a series of dummy IoT devices have been\ncreated, which will be attached to the context broker. Details of the\narchitecture and protocol used can be found in the\n[IoT Sensors tutorial](https://github.com/Fiware/tutorials.IoT-Sensors). The\nstate of each device can be seen on the UltraLight device monitor web page found\nat: `http://localhost:3000/device/monitor`\n\n![FIWARE Monitor](https://fiware.github.io/tutorials.Time-Series-Data/img/device-monitor.png)\n\n#### Device History\n\nOnce **QuantumLeap** has started aggregating data, the historical state of each\ndevice can be seen on the device history web page found at:\n`http://localhost:3000/device/history/urn:ngsi-ld:Store:001`\n\n![](https://fiware.github.io/tutorials.Time-Series-Data/img/history-graphs.png)\n\n# Architecture\n\nThis application builds on the components and dummy IoT devices created in\n[previous tutorials](https://github.com/Fiware/tutorials.IoT-Agent/). It will\nuse three FIWARE components: the\n[Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/), the\n[IoT Agent for Ultralight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/),\nand [QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/) .\n\nTherefore the overall architecture will consist of the following elements:\n\n-   The **FIWARE Generic Enablers**:\n\n    -   The FIWARE\n        [Orion Context Broker](https://fiware-orion.readthedocs.io/en/latest/)\n        which will receive requests using\n        [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2)\n    -   The FIWARE\n        [IoT Agent for Ultralight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/)\n        which will receive northbound measurements from the dummy IoT devices in\n        [Ultralight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual)\n        format and convert them to\n        [NGSI](https://fiware.github.io/specifications/OpenAPI/ngsiv2) requests\n        for the context broker to alter the state of the context entities\n    -   FIWARE [QuantumLeap](https://smartsdk.github.io/ngsi-timeseries-api/)\n        subscribed to context changes and persisting them into a **CrateDB**\n        database\n\n-   A [MongoDB](https://www.mongodb.com/) database:\n\n    -   Used by the **Orion Context Broker** to hold context data information\n        such as data entities, subscriptions and registrations\n    -   Used by the **IoT Agent** to hold device information such as device URLs\n        and Keys\n\n-   A [CrateDB](https://crate.io/) database:\n\n    -   Used as a data sink to hold time-based historical context data\n    -   offers an HTTP endpoint to interpret time-based data queries\n\n-   A **Context Provider**: - A webserver acting as set of\n    [dummy IoT devices](https://github.com/Fiware/tutorials.IoT-Sensors) using\n    the\n    [Ultralight 2.0](https://fiware-iotagent-ul.readthedocs.io/en/latest/usermanual/index.html#user-programmers-manual)\n    protocol running over HTTP. - Note the **Stock Management Frontend** and\n    **Context Provider NGSI** proxy are not used in this tutorial.\n\nSince all interactions between the elements are initiated by HTTP requests, the\nentities can be containerized and run from exposed ports.\n\nThe overall architecture can be seen below:\n\n![](https://fiware.github.io/tutorials.Time-Series-Data/img/architecture.png)\n\n# Prerequisites\n\n## Docker and Docker Compose\n\nTo keep things simple all components will be run using\n[Docker](https://www.docker.com). **Docker** is a container technology which\nallows to different components isolated into their respective environments.\n\n-   To install Docker on Windows follow the instructions\n    [here](https://docs.docker.com/docker-for-windows/)\n-   To install Docker on Mac follow the instructions\n    [here](https://docs.docker.com/docker-for-mac/)\n-   To install Docker on Linux follow the instructions\n    [here](https://docs.docker.com/install/)\n\n**Docker Compose** is a tool for defining and running multi-container Docker\napplications. A series of\n[YAML files](https://raw.githubusercontent.com/Fiware/tutorials.Historic-Context/master/cygnus)\nare used to configure the required services for the application. This means all\ncontainer services can be brought up in a single command. Docker Compose is\ninstalled by default as part of Docker for Windows and Docker for Mac, however\nLinux users will need to follow the instructions found\n[here](https://docs.docker.com/compose/install/)\n\nYou can check your current **Docker** and **Docker Compose** versions using the\nfollowing commands:\n\n```console\ndocker-compose -v\ndocker version\n```\n\nPlease ensure that you are using Docker version 18.03 or higher and Docker\nCompose 1.21 or higher and upgrade if necessary.\n\n## Cygwin for Windows\n\nWe will start up our services using a simple Bash script. Windows users should\ndownload [cygwin](http://www.cygwin.com/) to provide a command-line\nfunctionality similar to a Linux distribution on Windows.\n\n# Start Up\n\nBefore you start, you should ensure that you have obtained or built the\nnecessary Docker images locally. Please clone the repository and create the\nnecessary images by running the commands as shown:\n\n```console\ngit clone git@github.com:Fiware/tutorials.Time-Series-Data.git\ncd tutorials.Time-Series-Data\n\n./services create\n```\n\nThereafter, all services can be initialized from the command-line by running the\n[services](https://github.com/Fiware/tutorials.Time-Series-Data/blob/master/services)\nBash script provided within the repository:\n\n```console\n./services start\n```\n\n> :information_source: **Note:** If you want to clean up and start over again\n> you can do so with the following command:\n>\n> ```console\n> ./services stop\n> ```\n\n# Connecting FIWARE to a CrateDB Database via QuantumLeap\n\nIn the configuration, **QuantumLeap** listens to NGSI v2 notifications on port\n`8868` and persists historic context data to the **CrateDB**. **CrateDB** is\naccessible using port `4200` and can either be queried directly or attached to\nthe Grafana analytics tool. The rest of the system providing the context data\nhas been described in previous tutorials\n\n## CrateDB Database Server Configuration\n\n```yaml\ncratedb:\n    image: crate:2.3\n    hostname: cratedb\n    ports:\n        - \"4200:4200\"\n        - \"4300:4300\"\n    command:\n        -Ccluster.name=democluster -Chttp.cors.enabled=true\n        -Chttp.cors.allow-origin=\"*\"\n```\n\n## QuantumLeap Configuration\n\n```yaml\nquantumleap:\n    image: smartsdk/quantumleap\n    hostname: quantumleap\n    ports:\n        - \"8668:8668\"\n    depends_on:\n        - cratedb\n    environment:\n        - CRATE_HOST=cratedb\n```\n\n## Grafana Configuration\n\n```yaml\ngrafana:\n    image: grafana/grafana\n    depends_on:\n        - cratedb\n    ports:\n        - \"3003:3000\"\n    environment:\n        - GF_INSTALL_PLUGINS=crate-datasource,grafana-clock-panel,grafana-worldmap-panel\n```\n\nThe `quantumleap` container is listening on one port:\n\n-   The Operations for port for QuantumLeap - `8668` is where the service will\n    be listening for notifications from the Orion context broker and where users\n    can query data from.\n\nThe `CRATE_HOST` environment variable defines the location where the data will\nbe persisted.\n\nThe `cratedb` container is listening on two ports:\n\n-   The Admin UI is available on port `4200`\n-   The transport protocol is available on `port 4300`\n\nThe `grafana` container has connected up port `3000` internally with port `3003`\nexternally. This is because the Grafana UI is usually available on port `3000`,\nbut this port has already been taken by the dummy devices UI so it has been\nshifted to another port. The Grafana Environment variables are described within\ntheir own [documentation](http://docs.grafana.org/installation/configuration/).\nThe configuration ensures we will be able to connect to the **CrateDB** database\nlater on in the tutorial\n\n### Generating Context Data\n\nFor the purpose of this tutorial, we must be monitoring a system where the\ncontext is periodically being updated. The dummy IoT Sensors can be used to do\nthis. Open the device monitor page at `http://localhost:3000/device/monitor` and\nunlock a **Smart Door** and switch on a **Smart Lamp**. This can be done by\nselecting an appropriate command from the drop down list and pressing the `send`\nbutton. The stream of measurements coming from the devices can then be seen on\nthe same page:\n\n![](https://fiware.github.io/tutorials.IoT-Sensors/img/door-open.gif)\n","schema":"https://schema.getpostman.com/json/collection/v2.0.0/collection.json","isPublicCollection":true,"owner":"513743","team":157450,"collectionId":"dceacf1d-ab06-4cdb-a217-9956d295e6ab","publishedId":"RWEnkvDc","public":true,"publicUrl":"https://documenter-api.postman.tech/view/513743/RWEnkvDc","privateUrl":"https://go.postman.co/documentation/513743-dceacf1d-ab06-4cdb-a217-9956d295e6ab","customColor":{"top-bar":"FFFFFF","right-sidebar":"303030","highlight":"233C68"},"documentationLayout":"classic-double-column","version":"8.10.1","publishDate":"2020-01-02T11:08:15.000Z","activeVersionTag":"latest","documentationTheme":"light","metaTags":{},"logos":{}},"statusCode":200},"environments":[],"user":{"authenticated":false,"permissions":{"publish":false}},"run":{"button":{"js":"https://run.pstmn.io/button.js","css":"https://run.pstmn.io/button.css"}},"web":"https://www.getpostman.com/","team":{"logo":"https://res.cloudinary.com/postman/image/upload/t_team_logo_pubdoc/v1/team/d7085d490b9144732c65203aa6e3b68b31884d1c33a86b8a00d15da75147ae33","favicon":""},"isEnvFetchError":false,"languages":"[{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"HttpClient\"},{\"key\":\"csharp\",\"label\":\"C#\",\"variant\":\"RestSharp\"},{\"key\":\"curl\",\"label\":\"cURL\",\"variant\":\"cURL\"},{\"key\":\"dart\",\"label\":\"Dart\",\"variant\":\"http\"},{\"key\":\"go\",\"label\":\"Go\",\"variant\":\"Native\"},{\"key\":\"http\",\"label\":\"HTTP\",\"variant\":\"HTTP\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"OkHttp\"},{\"key\":\"java\",\"label\":\"Java\",\"variant\":\"Unirest\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"Fetch\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"jQuery\"},{\"key\":\"javascript\",\"label\":\"JavaScript\",\"variant\":\"XHR\"},{\"key\":\"c\",\"label\":\"C\",\"variant\":\"libcurl\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Axios\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Native\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Request\"},{\"key\":\"nodejs\",\"label\":\"NodeJs\",\"variant\":\"Unirest\"},{\"key\":\"objective-c\",\"label\":\"Objective-C\",\"variant\":\"NSURLSession\"},{\"key\":\"ocaml\",\"label\":\"OCaml\",\"variant\":\"Cohttp\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"cURL\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"Guzzle\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"HTTP_Request2\"},{\"key\":\"php\",\"label\":\"PHP\",\"variant\":\"pecl_http\"},{\"key\":\"powershell\",\"label\":\"PowerShell\",\"variant\":\"RestMethod\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"http.client\"},{\"key\":\"python\",\"label\":\"Python\",\"variant\":\"Requests\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"httr\"},{\"key\":\"r\",\"label\":\"R\",\"variant\":\"RCurl\"},{\"key\":\"ruby\",\"label\":\"Ruby\",\"variant\":\"Net::HTTP\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"Httpie\"},{\"key\":\"shell\",\"label\":\"Shell\",\"variant\":\"wget\"},{\"key\":\"swift\",\"label\":\"Swift\",\"variant\":\"URLSession\"}]","languageSettings":[{"key":"csharp","label":"C#","variant":"HttpClient"},{"key":"csharp","label":"C#","variant":"RestSharp"},{"key":"curl","label":"cURL","variant":"cURL"},{"key":"dart","label":"Dart","variant":"http"},{"key":"go","label":"Go","variant":"Native"},{"key":"http","label":"HTTP","variant":"HTTP"},{"key":"java","label":"Java","variant":"OkHttp"},{"key":"java","label":"Java","variant":"Unirest"},{"key":"javascript","label":"JavaScript","variant":"Fetch"},{"key":"javascript","label":"JavaScript","variant":"jQuery"},{"key":"javascript","label":"JavaScript","variant":"XHR"},{"key":"c","label":"C","variant":"libcurl"},{"key":"nodejs","label":"NodeJs","variant":"Axios"},{"key":"nodejs","label":"NodeJs","variant":"Native"},{"key":"nodejs","label":"NodeJs","variant":"Request"},{"key":"nodejs","label":"NodeJs","variant":"Unirest"},{"key":"objective-c","label":"Objective-C","variant":"NSURLSession"},{"key":"ocaml","label":"OCaml","variant":"Cohttp"},{"key":"php","label":"PHP","variant":"cURL"},{"key":"php","label":"PHP","variant":"Guzzle"},{"key":"php","label":"PHP","variant":"HTTP_Request2"},{"key":"php","label":"PHP","variant":"pecl_http"},{"key":"powershell","label":"PowerShell","variant":"RestMethod"},{"key":"python","label":"Python","variant":"http.client"},{"key":"python","label":"Python","variant":"Requests"},{"key":"r","label":"R","variant":"httr"},{"key":"r","label":"R","variant":"RCurl"},{"key":"ruby","label":"Ruby","variant":"Net::HTTP"},{"key":"shell","label":"Shell","variant":"Httpie"},{"key":"shell","label":"Shell","variant":"wget"},{"key":"swift","label":"Swift","variant":"URLSession"}],"languageOptions":[{"label":"C# - HttpClient","value":"csharp - HttpClient - C#"},{"label":"C# - RestSharp","value":"csharp - RestSharp - C#"},{"label":"cURL - cURL","value":"curl - cURL - cURL"},{"label":"Dart - http","value":"dart - http - Dart"},{"label":"Go - Native","value":"go - Native - Go"},{"label":"HTTP - HTTP","value":"http - HTTP - HTTP"},{"label":"Java - OkHttp","value":"java - OkHttp - Java"},{"label":"Java - Unirest","value":"java - Unirest - Java"},{"label":"JavaScript - Fetch","value":"javascript - Fetch - JavaScript"},{"label":"JavaScript - jQuery","value":"javascript - jQuery - JavaScript"},{"label":"JavaScript - XHR","value":"javascript - XHR - JavaScript"},{"label":"C - libcurl","value":"c - libcurl - C"},{"label":"NodeJs - Axios","value":"nodejs - Axios - NodeJs"},{"label":"NodeJs - Native","value":"nodejs - Native - NodeJs"},{"label":"NodeJs - Request","value":"nodejs - Request - NodeJs"},{"label":"NodeJs - Unirest","value":"nodejs - Unirest - NodeJs"},{"label":"Objective-C - NSURLSession","value":"objective-c - NSURLSession - Objective-C"},{"label":"OCaml - Cohttp","value":"ocaml - Cohttp - OCaml"},{"label":"PHP - cURL","value":"php - cURL - PHP"},{"label":"PHP - Guzzle","value":"php - Guzzle - PHP"},{"label":"PHP - HTTP_Request2","value":"php - HTTP_Request2 - PHP"},{"label":"PHP - pecl_http","value":"php - pecl_http - PHP"},{"label":"PowerShell - RestMethod","value":"powershell - RestMethod - PowerShell"},{"label":"Python - http.client","value":"python - http.client - Python"},{"label":"Python - Requests","value":"python - Requests - Python"},{"label":"R - httr","value":"r - httr - R"},{"label":"R - RCurl","value":"r - RCurl - R"},{"label":"Ruby - Net::HTTP","value":"ruby - Net::HTTP - Ruby"},{"label":"Shell - Httpie","value":"shell - Httpie - Shell"},{"label":"Shell - wget","value":"shell - wget - Shell"},{"label":"Swift - URLSession","value":"swift - URLSession - Swift"}],"layoutOptions":[{"value":"classic-single-column","label":"Single Column"},{"value":"classic-double-column","label":"Double Column"}],"versionOptions":[],"environmentOptions":[{"value":"0","label":"No Environment"}],"canonicalUrl":"https://documenter.gw.postman.com/view/metadata/RWEnkvDc"}