![\"GitHub\"](\"https://fiware.github.io/tutorials.Time-Series-Data/icon/GitHub-Mark-32px.png\"){kind=icon}
FIWARE 301: Persisting Context DataThis tutorial is an introduction to FIWARE QuantumLeap - a generic\nenabler which is used to persist context data into a CrateDB database. The tutorial activates the IoT sensors\nconnected in the previous tutorial and persists measurements from those\nsensors into the database. To retrieve time-based aggregations of such data, users can either use QuantumLeap query\nAPI or connect directly to the CrateDB HTTP endpoint. Results are visualised on a graph or via the Grafana time\nseries analytics tool.
\nThe docker-compose files for this tutorial can be found on GitHub:
FIWARE 301: Persisting Context Data
\n\n\"Forever is composed of nows.\"
\n— Emily Dickinson
\n
FIWARE QuantumLeap is an time-based data-persistence generic enabler\ncreated specifically to persist and query time-series database (currently CrateDB and TimescaleDB). The component can\nrespond to NGSI-v2 or NGSI-LD subscriptions.
\nCrateDB is a distributed SQL DBMS designed for use with the internet of Things. It is capable of\ningesting a large number of data points per second and can be queried in real-time. The database is designed for the\nexecution of complex queries such as geospatial and time series data. Retrieval of this historic context data allows for\nthe creation of graphs and dashboards displaying trends over time.
\nTimescaleDB scales PostgreSQL for time-series data via automatic partitioning across time\nand space (partitioning key), yet retains the standard PostgreSQL interface. In other words, TimescaleDB exposes what\nlook like regular tables, but are actually only an abstraction (or a virtual view) of many individual tables comprising\nthe actual data. In combination with PostGIS extension can support geo-timeseries.
\nThe appropriate use of time series data analysis will depend on your use case and the reliability of the data\nmeasurements you receive. Time series data analysis can be used to answer questions such as:
\nIt can also be used to reduce the significance of each individual data point to exclude outliers by smoothing.
\nGrafana is an open source software for time series analytics tool which will be used during this\ntutorial. It integrates with a variety of time-series databases including CrateDB and TimescaleDB. It is\navailable licensed under the Apache License 2.0. More information can be found at https://grafana.com/.
For the purpose of this tutorial, a series of dummy agricultural IoT devices have been created, which will be attached\nto the context broker. Details of the architecture and protocol used can be found in the\nIoT Sensors tutorial The state of each device can be\nseen on the UltraLight device monitor web page found at: http://localhost:3000/device/monitor
![\"FIWARE](\"https://fiware.github.io/tutorials.Time-Series-Data/img/farm-devices.png\")
Once QuantumLeap has started aggregating data, the historical state of each device can be seen on the device history\nweb page found at: http://localhost:3000/device/history/urn:ngsi-ld:Farm:001
![\"\"](\"https://fiware.github.io/tutorials.Time-Series-Data/img/history-graphs.png\")
This application builds on the components and dummy IoT devices created in\nprevious tutorials. It will use three FIWARE components: the\nOrion Context Broker, the\nIoT Agent for Ultralight 2.0, and\nQuantumLeap .
\nTherefore the overall architecture will consist of the following elements:
\nThe FIWARE Generic Enablers:
\nA MongoDB database:
\nA CrateDB database:
\nThe Tutorial Application does the following:
\n@context files defining the context entities within the system.Since all interactions between the elements are initiated by HTTP requests, the entities can be containerized and run\nfrom exposed ports.
\nThe overall architecture can be seen below:
\n![\"\"](\"https://fiware.github.io/tutorials.Time-Series-Data/img/architecture.png\")
To keep things simple all components will be run using Docker. Docker is a container\ntechnology which allows to different components isolated into their respective environments.
\nDocker Compose is a tool for defining and running multi-container Docker applications. A series of\nYAML files are used to\nconfigure the required services for the application. This means all container services can be brought up in a single\ncommand. Docker Compose is installed by default as part of Docker for Windows and Docker for Mac, however Linux users\nwill need to follow the instructions found here
\nYou can check your current Docker and Docker Compose versions using the following commands:
\ndocker-compose -v\ndocker version\nPlease ensure that you are using Docker version 18.03 or higher and Docker Compose 1.21 or higher and upgrade if\nnecessary.
\nWe will start up our services using a simple Bash script. Windows users should download cygwin\nto provide a command-line functionality similar to a Linux distribution on Windows.
\nBefore you start, you should ensure that you have obtained or built the necessary Docker images locally. Please clone\nthe repository and create the necessary images by running the commands as shown:
\ngit clone https://github.com/FIWARE/tutorials.Time-Series-Data.git\ncd tutorials.Time-Series-Data\ngit checkout NGSI-LD\n\n./services create\nThereafter, all services can be initialized from the command-line by running the\nservices Bash script provided within the\nrepository:
\n./services start\n\n\n:information_source: Note: If you want to clean up and start over again you can do so with the following command:
\n\n./services stop\n
In the configuration, QuantumLeap listens to NGSI LD notifications on port 8868 and persists historic context data\nto the CrateDB. CrateDB is accessible using port 4200 and can either be queried directly or attached to the\nGrafana analytics tool. The rest of the system providing the context data has been described in previous tutorials
crate-db:\n image: crate:4.1.4\n hostname: crate-db\n ports:\n - \"4200:4200\"\n - \"4300:4300\"\n command:\n crate -Clicense.enterprise=false -Cauth.host_based.enabled=false -Ccluster.name=democluster\n -Chttp.cors.enabled=true -Chttp.cors.allow-origin=\"*\"\n environment:\n - CRATE_HEAP_SIZE=2g\nIf CrateDB exits immediately with a\nmax virtual memory areas vm.max_map_filling [65530] is too low, increase to at least [262144] error, this can be fixed\nby running the sudo sysctl -w vm.max_map_filling=262144 command on the host machine. For further information look\nwithin the CrateDB\ndocumentation and Docker\ntroubleshooting guide
quantumleap:\n image: smartsdk/quantumleap\n hostname: quantumleap\n ports:\n - \"8668:8668\"\n depends_on:\n - crate-db\n environment:\n - CRATE_HOST=crate-db\ngrafana:\n image: grafana/grafana\n depends_on:\n - cratedb\n ports:\n - \"3003:3000\"\n environment:\n - GF_INSTALL_PLUGINS=https://github.com/orchestracities/grafana-map-plugin/archive/master.zip;grafana-map-plugin,grafana-clock-panel,grafana-worldmap-panel\nThe quantumleap container is listening on one port:
8668 is where the service will be listening for notifications from the\nOrion context broker and where users can query data from.The CRATE_HOST environment variable defines the location where the data will be persisted.
The cratedb container is listening on two ports:
4200port 4300The grafana container has connected up port 3000 internally with port 3003 externally. This is because the Grafana\nUI is usually available on port 3000, but this port has already been taken by the dummy devices UI so it has been\nshifted to another port. The Grafana Environment variables are described within their own\ndocumentation. The configuration ensures we will be able to\nconnect to the CrateDB database later on in the tutorial. The configuration also imports a custom map plugin that\nhelps you in displaying NGSI v2 entities over a map.
For the purpose of this tutorial, we must be monitoring a system where the context is periodically being updated. The\ndummy IoT Sensors can be used to do this.
\nDetails of various buildings around the farm can be found in the tutorial application. Open\nhttp://localhost:3000/app/farm/urn:ngsi-ld:Building:farm001 to display a building with an associated filling sensor\nand thermostat.
![\"\"](\"https://fiware.github.io/tutorials.Subscriptions/img/fmis.png\")
Remove some hay from the barn, update the thermostat and open the device monitor page at\nhttp://localhost:3000/device/monitor and start a Tractor and switch on a Smart Lamp. This can be done by\nselecting an appropriate command from the drop down list and pressing the send button. The stream of measurements\ncoming from the devices can then be seen on the same page.
The rate of change of the Filling Sensor is driven by events in the real-world. We need to receive every event to be\nable to aggregate the results.
\nThis is done by making a POST request to the /ngsi-ld/v1/subscriptions/ endpoint of the Orion-LD Context Broker.
NGSILD-Tenant headers is used to filter the subscription to only listen to measurements from the attached IoT\nSensorsentities type in the request body ensures that QuantumLeap will be informed of all FillingSensor\ndata changes.notification URL must match the exposed port.With NGSI-LD the observedAt property-of-property holds the timestamp of the measure. Because the attribute being\nmonitored contains this property-of-property, the time_index column within the CrateDB database will match the\ndata found within the MongoDB database used by the Orion Context Broker rather than using the creation time of\nthe record within the CrateDB itself.
The heart rate and GPS reading of the Animal Collars on the animals on the farm are constantly changing, we only need to\nsample the values to be able to work out relevant statistics such as minimum and maximum values and rates of change.
\nThis is done by making a POST request to the /ngsi-ld/v1/subscriptions/ endpoint of the Orion Context Broker and\nincluding the throttling attribute in the request body.
NGSILD-Tenant headers is used to filter the subscription to only listen to measurements from the attached IoT\nSensorsentities type in the request body ensures that QuantumLeap will be informed of all Device data\nchanges.notification URL must match the exposed port.throttling value defines the rate that changes are sampled.Before anything, check the subscriptions you created in steps one and two are working (i.e., at least one\nnotification for each was sent).
\n","urlObject":{"protocol":"http","path":["ngsi-ld","v1","subscriptions",""],"host":["localhost:1026"],"query":[],"variable":[]}},"response":[],"_postman_id":"f6725b7c-5ea3-437b-aef2-73db48bb0560"}],"id":"ab92a669-5e8d-47f8-a00f-3c6db86b01c9","description":"Once a dynamic context system is up and running, we need to inform Quantum\nLeap directly of changes in context. As expected this is done using the\nsubscription mechanism of the Orion Context Broker. The attrsFormat=legacy\nattribute is not required since QuantumLeap accepts NGSI v2 notifications\ndirectly.
Subscriptions will be covered in the next subsections. More details about\nsubscriptions can be found in previous tutorials or in the\nsubscriptions section\nof QuantumLeap docs.
\n","event":[{"listen":"prerequest","script":{"id":"4f9ade89-f6d9-47bb-b212-13bf4fb6c606","type":"text/javascript","exec":[""]}},{"listen":"test","script":{"id":"8957dc7f-7d70-4126-af3d-6d39c1a9744a","type":"text/javascript","exec":[""]}}],"_postman_id":"ab92a669-5e8d-47f8-a00f-3c6db86b01c9"},{"name":"Time Series Data Queries (QuantumLeap API)","item":[{"name":"QuantumLeap - Obtain version information","id":"b1152caa-fdaa-4390-88f7-c118e9c158cd","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"GET","header":[],"url":"http://localhost:8668/version","urlObject":{"protocol":"http","path":["version"],"host":["localhost:8668"],"query":[],"variable":[]}},"response":[],"_postman_id":"b1152caa-fdaa-4390-88f7-c118e9c158cd"},{"name":"List the first N Sampled Values","id":"326f8199-6dbd-4963-a8a5-5d1fe153be56","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"GET","header":[{"key":"Accept","value":"application/json"},{"key":"Fiware-Service","value":"openiot","type":"text"}],"body":{"mode":"raw","raw":""},"url":"http://localhost:8668/v2/entities/urn:ngsi-ld:FillingLevelSensor:filling001/attrs/filling?limit=3","description":"This example shows the first 3 sampled luminosity values from Lamp:001.
Note the use of Fiware-Service and Fiware-ServicePath headers. These are\nrequired only when data are pushed to orion using such headers (in multitenancy\nscenarios). Failing to align these headers will result in no data being\nreturned.
This example shows the fourth, fifth and sixth sampled count values from Motion:001.
\n","urlObject":{"protocol":"http","path":["v2","entities","urn:ngsi-ld:FillingLevelSensor:filling001","attrs","filling"],"host":["localhost:8668"],"query":[{"key":"limit","value":"6"},{"key":"offset","value":"3"}],"variable":[]}},"response":[],"_postman_id":"8087621b-9124-49df-9822-935379d4c637"},{"name":"List the latest N Sampled Values","id":"d142d0d8-2401-4111-bc4f-e0351497f06d","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"GET","header":[{"key":"Accept","value":"application/json"},{"key":"Fiware-Service","value":"openiot","type":"text"}],"body":{"mode":"raw","raw":""},"url":"http://localhost:8668/v2/entities/urn:ngsi-ld:Device:filling001/attrs/filling?lastN=3","description":"This example shows latest three sampled count values from Motion:001.
This example shows total count values from Motion:001 over each minute.
You need QuantumLeap version >= 0.4.1. You can check your version with a\nsimple GET like:
\ncurl -X GET \\\n 'http://localhost:8668/v2/version' \\\n -H 'Accept: application/json'\nThis example shows minimum luminosity values from Lamp:001 over each minute.
You need QuantumLeap version >= 0.4.1. You can check your version with a\nsimple GET like:
\ncurl -X GET \\\n 'http://localhost:8668/v2/version' \\\n -H 'Accept: application/json'\nThis example shows maximum luminosity value of Lamp:001 that occurred\nbetween from 2018-06-27T09:00:00 to 2018-06-30T23:59:59.
Alter the fromDate and toDate parameters to retrieve your current data
This example shows the latest four sampled luminosity values of lamps that\nare within a 5 km radius from 52°33'16.9\"N 13°23'55.0\"E (Bornholmer Straße\n65, Berlin, Germany). If you have turned on all the lamps available on the\ndevice monitor page, you should be able to see data for Lamp:001 and\nLamp:004.
\n\n","urlObject":{"protocol":"http","path":["","v2","types","Device","attrs","heartRate"],"host":["localhost:8668"],"query":[{"key":"lastN","value":"4"},{"key":"georel","value":"near;maxDistance:5000"},{"key":"geometry","value":"point"},{"key":"coords","value":"52.518,13.357"}],"variable":[]}},"response":[],"_postman_id":"38aeb194-de13-42bb-a32d-e5aec3527617"},{"name":"List the latest N Sampled Values of Devices in an Area","id":"3130098c-4190-4c60-b274-3878b927a214","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"GET","header":[{"key":"Content-Type","value":"application/json"},{"key":"Fiware-Service","type":"text","value":"openiot"}],"body":{"mode":"raw","raw":""},"url":"http://localhost:8668/v2/types/Device/attrs/heartRate?lastN=4&georel=coveredBy&geometry=polygon&coords=52.5537,13.3996;52.5557,13.3996;52.5557,13.3976;52.5537,13.3976;52.5537,13.3996","description":"Note: Geographical queries are only available\nstarting from version
\n0.5of QuantumLeap which implements the full\nset of queries detailed in the Geographical Queries section of the\nNGSI v2 specification.
This example shows the latest four sampled luminosity values of lamps that\nare inside a square of side 200 m centred at 52°33'16.9\"N 13°23'55.0\"E\n(Bornholmer Straße 65, Berlin, Germany). Even if you have turned on all the\nlamps available on the device monitor page, you should only see data for\nLamp:001.
\n\n","urlObject":{"protocol":"http","path":["v2","types","Device","attrs","heartRate"],"host":["localhost:8668"],"query":[{"key":"lastN","value":"4"},{"key":"georel","value":"coveredBy"},{"key":"geometry","value":"polygon"},{"key":"coords","value":"52.5537,13.3996;52.5557,13.3996;52.5557,13.3976;52.5537,13.3976;52.5537,13.3996"}],"variable":[]}},"response":[],"_postman_id":"3130098c-4190-4c60-b274-3878b927a214"}],"id":"fd447cb1-1962-4079-bda7-1f631d9eb781","description":"Note: Geographical queries are only available\nstarting from version
\n0.5of QuantumLeap which implements the full\nset of queries detailed in the Geographical Queries section of the\nNGSI v2 specification.
CrateDB offers an\nHTTP Endpoint\nthat can be used to submit SQL queries. The endpoint is accessible under\n<servername:port>/_sql.
SQL statements are sent as the body of POST requests in JSON format, where the\nSQL statement is the value of the stmt attribute.
\n\nWhen to query CrateDB and when QuantumLeap?. As a rule of thumb,\nprefer working always with QuantumLeap for the following reasons:
\n\n
\n- Your experience will be closer to FIWARE NGSI APIs like Orion's.
\n- Your application will not be tied to CrateDB's specifics nor QuantumLeap's\nimplementation details, which could change and break your app.
\n- QuantumLeap can be easily extended to other backends and your app will get\ncompatibility for free.
\n- If your deployment is distributed, you won't need to expose the ports of\nyour database to the outside.
\n
If your are sure your query is not supported by QuantumLeap, you may have to\nend up querying CrateDB, however, please open an issue in\nQuantumLeap's repo so\nthe team is aware.
\n","event":[{"listen":"prerequest","script":{"id":"019f1ccb-6a46-4caa-adf6-fc3c4236f789","type":"text/javascript","exec":[""]}},{"listen":"test","script":{"id":"ba05f53d-adef-4a40-a2d7-cd4a32edc172","type":"text/javascript","exec":[""]}}],"_postman_id":"fd447cb1-1962-4079-bda7-1f631d9eb781"},{"name":"Time Series Data Queries (Crate-DB)","item":[{"name":"Read Schemas","id":"4900121c-04c0-43fe-96a9-fd3815c35044","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"POST","header":[{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\"stmt\":\"SHOW SCHEMAS\"}"},"url":"http://localhost:4200/_sql","description":"Another way to see if data are being persisted is to check if a table_schema\nhas been created. This can be done by making a request to the CrateDB HTTP\nendpoint as shown below.
Schema names are formed with the mt prefix followed by fiware-service header\nin lower case. The IoT Agent is forwarding measurements from the dummy IoT\ndevices, with the FIWARE-Service header openiot. These are being persisted\nunder the mtopeniot schema.
If the mtopeniot does not exist, then the subscription to QuantumLeap has\nnot been set up correctly. Check that the subscription exists, and has been\nconfigured to send data to the correct location.
Quantum Leap will persist data into separate tables within the Crate-DB database based on the entity type. Table names are formed with the et prefix and the entity type name in lowercase.
The response shows that both Motion Sensor data and Smart Lamp data are being persisted in the database.
\n","urlObject":{"protocol":"http","path":["_sql"],"host":["localhost:4200"],"query":[],"variable":[]}},"response":[],"_postman_id":"9e55e4b4-a44f-4726-9f18-17f085a32b99"},{"name":"List the first N Sampled Values","id":"9b695415-8755-4fdb-853b-d72a04febc7a","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"POST","header":[{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\"stmt\":\"SELECT * FROM mtopeniot.etfillinglevelsensor WHERE entity_id = 'urn:ngsi-ld:Device:filling001' ORDER BY time_index ASC LIMIT 3\"}"},"url":"http://localhost:4200/_sql","description":"This example shows the first 3 sampled luminosity values from Lamp:001.
The SQL statement uses ORDER BY and LIMIT clauses to sort the data. More details can be found under within the Crate-DB documentation
This example shows the fourth, fifth and sixth sampled count values from Motion:001.
\nThe SQL statement uses an OFFSET clause to retrieve the required rows. \nMore details can be found under within the Crate-DB documentation
This example shows latest three sampled count values from Motion:001.
The SQL statement uses an ORDER BY ... DESC clause combined with a LIMIT clause to retrieve \nthe last N rows. More details can be found under within the Crate-DB documentation
This example shows total count values from Motion:001 over each minute.
The SQL statement uses a SUM function and GROUP BY clause to retrieve the relevant data.\nCrate-DB offers a range of Date-Time Function to truncate and convert the timestamps into data which can be grouped.
This example shows minimum luminosity values from Lamp:001 over each minute.
The SQL statement uses a MIN function and GROUP BY clause to retrieve the relevant data.\nCrate-DB offers a range of Date-Time Function to truncate and convert the timestamps into data which can be grouped.
This example shows maximum luminosity values from Lamp:001 over each minute.
The SQL statement uses a MAX function and GROUP BY clause to retrieve the relevant data.\nCrate-DB offers a range of Date-Time Function to truncate and convert the timestamps into data which can be grouped.
This example shows the average of luminosity values from Lamp:001 over each minute.
The SQL statement uses a AVG function and GROUP BY clause to retrieve the relevant data.\nCrate-DB offers a range of Date-Time Function to truncate and convert the timestamps into data which can be grouped.
Crate-DB offers an HTTP Endpoint that can be used to submit SQL queries. The endpoint is accessible under <servername:port>/_sql.
SQL statements are sent as the body of POST requests in JSON format, where the SQL statement is the value of the stmt attribute.