![\"FIWARE](\"https://fiware.github.io/tutorials.IoT-over-MQTT/img/device-monitor.png\")
This tutorial uses introduces the use of the MQTT protocol across IoT devices connecting to FIWARE. The UltraLight 2.0 IoT Agent\ncreated in the previous tutorial is reconfigured to\ncommunicate with a set of dummy IoT devices using MQTT
\nMQTT is publish-subscribe-based messaging protocol used in the Internet of Things. It works on top of the TCP/IP\nprotocol, and is designed for connections with remote locations where a \"small code footprint\" is required or the\nnetwork bandwidth is limited. The goal is to provide a protocol, which is bandwidth-efficient and uses little\nbattery power.
\nThe previous tutorial used HTTP as its transport mechanism between\nthe devices and the IoT Agent. HTTP uses a request/response paradigm where each device connects directly to the IoT Agent.\nMQTT is different in that publish-subscribe is event-driven and pushes messages to clients. It requires an additional\ncentral communication point (known as the MQTT broker) which it is in charge of dispatching all messages between the senders and the rightful receivers. Each client that publishes a message to the broker, includes a topic into the message. The topic is the routing information for the broker. Each client that wants to receive messages subscribes to a certain topic and the broker delivers all messages with the matching topic to the client. Therefore the clients don’t have to know each other, they only communicate over the topic. This architecture enables highly scalable solutions without dependencies between the data producers and the data consumers.
\nThe UltraLight 2.0 IoT Agent will only send or interpret messages using the UltraLight 2.0 syntax,\nhowever it can be used to send and receive messages over multiple transport mechanisms. Therefore we\nare able to use the same FIWARE generic enabler to connect to a wider range of IoT devices.
\nMosquitto is a readily available, open source MQTT broker which will be used during this tutorial.\nIt is available licensed under EPL/EDL. More information can be found at https://mosquitto.org/
\nFor the purpose of this tutorial, a series of dummy IoT devices have been created, which will be attached to the context broker. Details of the architecture and protocol used can be found in the IoT Sensors tutorial\nThe state of each device can be seen on the UltraLight device monitor web-page found at: http://localhost:3000/device/monitor
This application builds on the components created in previous tutorials. It\nwill make use of two FIWARE components - the Orion Context Broker and the IoT Agent for UltraLight 2.0. Usage of the Orion Context Broker is sufficient for an application to qualify as “Powered by FIWARE”.\nBoth the Orion Context Broker and the IoT Agent rely on open source MongoDB technology to keep persistence of the information they hold. We will also be using the dummy IoT devices created in the previous tutorial \nAdditionally will add an instance of the Mosquitto MQTT broker which is open source and available under the EPL/EDL.
\nTherefore the overall architecture will consist of the following elements:
\nSince all interactions between the elements are initiated by HTTP or MQTT requests over TCP, the entities can be containerized and run from exposed ports.
\n![\"\"](\"https://fiware.github.io/tutorials.IoT-over-MQTT/img/architecture.png\")
The necessary configuration information for wiring up the Mosquitto MQTT Broker, the IoT devices and the IoT Agent can be seen in the services section of the associated docker-compose.yml file:
mosquitto:\n image: eclipse-mosquitto\n hostname: mosquitto\n container_name: mosquitto\n networks:\n - default\n expose:\n - \"1883\"\n - \"9001\"\n ports:\n - \"1883:1883\"\n - \"9001:9001\"\n volumes:\n - ./mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf\n \nThe mosquitto container is listening on two ports:
3000 is exposed so we can post MQTT topics9001 is the standard port for HTTP/Websocket communicationsThe attached volume is a configuration file used to increase the debug level of the MQTT Message Broker.
\n context-provider:\n image: fiware/cp-web-app:latest\n hostname: context-provider\n container_name: context-provider\n networks:\n - default\n expose:\n - \"3000\"\n - \"3001\"\n ports:\n - \"3000:3000\"\n - \"3001:3001\"\n environment:\n - \"DEBUG=proxy:*\"\n - \"PORT=3000\"\n - \"DUMMY_DEVICES_PORT=3001\"\n - \"DUMMY_DEVICES_API_KEY=4jggokgpepnvsb2uv4s40d59ov\"\n - \"DUMMY_DEVICES_TRANSPORT=MQTT\"\nThe context-provider container is listening on two ports:
3000 is exposed so we can see the web-page displaying the Dummy IoT devices.3001 is exposed purely for tutorial access - so that cUrl or Postman can make UltraLight commands\nwithout being part of the same network.The context-provider container is driven by environment variables as shown:
The other context-provider container configuration values described in the YAML file are not used in this tutorial.
The IoT Agent for UltraLight 2.0 can be instantiated within a Docker container. An official Docker image is available from Docker Hub tagged fiware/iotagent-ul. The \nnecessary configuration can be seen below:
iot-agent:\n image: fiware/iotagent-ul:latest\n hostname: iot-agent\n container_name: fiware-iot-agent\n depends_on:\n - mongo-db\n networks:\n - default\n expose:\n - \"4041\"\n - \"7896\"\n ports:\n - \"4041:4041\"\n - \"7896:7896\"\n environment:\n - \"IOTA_CB_HOST=orion\"\n - \"IOTA_CB_PORT=1026\"\n - \"IOTA_NORTH_PORT=4041\"\n - \"IOTA_REGISTRY_TYPE=mongodb\"\n - \"IOTA_LOG_LEVEL=DEBUG\"\n - \"IOTA_TIMESTAMP=true\"\n - \"IOTA_MONGO_HOST=mongo-db\"\n - \"IOTA_MONGO_PORT=27017\"\n - \"IOTA_MONGO_DB=iotagentul\"\n - \"IOTA_PROVIDER_URL=http://iot-agent:4041\"\n - \"IOTA_MQTT_HOST=mosquitto\"\n - \"IOTA_MQTT_PORT=1883\"\nThe iot-agent container relies on the precence of the Orion Context Broker and uses a MongoDB database to hold device information such as device URLs and Keys. The container is listening on two ports:
7896 is exposed to receive Ultralight measurements over HTTP from the Dummy IoT devices4041 is exposed purely for tutorial access - so that cUrl or Postman can make provisioning commands\nwithout being part of the same network.The iot-agent container is driven by environment variables as shown:
As you can see, use of the MQTT transport is driven by only two environment variables IOTA_MQTT_HOST and IOTA_MQTT_PORT
To keep things simple all components will be run using Docker. Docker is a container technology which allows to different components isolated into their respective environments.
\nDocker Compose is a tool for defining and running multi-container Docker applications. A \nYAML file is used\nconfigure the required services for the application. This means all container services can be brought up in a single \ncommand. Docker Compose is installed by default as part of Docker for Windows and Docker for Mac, however Linux users \nwill need to follow the instructions found here
\nWe will start up our services using a simple Bash script. Windows users should download cygwin to provide a command line functionality similar to a Linux distribution on Windows.
\nBefore you start you should ensure that you have obtained or built the necessary Docker images locally. Please clone the repository and create the necessary images by running the commands as shown:
\ngit clone git@github.com:Fiware/tutorials.IoT-over-MQTT.git\ncd tutorials.IoT-over-MQTT\n\n./services create\n\n\nNote The
\ncontext-providerimage has not yet been pushed to Docker hub.\nFailing to build the Docker sources before proceeding will result in the following error:Pulling context-provider (fiware/cp-web-app:latest)...\nERROR: The image for the service you're trying to recreate has been removed.\n
Thereafter, all services can be initialized from the command line by running the services Bash script provided within the repository:
\n./services start\n\n\n","schema":"https://schema.getpostman.com/json/collection/v2.0.0/collection.json","toc":[{"content":"What is MQTT?","slug":"what-is-mqtt"},{"content":"Architecture","slug":"architecture"},{"content":"Prerequisites","slug":"prerequisites"},{"content":"Start Up","slug":"start-up"}],"owner":"513743","collectionId":"fddcfd01-0c04-4da3-bc34-2ad6ff9fbf0e","publishedId":"RWEiKxtp","public":true,"customColor":{"top-bar":"FFFFFF","right-sidebar":"303030","highlight":"45D3DD"},"publishDate":"2020-01-02T11:03:30.000Z"},"item":[{"name":"Checking Mosquitto Health","item":[],"id":"f051f52b-2e72-46e3-8f0f-161e1bf98df6","description":":information_source: Note: If you want to clean up and start over again you can do so with the following command:
\n\n./services stop\n
We will start by mimicking the roles of both the IoT Agent and a dummy IoT device and send and receive some messages\nusing MQTT. This section of the tutorial requires several open terminals.
\nEventually once we have wired by the system correctly, IoT Agent will subscribe to all relevent events to listen for\nnorthbound traffic in the form of sensor measurements. It therefore will need to make a subscription\nacross all topics. Similarly an accuator must subscribe to a single topic to receive events which effect itself when \ncommands are sent southbound. To check that the lines of communication are open, we can subscribe to a given topic,\nand see that we are able to receive something when a message is published.
\nOpen a new terminal, and create a new running mqtt-subscriber Docker container as follows:
docker run -it --rm --name mqtt-subscriber \\\n --network fiware_default efrecon/mqtt-client sub -h mosquitto -t \"#\"\nThe terminal will then be ready to receive events
\nA sensor sending northbound measurements will publish to those measurements to the MQTT Broker to be passed on to \nany subscriber than wants them. The sensor will not need to make a connection to the subscriber directly.
\nOpen a new terminal, and run a mqtt-publisher Docker container to send a message as follows:
docker run -it --rm -name mqtt-publisher \\\n --network fiware_default efrecon/mqtt-client pub -h mosquitto -m \"HELLO WORLD\" -t \"test\"\nIf the MQTT Broker is functioning correctly, the message should be received in the other terminal
\nHELLO WORLD\nTo terminate the MQTT subscriber, run the following Docker command:
\ndocker stop mqtt-subscriber\nTo show that the communication occurred via the MQTT Broker, we can inspect the log of the mosquitto Docker container as shown:
docker logs --tail 10 mosquitto\n1529661883: New client connected from 172.18.0.5 as mqttjs_8761e518 (c1, k0).\n1529662472: New connection from 172.18.0.7 on port 1883.\n1529662472: New client connected from 172.18.0.7 as mosqpub|1-5637527c63c1 (c1, k60).\n1529662472: Client mosqpub|1-5637527c63c1 disconnected.\n1529662614: New connection from 172.18.0.7 on port 1883.\n1529662614: New client connected from 172.18.0.7 as mosqsub|1-64b27d675f58 (c1, k60).\n1529662623: New connection from 172.18.0.8 on port 1883.\n1529662623: New client connected from 172.18.0.8 as mosqpub|1-ef03e74b0270 (c1, k60).\n1529662623: Client mosqpub|1-ef03e74b0270 disconnected.\n1529667841: Socket error on client mosqsub|1-64b27d675f58, disconnecting.\nOnce the IoT Agent is running, You can check the status by making an HTTP request to the exposed port. \nIf the response is blank, this is usually because the MongoDB database holding the context information is not running or not connected.
\n\n\n","urlObject":{"protocol":"http","path":["iot","about"],"host":["localhost:4041"],"query":[],"variable":[]}},"response":[],"_postman_id":"8d6628ba-66ee-433f-93da-844477a096a3"},{"name":"IoT Agent - Create a Service Group","id":"7d232a2d-6522-4230-9a18-bd35ee197c1e","request":{"method":"POST","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n \"services\": [\n {\n \"apikey\": \"4jggokgpepnvsb2uv4s40d59ov\",\n \"cbroker\": \"http://orion:1026\",\n \"entity_type\": \"Thing\",\n \"resource\": \"\"\n }\n ]\n}"},"url":"http://localhost:4041/iot/services","description":"Troubleshooting: What if the response is blank ?
\n\n
\n- To check that a docker container is running try
\n\ndocker ps\nYou should see several containers running. If
\niot-agentis not running, you can restart the containers as necessary.
Invoking group provision is always the the first step in connecting devices. For MQTT communication, provisioning supplies the authentication key so the IoT Agent will know which topic it must\nsubscribe to.
\nIt is possible to set up default commands and attributes for all devices as well, but this\nis not done within this tutorial as we will be provisioning each device separately.
\nThis example provisions an anonymous group of devices. It tells the IoT Agent that a series of devices\nwill be communicating by sending messages to the /4jggokgpepnvsb2uv4s40d59ov topic
The resource attribute is left blank since HTTP communication is not being used.
The URL location of cbroker is an optional attribute - if it is not provided, the IoT\nAgent uses the default context broker URL as defined in the configuration file, however\nit has been added here for completeness.
It is common good practice to use URNs following the NGSI-LD draft recommendation when creating entities. Furthermore it is easier to understand\nmeaningful names when defining data attributes. These mappings can be defined by provisioning a device individually.
\nThree types of meaasurement attributes can be provisioned:
\nattributes are active readings from the devicelazy attributes are only sent on request - The IoT Agent will inform the device to return the measurementstatic_attributes are as the name suggests static data about the device (such as relationships) passed on \nto the context broker.\n\nNote: in the case where individual
\nids are not required, or aggregated data is sufficient \ntheattributescan be defined within the provisioning service rather than individually.
In the request we are assiociating the device motion001 with the URN urn:ngsd-ld:Motion:001\nand mapping the device reading c with the context attribute count (which is defined as an Integer)\nA refStore is defined as a static_attribute, placing the device within Store urn:ngsi-ld:Store:001
To make a dummy measurement, send a topic to the MQTT Broker as shown.
\ndocker run -it --rm --name mqtt-publisher --network \\\n fiware_default efrecon/mqtt-client pub -h mosquitto -m \"c|1\" -t \"/4jggokgpepnvsb2uv4s40d59ov/motion001/attrs\"\nThen make the context data request.
\nThis example returns the data for the Motion entity with the id=urn:ngsd-ld:Motion:001 within the context data.
The response shows that the device with id=motion001 has been successfully identifed by the IoT Agent and mapped to the\nentity id=urn:ngsd-ld:Motion:001. This new entity has been created within the context data. The c \nattribute from the dummy device measurement request has been mapped to the more meaningful count attribute\nwithin the context. As you will notice, a TimeInstant attribute has been added to both the entity and the\nmeta data of the attribute - this represents the last time the entity and attribute have been updated, and is\nautomatically added to each new entity because the IOTA_TIMESTAMP environment variable was set when the\nIoT Agent was started up.
Provisioning an actuator is similar to provisioning a sensor. The transport attribute defines the communications\nprotocol to be used. For MQTT communications, the endpoint attribute is not required as there is no HTTP url \nwhere the device is listening for commands. The array of commands\nis mapped to directly to messages sent to the /<api-key>/<device-id>/cmd topic \nThe commands array includes a list of each command that can be invoked.
The example below provisions a bell with the deviceId=bell001.
Before we wire-up the context broker, we can test that a command can be sent from the IoT Agent to a\ndevice by making a REST request directly to the IoT Agent's North Port using the /v1/updateContext endpoint.\nIt is this endpoint that will eventually be invoked by the context broker once we have connected it up.\nTo test the configuration you can run the command directly as shown.
If you are viewing the device monitor page, you can also see the state of the bell change.
\n![](\"https://fiware.github.io/tutorials.IoT-Agent/img/bell-ring.gif\")
The result of the command to ring the bell can be read by querying the entity within the Orion Context Broker.
\nThe TimeInstant shows last the time any command associated with the entity has been invoked. the result of ring command can be see in the value of the ring_info attribute
Provisioning a device which offers both commands and measurements is merely a matter of \nmaking an HTTP POST request with both attributes and command attributes in the body of the \nrequest. Once again the transport=MQTT attribute defines the communications protocol to be used, \nand no endpoint attribute is required as there is no HTTP url where the device is listening \nfor commands.
Provisioning a device which offers both commands and measurements is merely a matter of making an HTTP POST request\nwith both attributes and command attributes in the body of the request.
This example provisions a smart door with two commands on and off and maps two attributes.
This example lists all provisioned devices by making a GET request to the /iot/devices endpoint.
The response includes all the commands and attributes mappings associated with all provisioned IoT devices.
\n","urlObject":{"protocol":"http","path":["iot","devices"],"host":["localhost:4041"],"query":[],"variable":[]}},"response":[],"_postman_id":"de6d15a4-f295-45c1-8299-0fe6bb3e1329"}],"id":"d48dcdbb-92a8-4f1b-aa7a-2f7b71817417","description":"The IoT Agent acts as a middleware between the IoT devices and the context broker. It therefore\nneeds to be able to create context data entities with unique ids. Once a service has been provisioned\nand an unknown device makes a measurement the IoT Agent add this to the context using the supplied\n<device-id> (unless the device is recognized and can be mapped to a known id.
There is no guarantee that every supplied IoT device <device-id> will always be unique, therefore \nall provisioning requests to the IoT Agent require two mandatory headers:
fiware-service header is defined so that entities for a given service can be held in a separate mongoDB database.fiware-servicepath can be used to differenciate between arrays of devices.For example within a smart city application you would expect different fiware-service headers for different\ndepartments (e.g. parks, transport, refuse collection etc.) and each fiware-servicepath would refer to specific park \nand so on. This would mean that data and devices for each service can be identified and separated as needed, but the\ndata would not be siloed - for example data from a Smart Bin within a park can be combined with the GPS Unit \nof a refuse truck to alter the route of the truck in an efficient manner.
The Smart Bin and GPS Unit are likely to come from different manufacturers and it cannot be \nguaranteed that that there is no overlap within <device-ids>s used. The use of the fiware-service and\nfiware-servicepath headers can ensure that this is always the case, and allows the context broker to identify\nthe original source of the context data.
This example registers the ring command for the entity with the id=\"urn:ngsi-ld:Bell:001\"
To register the command we need to inform Orion that the URL http://orion:1026/v1 is able to provide\nthe missing ring attribute. This will then be forwarded on to the IoT Agent. As you see this is an NGSI v1 \nendpoint and therefore the legacyForwarding attribute must also be set.
Entity type, to avoid ambiguity in case there are several entities with the same entity id
\n","type":"text/plain"},"key":"type","value":"Store"},{"disabled":true,"description":{"content":"Ordered list of attribute names to display
\n","type":"text/plain"},"key":"attrs","value":"name"}],"variable":[]}},"response":[],"_postman_id":"69240a49-f0e6-47f0-89d4-8c2f650a6f02"},{"name":"Ring Bell","id":"eeb5e73d-15fe-47d3-9369-da47ff875fe9","request":{"method":"PATCH","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n\t\"ring\": {\n \t\"type\" : \"command\",\n \t\"value\" : \"\"\n\t}\n}"},"url":"http://localhost:1026/v2/entities/urn:ngsi-ld:Bell:001/attrs","description":"To invoke the ring command, the ring attribute must be updated in the context.
If you are viewing the device monitor page, you can also see the state of the bell change.
\n
The result of the command to ring the bell can be read by querying the entity within the Orion Context Broker.
\nThe TimeInstant shows last the time any command associated with the entity has been invoked. the result of ring command can be see in the value of the ring_info attribute
The Bell entity has been mapped to id=\"urn:ngsi-ld:Bell:001\" with an entity type=\"Bell\"
This example registers the lock, unlock, open and close commands for the entity with\nthe id=\"urn:ngsi-ld:Door:001\"
To register the commands we need to inform Orion that the URL http://orion:1026/v1 is able to provide\nthe missing attributes. This will then be forwarded on to the IoT Agent. As you see this is an NGSI v1 \nendpoint and therefore the legacyForwarding attribute must also be set.
Entity type, to avoid ambiguity in case there are several entities with the same entity id
\n","type":"text/plain"},"key":"type","value":"Store"},{"disabled":true,"description":{"content":"Ordered list of attribute names to display
\n","type":"text/plain"},"key":"attrs","value":"name"}],"variable":[]}},"response":[],"_postman_id":"d4b0b1be-4fa6-4a1b-a460-3ab875105849"},{"name":"Open a Door","id":"62dd2cf7-f7e6-4b14-b760-a7e81f0da36a","request":{"method":"PATCH","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n\t\"open\": {\n \t\"type\" : \"command\",\n \t\"value\" : \"\"\n\t}\n}"},"url":"http://localhost:1026/v2/entities/urn:ngsi-ld:Door:001/attrs","description":"To invoke the open command, the open attribute must be updated in the context.
The result of the invocation of Smart Door commands can be read by querying the entity within the Orion Context Broker.
\nThe TimeInstant shows last the time any command associated with the entity has been invoked.
open command can be see in the value of the open_info attributeclose command can be see in the value of the close_info attributelock command can be see in the value of the lock_info attributeunlock command can be see in the value of the unlock_info attributeThe Smart Door entity has been mapped to id=\"urn:ngsi-ld:Door:001\" with an entity type=\"Door\"
This example registers the on and off commands for the entity with\nthe id=\"urn:ngsi-ld:Lamp:001\"
To register the commands we need to inform Orion that the URL http://orion:1026/v1 is able to provide\nthe missing attributes. This will then be forwarded on to the IoT Agent. As you see this is an NGSI v1 \nendpoint and therefore the legacyForwarding attribute must also be set.
Entity type, to avoid ambiguity in case there are several entities with the same entity id
\n","type":"text/plain"},"key":"type","value":"Store"},{"disabled":true,"description":{"content":"Ordered list of attribute names to display
\n","type":"text/plain"},"key":"attrs","value":"name"}],"variable":[]}},"response":[],"_postman_id":"04a8ed23-b4ef-4559-9970-f779e32b91c2"},{"name":"Switch On a Lamp","id":"2ab37c1d-b8f3-4dd4-8057-a06f1241d079","request":{"method":"PATCH","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n\t\"on\": {\n \t\"type\" : \"command\",\n \t\"value\" : \"\"\n\t}\n}"},"url":"http://localhost:1026/v2/entities/urn:ngsi-ld:Lamp:001/attrs","description":"To switch on the Smart Lamp, the on attribute must be updated in the context.
The result of the invocation of Smart Lamp commands can be read by querying the entity within the Orion Context Broker.
\nThe TimeInstant shows last the time any command associated with the entity has been invoked.
on command can be see in the value of the on_info attributeoff command can be see in the value of the off_info attributeThe Smart Lamp entity has been mapped to id=\"urn:ngsi-ld:Lamp:001\" with an entity type=\"Lamp\"
This example returns the data of all Store entities within the context data
Having connected up the IoT Agent to the IoT devices, we now need to inform the Orion Context Broker that the commands\nare available. In other words we need to register the IoT Agent as a Context Provider for the command attributes.
\nOnce the commands have been registered it will be possible to ring the Bell, open and close the Smart Door and\nswitch the Smart Lamp on and off by sending requests to the Orion Context Broker, rather than sending UltraLight 2.0\nrequests directly t the IoT devices as we did in the previous tutorial
\n","event":[{"listen":"prerequest","script":{"id":"e5aa0b75-9618-464b-9547-bccf5360d530","type":"text/javascript","exec":[""]}},{"listen":"test","script":{"id":"862fb9fb-be99-4b36-bb84-8ec7f5638ede","type":"text/javascript","exec":[""]}}],"_postman_id":"10053cd1-528f-4891-baa2-c1bb09ca245d"}],"event":[{"listen":"prerequest","script":{"id":"e64b3c59-ab46-46cf-b504-30a027ca2986","type":"text/javascript","exec":[""]}},{"listen":"test","script":{"id":"04676146-332d-41b3-a65e-b9fd5a527aca","type":"text/javascript","exec":[""]}}],"variable":[{"key":"iot-agent","value":"localhost:4041"},{"key":"orion","value":"localhost:1026"},{"key":"ultralight","value":"localhost:3001"}]}