![\"FIWARE](\"https://nexus.lab.fiware.org/repository/raw/public/badges/chapters/iot-agents.svg\")
\n![\"NGSI](\"https://img.shields.io/badge/NGSI-v2-5dc0cf.svg\"){kind=icon}
\n
This tutorial a wires up the dummy IoT devices which are responding using a custom XML\nmessage format. A custom IoT Agent is created based on the IoT Agent Node.js\nlibrary and the framework found in the\nIoT Agent for Ultralight\ndevices so that measurements can be read and commands can be sent using\nNGSI-v2 requests sent to the\nOrion Context Broker.
\nThe tutorial uses cUrl commands throughout, but is also available as\nPostman documentation
\nThe docker-compose files for this tutorial can be found on GitHub:
![\"GitHub\"](\"https://fiware.github.io/tutorials.Custom-IoT-Agent/icon/GitHub-Mark-32px.png\"){kind=icon}
FIWARE 205: Creating a Custom IoT Agent
As defined previously, an IoT Agent is a component that lets a group of devices send their data to and be managed from a\nContext Broker using their own native protocols. Every IoT Agent is defined for a single payload format, although they\nmay be able to use multiple disparate transports for that payload.
\nIoT Agents for many standard payloads exist, however it is possible to envisage that additional payloads may be needed\nas many potential sources of context data have already their own well-defined de-facto or de-jure standards for passing\ndata around systems. As an example the ISOXML standard\niso:11783 is frequently used with Agricultural\nMachinery.
\nThe process for creating your own IoT Agent is relatively simple. It is best achieved through selecting an IoT Agent\nwhich uses the required data transport and rewriting/amending the payload processing code to handle the payloads in\nquestion.
\nFor the purpose of this tutorial we will amend code from the existing Ultralight IoT Agent to process a similar custom\nXML format. A direct comparison of the two IoT Agents can be seen below:
\nAs can be seen, the supported communications transports (HTTP, MQTT, AMPQ) remain the same, it is processing of the\ncustom payload which will need to be adapted to ensure that the XML devices can communicate with the IoT Agent.
\nIt should be noted that, depending on your use case, it also may be necessary to create an additional middleware for\ncommunications purposes. In this example the devices are capable of sending measures and listening and responding to\ncommands directly on two separate comms channels. A different paradigm is used within the\nLoRaWAN and OPC-UA IoT Agents where an\nHTTP middleware responds to the IoT Agent, and it is then responsible for converting the communications to the\nlower-level CoAP transport used by the devices.
\nThe aim of this tutorial is to improve developer understanding of how to create their own custom IoT Agents, a series of\nsimple modificiations has been made to the code of the Ultralight IoT Agent demonstrating how to make changes. The\ntutorial consists of a walk-through of the relevant code and a series of HTTP requests to connect the new IoT Agent. The\ncode can be found within the current\nGitHub Repository
\nThe benefit of modifying an existing IoT Agent is that the developer will be able to reuse the common functionality\nfound across all IoT Agents. This includes functions such as:
\nThis base functionality has been abstracted out into a common\nIoT Agent framework library
\nFor the purpose of this tutorial, a series of dummy IoT devices have been created, which will be attached to the context\nbroker. Details of the architecture and protocol used can be found in the\nIoT Sensors tutorial The state of each device can be seen on the JSON\ndevice monitor web page found at: http://localhost:3000/device/monitor
![\"FIWARE](\"https://fiware.github.io/tutorials.IoT-Agent-JSON/img/device-monitor.png\")
This application builds on the components created in\nprevious tutorials. It will make use of one FIWARE component - the\nOrion Context Broker plus a Custom IoT Agent for XML. Usage of the\nOrion Context Broker is sufficient for an application to qualify as “Powered by FIWARE”. Both the Orion Context Broker\nand the IoT Agent and rely on open source MongoDB technology to keep persistence of the\ninformation they hold. We will also be using the dummy IoT devices created in the\nprevious tutorial, however they have been already been adapted to\nrespond to the custom XML messaging format.
\nTherefore the overall architecture will consist of the following elements:
\nSince all interactions between the elements are initiated by HTTP requests, the entities can be containerized and run\nfrom exposed ports.
\n![\"\"](\"https://fiware.github.io/tutorials.IoT-Agent-JSON/img/architecture.png\")
The necessary configuration information for wiring up the IoT devices and the IoT Agent can be seen in the services\nsection of the associated docker-compose.yml file:
tutorial:\n image: fiware/tutorials.context-provider\n hostname: iot-sensors\n container_name: fiware-tutorial\n networks:\n - default\n expose:\n - \"3000\"\n - \"3001\"\n ports:\n - \"3000:3000\"\n - \"3001:3001\"\n environment:\n - \"DEBUG=tutorial:*\"\n - \"PORT=3000\"\n - \"IOTA_HTTP_HOST=iot-agent\"\n - \"IOTA_HTTP_PORT=7896\"\n - \"DUMMY_DEVICES_PORT=3001\"\n - \"DUMMY_DEVICES_API_KEY=4jggokgpepnvsb2uv4s40d59ov\"\n - \"DUMMY_DEVICES_TRANSPORT=HTTP\"\n - \"DUMMY_DEVICES_PAYLOAD=XML\"\nThe tutorial container is listening on two ports:
3000 is exposed so we can see the web page displaying the Dummy IoT devices.3001 is exposed purely for tutorial access - so that cUrl or Postman can make JSON commands without being\npart of the same network.The tutorial container is driven by environment variables as shown:
The other tutorial container configuration values described in the YAML file are not used in this tutorial.
The code for the custom XML IoT Agent can be found within the\nGitHub Repository associated to this\ntutorial. It is a copy of the 1.12.0 version of the IoT Agent for Ultralight, lightly modified as described below. The\nassociated Dockerfile merely\ncopies the code into an appropiate location within a Docker container running Node.js. This allows the component to be\ninstansiated using a docker-compose.yaml file. The necessary configuration can be seen below:
iot-agent:\n image: fiware/iotagent-xml\n build:\n context: iot-agent\n dockerfile: Dockerfile\n hostname: iot-agent\n container_name: fiware-iot-agent\n depends_on:\n - mongo-db\n networks:\n - default\n expose:\n - \"4041\"\n - \"7896\"\n ports:\n - \"4041:4041\"\n - \"7896:7896\"\n environment:\n - IOTA_CB_HOST=orion\n - IOTA_CB_PORT=1026\n - IOTA_NORTH_PORT=4041\n - IOTA_REGISTRY_TYPE=mongodb\n - IOTA_LOG_LEVEL=DEBUG\n - IOTA_TIMESTAMP=true\n - IOTA_CB_NGSI_VERSION=v2\n - IOTA_AUTOCAST=true\n - IOTA_MONGO_HOST=mongo-db\n - IOTA_MONGO_PORT=27017\n - IOTA_MONGO_DB=iotagentjson\n - IOTA_HTTP_PORT=7896\n - IOTA_PROVIDER_URL=http://iot-agent:4041\n - IOTA_DEFAULT_RESOURCE=/iot/xml\nThe iot-agent container relies on the precence of the Orion Context Broker and uses a MongoDB database to hold device\ninformation such as device URLs and Keys. The container is listening on two ports:
7896 is exposed to receive JSON measurements over HTTP from the Dummy IoT devices4041 is exposed purely for tutorial access - so that cUrl or Postman can make provisioning commands without\nbeing part of the same network.The iot-agent container is driven by environment variables as shown:
Once the IoT Agent is running, You can check the status by making an HTTP request to the exposed port. \nIf the response is blank, this is usually because the MongoDB database holding the context information is not running or not connected.
\nThis is standard functionality coming directly from the IoT Agent Node.js library and does not involve an code changes.
\n","urlObject":{"protocol":"http","path":["iot","about"],"host":["localhost:4041"],"query":[],"variable":[]}},"response":[],"_postman_id":"712284a0-510a-4297-a6dd-674aee83869b"},{"name":"IoT Agent - Create a Service Group","id":"0a79c36c-a6ad-46a0-92dc-29e15c1b31ce","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"POST","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n \"services\": [\n {\n \"apikey\": \"4jggokgpepnvsb2uv4s40d59ov\",\n \"cbroker\": \"http://orion:1026\",\n \"entity_type\": \"Thing\",\n \"resource\": \"/iot/xml\"\n }\n ]\n}"},"url":"http://localhost:4041/iot/services","description":"In the example the IoT Agent is informed that the /iot/xml endpoint will be used and that devices will authenticate\nthemselves by including the token 4jggokgpepnvsb2uv4s40d59ov. For the custom XML IoT Agent this means devices will be\nsending POST requests to:
http://iot-agent:7896/iot/xml\nWhere the <measure> holds the relevant device ID and API key.
<measure device=\"motion001\" key=\"4jggokgpepnvsb2uv4s40d59ov\">\n <c value=\"3\"/>\n</measure>\nThis syntax differs from the Ultralight IoT Agent where the device ID and API key are sent as URL parameters.
\nThe relevant changes can be found in the HTTPBindings.js file where an XML parser is instanciated.
const xmlBodyParser = require(\"express-xml-bodyparser\");\nhttpBindingServer.router.post(\n config.getConfig().iota.defaultResource || constants.HTTP_MEASURE_PATH,\n ...xmlBodyParser({ trim: false, explicitArray: false }),\n checkMandatoryParams(false),\n ...etc\n);\nThis means that attributes from the XML request can be accessed using the square-bracket syntax. Since both apiKey and\ndeviceId are required parameters, they can be found within the received <measure>.
function checkMandatoryParams(queryPayload) {\n return function(req, res, next) {\n var notFoundParams = [],\n error;\n\n req.apiKey = req.body[\"measure\"][\"$\"][\"key\"];\n req.deviceId = req.body[\"measure\"][\"$\"][\"device\"];\n\n if (!req.apiKey) {\n notFoundParams.push(\"API Key\");\n }\n\n if (!req.deviceId) {\n notFoundParams.push(\"Device Id\");\n }\n\n // CHeck if retrievingParam\n if (queryPayload && !req.query.d && req.query.getCmd !== \"1\") {\n notFoundParams.push(\"Payload\");\n }\n\n if (req.method === \"POST\" && !req.is(\"application/xml\")) {\n error = new errors.UnsupportedType(\"application/xml\");\n }\n\n if (notFoundParams.length !== 0) {\n next(new errors.MandatoryParamsNotFound(notFoundParams));\n } else {\n next(error);\n }\n };\n}\nThis function also checks that the appropriate MIME type has been received and fails fast if the incoming message does\nnot hold sufficient information.
\n","urlObject":{"protocol":"http","path":["iot","services"],"host":["localhost:4041"],"query":[],"variable":[]}},"response":[],"_postman_id":"0a79c36c-a6ad-46a0-92dc-29e15c1b31ce"},{"name":"IoT Agent - Provision a Sensor","id":"544ef54f-f23a-4121-bad3-7194517acee6","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"POST","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n \"devices\": [\n {\n \"device_id\": \"motion001\",\n \"entity_name\": \"urn:ngsi-ld:Motion:001\",\n \"entity_type\": \"Motion\",\n \"timezone\": \"Europe/Berlin\",\n \"attributes\": [\n { \"object_id\": \"c\", \"name\":\"count\", \"type\":\"Integer\"}\n ],\n \"static_attributes\": [\n {\"name\":\"refStore\", \"type\": \"Relationship\",\"value\": \"urn:ngsi-ld:Store:001\"}\n ]\n }\n ]\n}\n"},"url":"http://localhost:4041/iot/devices","description":"As expected the HTTP command to provision a device does not change based on the underlying payload or transport\nprotocol since we are using the same HTTP transport as the original Ultralight IoT Agent. internal_atttributes can be\nused to supply additional information for the custom IoT Agent if necessary. In the request we are associating the\ndevice motion001 with the URN urn:ngsi-ld:Motion:001 and mapping the device reading c with the context attribute\ncount (which is defined as an Integer) A refStore is defined as a static_attribute, placing the device within\nStore urn:ngsi-ld:Store:001
You can simulate a dummy IoT device measurement coming from the Motion Sensor device motion001, by making the\nfollowing XML request
Both the payload and the Content-Type have been updated. The dummy devices made a similar Ultralight request in the\nprevious tutorials when the door was unlocked, you will have seen the state of each motion sensor changing and a\nNorthbound request will be logged in the device monitor.
Now the IoT Agent is connected, the service group has defined the resource upon which the IoT Agent is listening\n(iot/xml) and the API key used to authenticate the request (4jggokgpepnvsb2uv4s40d59ov) is found in the body. Since\nboth of these are recognized, the measurement is valid.
The next step is to parse the payload extract the attributes. This can be found in the amended parse method of the\nxmlparser.js file
function parse(payload) {\n let result = [];\n const keys = Object.keys(payload[\"measure\"]);\n for (let i = 0; i < keys.length; i++) {\n if (keys[i] !== \"$\") {\n let obj = {};\n obj[keys[i]] = payload[\"measure\"][keys[i]][\"$\"].value;\n result.push(obj);\n }\n }\n return result;\n}\nparse() returns a JSON array of key-value pairs which can then be mapped from device attribute names (like c) to\nentity attribute names (like count) - obviously the mapping is based on the values sent in the original provisioning.
You can see that a measurement has been recorded, by retrieving the entity data from the context broker. Don't forget to\nadd the fiware-service and fiware-service-path headers.
The response shows that the device with id=motion001 has been successfully identifed by the IoT Agent and mapped to the\nentity id=urn:ngsd-ld:Motion:001. This new entity has been created within the context data. The c \nattribute from the dummy device measurement request has been mapped to the more meaningful count attribute\nwithin the context. As you will notice, a TimeInstant attribute has been added to both the entity and the\nmeta data of the attribute - this represents the last time the entity and attribute have been updated, and is\nautomatically added to each new entity because the IOTA_TIMESTAMP environment variable was set when the\nIoT Agent was started up. The refStore attribute comes from the static_attributes set when the device was provisioned.
Provisioning an actuator is similar to provisioning a sensor. This time an endpoint attribute holds the location where\nthe IoT Agent needs to send the JSON command and the commands array includes a list of each command that can be\ninvoked. The example below provisions a bell with the deviceId=bell001. The endpoint is\nhttp://iot-sensors:3001/iot/bell001 and it can accept the ring command. The transport=HTTP attribute defines the\ncommunications protocol to be used.
As is the case with provisioning a measure, this request remains unchanged. Implicitly within the structure of the IoT\nAgent the provisioning of commands fulfills the following implied contract:
\n/v2/op/update endpoint)<command>State attribute, amending and forwarding the request on a /cmd endpoint to\nthe device (or alternatively to a middleware responsible for the device).The first two items - listening to context changes from the context broker follow the well-defined NGSI syntax and\ntherefore are common to all IoT Agents. However the third item - what to do to prepare the message for ongoing\nconsumption will vary according to the protocol which is being abstracted out.
\n","urlObject":{"protocol":"http","path":["iot","devices"],"host":["localhost:4041"],"query":[],"variable":[]}},"response":[],"_postman_id":"1e4c5df9-21fd-495a-a78a-04c9180be0f8"},{"name":"IoT Agent - Invoke a Command","id":"652cfcde-ac5f-44b3-bc21-0e699c29cb35","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"POST","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n \"actionType\": \"update\",\n \"entities\": [\n {\n \"type\": \"Bell\",\n \"id\": \"urn:ngsi-ld:Bell:001\",\n \"ring\" : {\n \"type\": \"command\",\n \"value\": \"\"\n }\n }\n ]\n}"},"url":"http://localhost:4041/v2/op/update","description":"Before we wire-up the context broker, we can test that a command can be send to a device by making a REST request\ndirectly to the IoT Agent's North Port using the /v2/op/update endpoint. It is this endpoint that will eventually be\ninvoked by the context broker once we have connected it up. To test the configuration you can run the command directly\nas shown
If you are viewing the device monitor page, you can also see the state of the bell change.
\n![](\"https://fiware.github.io/tutorials.IoT-Agent/img/bell-ring.gif\")
The result of the command to ring the bell can be read by querying the entity within the Orion Context Broker.
\nWithin the custom IoT Agent the start() function sets a series of handler functions to fire when a request arrives\nfrom the context broker.
iotAgentLib.setProvisioningHandler(deviceProvisioningHandler);\niotAgentLib.setConfigurationHandler(configurationHandler);\niotAgentLib.setCommandHandler(commandHandler);\niotAgentLib.setDataUpdateHandler(updateHandler);\nThis is passed to the appropriate transport binding and in this case, the commandHandler() method within\nHTTPBindings.js is fired. It provides an HTTP error handler, but delegates the actual work of creating and sending the\ncommand to the device to generateCommandExecution().
function generateCommandExecution(apiKey, device, attribute) {\n...\n const options = {\n url: device.endpoint,\n method: 'POST',\n body: xmlParser.createCommandPayload(device, cmdName, cmdAttributes),\n headers: {\n 'fiware-service': device.service,\n 'fiware-servicepath': device.subservice\n }\n };\n... etc\nThe payload itself, in other words how a command must be created such that it can be interpreted by the device, is\nunique to our custom XML messaging protocol and is generated in createCommandPayload() method in xmlParser.js.
function createCommandPayload(device, command, attributes) {\n if (typeof attributes === \"object\") {\n let payload = \"<\" + command + ' device=\"' + device.id + '\">';\n\n Object.keys(attributes).forEach(function(key, value) {\n payload = payload + \"<\" + key + \">\" + value + \"</\" + key + \">\";\n });\n payload = payload + \"</\" + command + \">\";\n return payload;\n } else {\n return \"<\" + command + ' device=\"' + device.id + '\"/>';\n }\n}\nThis is an amendment from the Ultralight protocol where the @ and | symbol is generated for Ultralight devices.
Howver creating a payload is only half the job, it must be sent to the device and understood, so communications must be\ncompleted using a well-defined communications handshake. So after generating the payload the sendXMLCommandHTTP()\nmethod of HTTPBindings.js sends the message and passes the response to the result() method in xmlParser.js to\ninterprets the command response from the device.
function result(payload) {\n const xmlToJson = require(\"xml-parser\");\n const data = xmlToJson(payload);\n const result = {};\n result.deviceId = data.root.attributes.device;\n result.command = data.root.attributes.command;\n result.result = data.root.name;\n\n return result;\n}\nFinally the success or failure of the command is updated into the context broker using common code from the IoT Agent\nnode library.
\nAs is typical for IoT Agents, creation of payloads and the handshake of communications has been split into two separate\nconcerns for ease of maintenance. Therefore since in our case only the payload has changed, it is only the XML payload\nside of the code that needs modification to fulfil our custom use case.
\n","urlObject":{"protocol":"http","path":["v2","op","update"],"host":["localhost:4041"],"query":[],"variable":[]}},"response":[],"_postman_id":"652cfcde-ac5f-44b3-bc21-0e699c29cb35"},{"name":"Context Broker - Read the Command Result","id":"22ec0981-8167-45a8-a72a-9cd5a9eb0c44","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"GET","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"}],"url":"http://localhost:1026/v2/entities/urn:ngsi-ld:Bell:001?options=keyValues","description":"The result of the command to ring the bell can be read by querying the entity within the Orion Context Broker.
\n","urlObject":{"protocol":"http","path":["v2","entities","urn:ngsi-ld:Bell:001"],"host":["localhost:1026"],"query":[{"key":"options","value":"keyValues"}],"variable":[]}},"response":[],"_postman_id":"22ec0981-8167-45a8-a72a-9cd5a9eb0c44"},{"name":"IoT Agent - Provision a Smart Door","id":"664521e3-36d4-4473-b136-9bc85b6abb21","protocolProfileBehavior":{"disableBodyPruning":true},"request":{"method":"POST","header":[{"key":"fiware-service","value":"openiot"},{"key":"fiware-servicepath","value":"/"},{"key":"Content-Type","value":"application/json"}],"body":{"mode":"raw","raw":"{\n \"devices\": [\n {\n \"device_id\": \"door001\",\n \"entity_name\": \"urn:ngsi-ld:Door:001\",\n \"entity_type\": \"Door\",\n \"protocol\": \"PDI-IoTA-UltraLight\",\n \"transport\": \"HTTP\",\n \"endpoint\": \"http://context-provider:3001/iot/door001\",\n \"commands\": [ \n {\"name\": \"unlock\",\"type\": \"command\"},\n {\"name\": \"open\",\"type\": \"command\"},\n {\"name\": \"close\",\"type\": \"command\"},\n {\"name\": \"lock\",\"type\": \"command\"}\n ],\n \"attributes\": [\n \t{\"object_id\": \"s\", \"name\": \"state\", \"type\":\"Text\"}\n ],\n \"static_attributes\": [\n {\"name\":\"refStore\", \"type\": \"Relationship\",\"value\": \"urn:ngsi-ld:Store:001\"}\n ]\n }\n ]\n}\n"},"url":"http://localhost:4041/iot/devices","description":"Provisioning a device which offers both commands and measurements is merely a matter of making an HTTP POST request\nwith both attributes and command attributes in the body of the request.
This example provisions a smart door with four commands lock, unlock, open and close and maps a single attribute.
Provisioning a device which offers both commands and measurements is merely a matter of making an HTTP POST request\nwith both attributes and command attributes in the body of the request.
This example provisions a smart door with two commands on and off and maps two attributes.
This example lists all provisioned devices by making a GET request to the /iot/devices endpoint.
The response includes all the commands and attributes mappings associated with all provisioned IoT devices.
\n","urlObject":{"protocol":"http","path":["iot","devices"],"host":["localhost:4041"],"query":[],"variable":[]}},"response":[],"_postman_id":"99bd9349-99fe-479d-b206-aee5d78f512b"}],"id":"e25235b5-dcce-46f6-8ecf-736cde03e560","description":"The following section consists of a series of HTTP commands used to provision the IoT Agent and receive measures and\nsend commands. The relevant amended code within the Custom IoT Agent is discussed as each action is processed.
\nTo follow the tutorial correctly please ensure you have the device monitor page available in your browser and click on\nthe page to enable audio before you enter any cUrl commands. The device monitor displays the current state of an array\nof dummy devices using XML syntax
\nThe device monitor can be found at: http://localhost:3000/device/monitor