Advanced Data Solutions : Mosquitto Client Tools and Azure IoT Hub. The beginning of something good....

Public cloud brings a paradigm shift in what can be done, the art of the possible is possible and in the context of this post, I am planning to connect my existing MQTT-based Smart Home system to Azure.

From ML (Machine Learning) through to anomaly detection and everything in between, the bells and whistles Azure graces myself is far and beyond the capabilities of any on premise system, coupled with low administrative effort of consuming a managed service.

Who has the longest showers? Who wakes up first? Which devices are powered on, yet there is no activity? These questions and many can all be answered from the thousands events occurring each day in my house, but first we need to get these events in to the Cloud, a.k.a. Azure.

The thing is whilst familiar with MQTT, I am new to Azure IoT (and Azure in general). So, in this multi-part series of blog posts, let's learn together as I walk you through my journey.

The inception

Today I use Mosquitto (in Docker container) locally with Home Assistant, Tasmota devices, a PLC and an Arduino Mega. I publish and consume thousands of messages each day (1.4 messages per second, which is mostly PIR data). The easiest path here is to not have these devices publish to Azure IoT Hub (the Azure service used as a Cloud gateway for IoT devices) directly, but to have Mosquitto replicate all messages locally through to Azure IoT Hub.

Remember the three laws of IoT:

The Law of Physics : Latency to the cloud can be unacceptable, think crash avoidance system and medical alerts. So some decision-making must continue to be executed locally on the device. High-value and safety-critical processes must always continue working.
The Law of Economics : The cost of bandwidth is not falling as fast as the cost of storage and compute.

We might act on low value data locally and upload high value and aggregate data to the cloud for analytics and storage.
The Law of the Land: For legal or compliance reasons, and concerns around privacy, some industries prefer to store data locally more-so, some governments impose data sovereignty restrictions on where data may be stored.

But here, it’s my snarky 10-year-old wondering why there is so much lag when a button is pressed (The law of physics). Therefore, some decision-making must continue to be executed locally and in the absence of Internet connection, my house must continue to function. Ideally something like this image below, once we are publishing events in Azure IoT Hub we are then able to leverage these in the Azure platform as described above.

What I want to do

The thing is, you need to crawl before you can walk and the first thing I did was to create an Azure IoT Hub instance, create a device identity and publish/subscribe MQTT (version 3.11 compliant) messages via Mosquitto client tools (mosquitto_pub / mosquitto_sub) to Azure IoT Hub, all this using the Azure CLI.

What we will do

The first thing I did was to leverage my favorite search engine to see if this has been done before I found this interesting guide (https://github.com/Azure-Samples/IoTMQTTSample/tree/master/src/Mosquitto_pub), and whilst light on details, 2 hours later my MQTT messages started flowing in my IoT Hub, but intially I had many questions. Why use a SAS token for auth? How do I create my Azure IoT Hub? How do I create my device? But more importantly how can I do this in a way that will be part of a build pipeline?

What is Azure IoT Hub?

Azure IoT Hub brings highly secure and reliable communication between your Internet of Things (IoT) application and the devices it manages. Some key features are per-device authentication, built-in device management and scaled provisioning. From an IoT perspective it provides AMQP, MQTT and HTTP endpoints (it's MQTT we will be focusing on in this post). It does a lot, but in the context of this post, think of it as your Cloud gateway for devices that you don't need to manage.

How about AZ-CLI?

Simply put, your CLI (Command Line Interface) for Azure. It is a cross-platform CLI to connect to Azure and execute administrative commands on Azure resources. It allows the execution of commands through a terminal using interactive command-line prompts or a script. The Azure CLI, known as AZ CLI is available across Azure services and is designed to get you working quickly with Azure, with an emphasis on automation and I see it as that interim step between the console and using a direct SDK or service API. You can find more information on Azure CLI here.

You can install the Azure CLI locally on Linux, Mac, or Windows computers. It can also be used from a browser through the Azure Cloud Shell (very cool :thumbs_up::thumbs_up:) or run from inside a Docker container.

I am not going to cover setting up Azure CLI in here. See the Azure CLI setup guide for more information on setting up and configuring for your subscription but suffice to say, Azure CLI ticks that box, which will allow build pipeline integration.Enough about this, lets get started.

The implementation

High Level Steps

Create an Azure IoT Hub instance
Extract Connection Strings
Create an IoT Device identity
Create a SAS Policy
TLS Certificate
Pulling it all together
Mosquitto_pub/Mosquitto_sub

Variables You Will Need To Substitute

This post serves as a high level walkthrough for establishing full duplex (publish and subscribe) communication from your local broker in to Azure. You will be presented with outputs from CLI or from your Azure subscription, that you will need to substitute with your values in order to pass in to the Mosquitto client tools. Some key values you need to substitute are the following:

Resource-group 
Device name
SAS token 
Connection string

Step 1: Create the Azure IoT Hub

You will need to firstly create your Azure IoT Hub instance. There are many SKU’s but the F1 SKU provides 8000 messages per day for free and is a good way to get started without incurring cost. Alternate SKU’s can be found here.

az iot hub create --resource-group BaldacchinoRG --name Baldacchino-IOTHub -sku F1 --partition-count 2

The output of this command will provide a lot of JSON. Extract your Azure IoT Hub endpoint. Copy and paste this in to a document in your favourite IDE, we will be adding many values to this document to build out the required.

"hostName": "BaldacchinoIotHub.azure-devices.net",

Step 2: Extract the Primary Connection String

You will need to identify the connection string of your IoT Hub

az iot hub connection-string show

This output of this command will provide your connection string (there is two, this will only show the primary). Copy and paste this in to your document.

[
  {
    "connectionString": "HostName=Baldacchino-IOTHub.azure-devices.net;SharedAccessKeyName=iothubowner;SharedAccessKey=******************************************************=",
    "name": "Baldacchino-IOTHub"
  }
]

Step 3: Create your device

You now need to create your MQTT device, for the purpose of this walkthrough we will use a SAS token for authentication.

az iot hub device-identity create -n Baldacchino-IOTHub -d Mosquitto

There is no JSON you will need to copy to your clipboard here but you will notice in the JSON output that the type is SAS and we are not using x509 certificates.

"type": "sas",
"x509Thumbprint": {
"primaryThumbprint": null,
"secondaryThumbprint": null
}
},
"capabilities": {
"iotEdge": false
}

Step 4: Generate a SAS token

There are multiple ways to provide Authentication. SAS tokens and x509 certificates are the common approaches. The SAS token is a string that you generate on the client side, and you pass this string to Azure IoT Hub for authentication. Azure IoT Hub then checks the SAS parameters and the signature to verify that it is valid. In a production environment you will likely prefer using X-509 certificates for this authentication but for the sake of my project, SAS token will work.

az iot hub generate-sas-token -d Mosquitto -n Baldacchino-IOTHub

Copy and paste the SharedAccessSignature from the JSON output in your document.

{
  "sas": "SharedAccessSignature sr=Baldacchino-IOTHub.azure-devices.net%2Fdevices%2FMosquitto&sig=%2BdJrIWgg6XIzaT9sIvRZGSMYGv9lRKwihG2JnyEXMO4%3D&se=1633391816"
}

Step 5: We need TLS

MQTT typically runs on port 1833 in an unsecure manner but Azure IoT Hub mandates MQTT over TLS on 8883. The root CA might not be part of your Operating System’s keychain and as such you will need to download the PEM file to use with Mosquitto.

You can download the certificate from GitHub at https://raw.githubusercontent.com/Azure/azure-iot-sdk-c/master/certs/certs.c. Copy the Baltimore certificate, to save you some time I have pasted this below, and save it as Baltimore.pem

You can validate your TLS certificate by using openSSL

openssl x509 -in Balitore.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 33554617 (0x20000b9)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
Validity
Not Before: May 12 18:46:00 2000 GMT
Not After : May 12 23:59:00 2025 GMT
Subject: C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:a3:04:bb:22:ab:98:3d:57:e8:26:72:9a:b5:79:
d4:29:e2:e1:e8:95:80:b1:b0:e3:5b:8e:2b:29:9a:
64:df:a1:5d:ed:b0:09:05:6d:db:28:2e:ce:62:a2:
62:fe:b4:88:da:12:eb:38:eb:21:9d:c0:41:2b:01:
52:7b:88:77:d3:1c:8f:c7:ba:b9:88:b5:6a:09:e7:
73:e8:11:40:a7:d1:cc:ca:62:8d:2d:e5:8f:0b:a6:
50:d2:a8:50:c3:28:ea:f5:ab:25:87:8a:9a:96:1c:
a9:67:b8:3f:0c:d5:f7:f9:52:13:2f:c2:1b:d5:70:
70:f0:8f:c0:12:ca:06:cb:9a:e1:d9:ca:33:7a:77:
d6:f8:ec:b9:f1:68:44:42:48:13:d2:c0:c2:a4:ae:
5e:60:fe:b6:a6:05:fc:b4:dd:07:59:02:d4:59:18:
98:63:f5:a5:63:e0:90:0c:7d:5d:b2:06:7a:f3:85:
ea:eb:d4:03:ae:5e:84:3e:5f:ff:15:ed:69:bc:f9:
39:36:72:75:cf:77:52:4d:f3:c9:90:2c:b9:3d:e5:
c9:23:53:3f:1f:24:98:21:5c:07:99:29:bd:c6:3a:
ec:e7:6e:86:3a:6b:97:74:63:33:bd:68:18:31:f0:
78:8d:76:bf:fc:9e:8e:5d:2a:86:a7:4d:90:dc:27:
1a:39
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
E5:9D:59:30:82:47:58:CC:AC:FA:08:54:36:86:7B:3A:B5:04:4D:F0
X509v3 Basic Constraints: critical
CA:TRUE, pathlen:3
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
Signature Algorithm: sha1WithRSAEncryption
85:0c:5d:8e:e4:6f:51:68:42:05:a0:dd:bb:4f:27:25:84:03:
bd:f7:64:fd:2d:d7:30:e3:a4:10:17:eb:da:29:29:b6:79:3f:
76:f6:19:13:23:b8:10:0a:f9:58:a4:d4:61:70:bd:04:61:6a:
12:8a:17:d5:0a:bd:c5:bc:30:7c:d6:e9:0c:25:8d:86:40:4f:
ec:cc:a3:7e:38:c6:37:11:4f:ed:dd:68:31:8e:4c:d2:b3:01:
74:ee:be:75:5e:07:48:1a:7f:70:ff:16:5c:84:c0:79:85:b8:
05:fd:7f:be:65:11:a3:0f:c0:02:b4:f8:52:37:39:04:d5:a9:
31:7a:18:bf:a0:2a:f4:12:99:f7:a3:45:82:e3:3c:5e:f5:9d:
9e:b5:c8:9e:7c:2e:c8:a4:9e:4e:08:14:4b:6d:fd:70:6d:6b:
1a:63:bd:64:e6:1f:b7:ce:f0:f2:9f:2e:bb:1b:b7:f2:50:88:
73:92:c2:e2:e3:16:8d:9a:32:02:ab:8e:18:dd:e9:10:11:ee:
7e:35:ab:90:af:3e:30:94:7a:d0:33:3d:a7:65:0f:f5:fc:8e:
9e:62:cf:47:44:2c:01:5d:bb:1d:b5:32:d2:47:d2:38:2e:d0:
fe:81:dc:32:6a:1e:b5:ee:3c:d5:fc:e7:81:1d:19:c3:24:42:
ea:63:39:a9

Step 6: Pulling it together

We will soon be using Mosquitto_pub and Mosquitto_sub to publish and subscribe to messages. I am going to assume you have these installed on your local environment.You can easily install this on a Debian based environment with the following command.

sudo apt install mosquitto-clients

For Windows and MAC, see the Mosquitto website. We now have everything we need for Mosquitto but publish and subscribe have different required parameters for publishing and subscribing.

What Mosquitto_pub needs:

mosquitto_pub \
-t "MQTT topic name" \
-i "pub_client" \
-u "username" \
-P "password" \
-h "host name" \
-V mqttv311 \
-p 8883 \
--cafile Baltimore.pem
-m '{"key":"value"}'

What Mosquitto_sub needs:

mosquitto_sub \
-t "MQTT topic name" \
-i "pub_client" \
-u "username" \
-P "password" \
-h "host name" \
-V mqttv311 \
-p 8883 \
--cafile Baltimore.pem

To publish a MQTT message to Azure, you can not just any topic name. It must be in the following format:

devices/{DeviceID}/messsages/events/

In my example Mosquitto is my DeviceID.

Step 7: Example Publishing and Subscribing

We now have all of the data and understand the required parameters to pass to both Mosquitto_pub and Mosquitto_sub. Now lets leverage these command line tools to build our completed commands and publish a message.

Substitute the values in the examples above with your data. My topic in this example is ‘devices/Mosquitto/messages/events/’. I am using ‘-d’ for verbose logging.

Mosquitto_pub:

mosquitto_pub -t "devices/Mosquitto/messages/events/" -i "Mosquitto" -u "Baldacchino-IOTHub.azure-devices.net.azure-devices.net/Mosquitto/?api-version=2018-06-30" -P "SharedAccessSignature sr=Baldacchino-IOTHub.azure-devices.net%2Fdevices%2FMosquitto&sig=O5Pt61EL3n1HLzt9G2%2FYJglpCk6m4I6XsbEu4WfnRoA%3D&se=1636984597" -h "Baldacchino-IOTHub.azure-devices.net" -V mqttv311 -p 8883 --cafile Balitore.pem -m '{"key":"value"}' -d

Client Mosquitto sending CONNECT
Client Mosquitto received CONNACK (0)
Client Mosquitto sending PUBLISH (d0, q0, r0, m1, 'devices/Mosquitto/messages/events/', ... (12 bytes))
Client Mosquitto sending DISCONNECT

Mosquitto_sub:

mosquitto_sub -t "devices/Mosquitto/messages/events/" -i "Mosquitto" -u "Baldacchino-IOTHub.azure-devices.net.azure-devices.net/Mosquitto/?api-version=2018-06-30" -P "SharedAccessSignature sr=Baldacchino-IOTHub.azure-devices.net%2Fdevices%2FMosquitto&sig=O5Pt61EL3n1HLzt9G2%2FYJglpCk6m4I6XsbEu4WfnRoA%3D&se=1636984597" -h "Baldacchino-IOTHub.azure-devices.net" -V mqttv311 -p 8883 --cafile c:\scripts\cert.pem -d

Client Mosquitto sending CONNECT
Client Mosquitto received CONNACK (0)
Client Mosquitto sending SUBSCRIBE (Mid: 1, Topic: devices/Mosquitto/messages/events/, QoS: 0, Options: 0x00)
Client Mosquitto received SUBACK
Subscribed (mid: 1): 0

We can validate the message has been received via Azure IoT Hub by monitoring incoming events on the end-point:

az iot hub monitor-events --hub-name Baldacchino-IOTHubStarting event monitor, use ctrl-c to stop...{"event": {"origin": "Mosquitto","module": "","interface": "","component": "","payload": "'{key:value}'"}}

Conclusion

We just walked through how you can use Mosquitto Client Tools (mosquitto_pub / mosquitto_sub) to publish and subscribe MQTT messages to Azure IoT Hub and you did it not via the Azure Portal but via Azure CLI :flexed_biceps::flexed_biceps:. Whilst this is a baby step, it is the step we need to take before we can look at MQTT broker (the Mosquitto daemon) to Cloud (Azure IoT Hub) replication.

MQTT is the de-facto IoT protocol for devices, from Alexa through to lights and locks and devices on the factory floor. It is as close to an IoT protocol standard that we have today. Azure IoT on the other hand is your gateway in to the world of Cloud. The Azure Cloud platform contains more than 200 (and growing) products and services designed to help you bring your solutions to life, solve today’s challenges and create the future. Join me in part 2 of this multi-part blog series where I will be extending upon what we have just built and will look at broker to Cloud communication before feeding the thousands of events that occur in my house daily in to Azure IoT Hub and beyond.

Think-big and happy automating.

Shane Baldacchino

Posted at https://sl.advdat.com/3j7YLy2

Friday, October 15, 2021

Mosquitto Client Tools and Azure IoT Hub. The beginning of something good....