Advanced Data Solutions : Solving Sensor Data Quality with Verified Telemetry

Importance of sensor data quality

Data from IoT devices is used to continuously make decisions that can have a wide-reaching impact. The quality of the data used in this decision-making needs to be trusted, to avoid a garbage-in / garbage-out scenario. Today, IoT devices are constructed with increasingly low-cost components and are often deployed in harsh environments, which can impact their reliability.

Data reliability drives the quality of insights derived, which in turn drives the quality of decisions made. Our customers across multiple domains have highlighted several non-trivial challenges in tackling this problem, and current approaches for increasing the quality of data do not scale and have several limitations. For instance, replicating sensors multiplies sensor cost, manual intervention to detect and fix faults is expensive and often infeasible, and finally data-centric approaches can be ineffective as faulty sensors can often mimic a working sensor.

Consider a faulty water flow sensor used to control the flow in a water reservoir, a faulty soil moisture sensor in an agricultural farm used to control irrigation, or a faulty temperature sensor used to monitor the conditions of goods in transit. In all such cases, faulty sensor data could lead to cascading ill effects and catastrophic decisions. Yet techniques to detect sensor faults efficiently and assess data fidelity are limited.

The solution: Verified Telemetry

Verified Telemetry is a comprehensive solution to establish the health of the sensors, and therefore the quality of the data drawn from those sensors, on IoT devices. Verified Telemetry achieves this by calculating a “sensor fingerprint”, a set of unique electrical characteristics that differ between a working and a faulty sensor. For more details on sensor fingerprinting, please see Project Dependable IoT.

Verified Telemetry deploys the following steps (shown in Figure 1 below):

Calibrate: With the device in a known working state, a reference fingerprint is collected from the sensor and stored in the device.
Collect: During typical operation, when data is collected from a sensor, a corresponding operational fingerprint is collected from the device.
Detect: The reference and operational fingerprints are compared on the device; a match indicates the sensor is working, a mismatch indicates a fault.
Report: Both the sensor data and the sensor status are reported back to Azure IoT Hub.

Getting started

Verified Telemetry provides both Device and Solution samples to get you started.

Devices

The device samples are available for both Azure RTOS and FreeRTOS to provide the device builder with the best option for their scenario. Samples are available for MXCHIP, STMicroelectronics, and ESPRESSIF devkits. The device software stack, shown in Figure 2 below, provides an overview of the layers utilized to build the final device sample.

The Verified Telemetry Azure RTOS Middleware extends the Azure IoT Middleware for Azure RTOS:

The Verified Telemetry FreeRTOS Middleware extends the Azure IoT Middleware for FreeRTOS:

Solutions

A Grafana Sample Solution is available that demonstrates how to communicate with the device, collect fingerprints, and view data quality analysis in real-time on the sensor data.

The graph below shows the sensor data plotted on a time series. The green points indicate sensor is working as expecte and the red points indicate faulty sensor data due to a malfunction.

*Figure 3: Time series sensor data in Grafana*

Who should use this?

Verified Telemetry is available as a Public Preview v2. We are still hard at work on implementation, but we are inviting feedback from device and solution developers who are facing sensor data quality issues and are interested to understand if Verified Telemetry fits your needs. Verified Telemetry currently targets analog sensors and is available in both FreeRTOS and Azure RTOS SDK flavors, it's open-source and licensed under the MIT license.

What’s next?

The current Verified Telemetry SDK supports analog sensors, and we are working on an update to add support for digital sensors soon.

Further, while fault detection is one of the applications for Verified Telemetry we support today, we will be extending Verified Telemetry to cover additional scenarios such as automated sensor drift detection. Lastly, we are focused on providing a seamless experience with various other Azure IoT services with the aim of reducing the onboarding overhead.