Thursday, October 14, 2021

Using Azure Percept to improve onsite workers safety

Here is a how to article where we explore the use of Azure Percept to improve workers safety and build a proof of concept that connects to a microcontroller to control an alarm. The article describes how to connect Azure Percept to Arduino, set firmware and demonstrates the use of .NET IoT in a docker container.




The importance of Worker Safety


According to OSHA, the most common causes of worker deaths on construction sites in America are the following

  1. Falls (accountable for 33.5% of construction worker deaths)
  2. Struck by an object (accountable for 11.1% of construction worker deaths)
  3. Electrocutions (accountable for 8.5% of construction worker deaths)
  4. Caught in/between (accountable for 5.5% of construction worker deaths)


Safety is very important on every job site. There are areas the workers must avoid. According to an article from Construction Dive: Elevator-related construction deaths on the rise


“While the number of total elevator-related deaths among construction and maintenance workers is relatively small when compared to total construction fatalities, the rate of such deaths doubled from 2003 to 2016, from 14 to 28, with a peak of 37 in 2015, according to a report from The Center for Construction Research and Training (CPWR). However, falls are the cause of most elevator-related fatalities, just like in the construction industry at large. More than 53% of elevator-related deaths were from falls, and of those incidents, almost 48% were from heights of more than 30 feet.“


Looking at these statistics, it’s being at the wrong place at the wrong time. Warning Signs are typically posted around the dangerous area but can be ignored or forgotten. 


What if we can use Azure Percept to monitor these dangerous areas and warn workers to stay away and avoid it. The goal is to have a deterrent, audible sound that reminds them about possible danger. Once worker leave the area and are not detected anymore, the audible sound stops automatically. Also during maintenance period, allow to remotely disable detection with authorizations needed to comply with regulations. 



Getting Started


We will look at the Hardware, Software requirements and Architecture diagram for this project. Then walk through step-by-step instructions on how to set up and deploy the application.



Azure Percept Device Kit


ELEGOO UNO R3 Super Starter Kit Compatible With Arduino IDE


Azure Subscription (needed for Azure Percept)


Visual Studio Code


Azure IoT Tools Extensions for VS Code


Azure IoT Edge Dev Tools


Docker with DockerHub account


Arduino IDE


Overal architecture



When the Camera detects a worker that is in the Dangerous Area, an alarm would sound and send telemetry messages to the cloud. Azure Percept Device Kit contains a people detection model that processes frames from the camera. When a worker is detected; it sends a message to the Arduino device to set the buzzer sound and sends the message to IoT Hub. 




Set People Detection Model

Once the Azure Percept Device Kit is connected to the cloud, we can specify to use People Detection Model by going to Vision Tab -> Deploy a Sample Model





Set up Arduino module

I followed the instructions on .NET IoT github page.


You can download the Firmata firmware that I used for this project at:


Here are the steps:

  • Open the Arduino IDE
  • Go to the library manager and check that you have the "ConfigurableFirmata" library installed
  • Open "Percept-Uno.ino" from the device binding folder or go to to create your own custom firmata firmware. Make sure you have at least the features checked that you will need.
  • Upload this sketch to your Arduino.

Send alert to arduino module

Open the Percept Edge Solution project in Visual Studio Code


This module can run locally on your machine if you have Azure IoT Edge Dev tool and also Azure IoT Tools Extensions for VS Code


To run it locally on your machine, you may have to change this module.json


Change repository to your Dockerhub username. 


"repository": "rondagdag/arduino-percept-module"


The Arduino Module is a C# application that controls the arduino device. It’s a docker container that uses .NET IoT bindings. Here are the
Nuget packages I used. 


    <PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.38.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="5.0.0" />
    <PackageReference Include="System.Runtime.Loader" Version="4.3.0" />
    <PackageReference Include="System.IO.Ports" Version="5.0.1" />
    <PackageReference Include="System.Device.Gpio" Version="1.5.0" />
    <PackageReference Include="Iot.Device.Bindings" Version="1.5.0" />   
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="5.0.0" />


The module tries to connect to two Arduino USB ports. I noticed that sometimes during reboot it can be either one of these ports that the arduino is connected to.





 In order to run this locally on your machine, you may have to change the port number to what Arduino IDE determines UNO is connected to (COM3 for Windows or /dev/ttyS1 for Mac)


It connects to 115200 Baud Rate.

The Buzzer is connected to PIN 12 shown below:


As messages are received from the IoT Edge Hub, it can be processed to detect if a worker has been detected. If the payload contains items in the NEURAL NETWORK node, then we can send the alert to Arduino.   


Here’s the code to analyze. 


It might be tricky to run this locally on your machine and may need some modifications. You may have to modify deployment template and the code to receive the payload coming from Simulated Temperature.


Notice how the ports are mapped on the docker container. /dev/ttyACM0 and /dev/ttyACM1


    "ArduinoModule": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "${MODULES.ArduinoModule}",
              "createOptions": {
                "HostConfig": {
                  "Privileged": true,
                  "Devices": [
                      "PathOnHost": "/dev/ttyACM0",
                      "PathInContainer": "/dev/ttyACM0",
                      "CgroupPermissions": "rwm"
                      "PathOnHost": "/dev/ttyACM1",
                      "PathInContainer": "/dev/ttyACM1",
                      "CgroupPermissions": "rwm"



Locally the EdgeHub mapping looks like this. The Simulated Temperature sensor is passing the output to the Arduino Module. Then the Arduino Module filters the data, controls the arduino and sends the telemetry up to the cloud.


"$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.2",
        "routes": {
          "sensorToArduinoModule": "FROM /messages/modules/SimulatedTemperatureSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/ArduinoModule/inputs/input1\")",
          "ArduinoModuleToIoTHub": "FROM /messages/modules/ArduinoModule/outputs/* INTO $upstream"
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200


You can test this out on the Simulator


Deploy setup

In order to deploy this to Azure Percept, IoT Hub device -> Set Modules -> Add -> IoT Edge Module


Fill in Module Name and Image URI.




Also the Container Create Options to map the correct port.



Specify the Routes. In this case we have to push data coming from Azure Eye Module To Arduino Module.




Here’s what the mapping would look like


"$edgeHub": {
            "properties.desired": {
                "routes": {
                    "AzureSpeechToIoTHub": "FROM /messages/modules/azureearspeechclientmodule/outputs/* INTO $upstream",
                    "AzureEyeModuleToArduinoModule": "FROM /messages/modules/azureeyemodule/outputs/* INTO BrokeredEndpoint(\"/modules/ArduinoModule/inputs/input1\")",
                    "ArduinoModuleToIoTHub": "FROM /messages/modules/ArduinoModule/outputs/* INTO $upstream"


Here’s an explainer video:





Keeping workers safe is very important in any job site. Azure Percept can help with people detection and connection with different alerting systems. We’ve demonstrated setting up the Azure Percept Dev Kit to use people detection. We’ve connected Arduino with a buzzer to trigger an audible sound. This actually enables Azure Percept Dev Kit to expand capabilities and broaden use cases. Let me know if this blog helped you in any way by commenting below, I am interested in learning different ideas and use cases. 



Reference Code:


Getting Started With Azure Percept:


Join the adventure, Get your own Percept here


Get Azure


Purchase Azure Percept

Available to customers – Build your Azure Percept

Ron Dagdag

Lead Software Engineer

During the day, Ron Dagdag is a Lead Software Engineer with 20+ years of experience working on a number of business applications using a diverse set of frameworks and languages. He currently supports developers at Spacee with their IoT, Cloud and ML development. On the side, Ron Dagdag is an active participant in the community as a Microsoft MVP, speaker, maker and blogger. He is passionate about Augmented Intelligence, studying the convergence of Augmented Reality/Virtual Reality, Machine Learning and the Internet of Things.


Posted at