The problem with devices you cannot see
The client, a global energy company that is not named publicly, had a large network of industrial and consumer devices. A device in the field is only useful to the business if someone can reach it, see how it is behaving and act on what it reports.
Doing that device by device does not scale. People need to set devices up, watch them and update them remotely, and the data they send has to arrive somewhere it can be used.
The goal was to connect these devices securely and turn their data into something teams could use, for operational efficiency, predictive maintenance and a better experience for customers.
What we knew before we built
No before-and-after baseline is published for this project, so this page does not offer an improvement figure. What can be stated is what the system set out to do.
- 4
- Objectives: connect devices, process real-time data, give useful insight and keep interactions reliable and secure
- 2
- Audiences for the insight: internal teams and end users
- 3
- Things to do remotely: provision, monitor and update firmware
That set three requirements for the work. Communication between devices and the cloud had to be secure. Data had to flow continuously and arrive clean enough to analyze. And the system had to have room for more devices and more data.
The approach we chose
We built around AWS managed services and event-driven serverless flows, so that processing happens when a device has something to report:
- AWS IoT Core. Device registry and MQTT-based communication.
- Python on AWS Lambda. Event-driven functions that process incoming data, exposed through API Gateway.
- Managed storage. AWS S3 and DynamoDB or RDS to hold device data.
- AWS CloudWatch. Monitoring and diagnostics for the system itself.
- TypeScript. The dashboards and interfaces people use.
Device data moves through four stages:
- Connect. Devices communicate with the cloud through AWS IoT Core over secured connections.
- Process. Streaming pipelines and serverless functions clean and process the data automatically.
- Store. Processed data is written to managed AWS storage.
- Act. Dashboards show usage trends, and alerts flag critical behavior or breached thresholds.
What we built
Our contribution was the cloud backend, the data pipelines and the interfaces on top of them.
Device connectivity and management
Devices communicate securely with cloud systems, and can be provisioned, monitored and given firmware updates remotely.
Real-time data pipelines
Streaming pipelines built on AWS managed services take in device data, with automated cleaning, processing and storage workflows.
Analytics and alerts
Performance dashboards show usage trends, and alerts and notifications flag critical device behavior or breached thresholds.
A cloud-native architecture
Infrastructure built from AWS services supports additional devices and heavier data loads as the network grows.
A connected device only helps the business once someone can see what it is telling them and act on it.
The guardrails around connected devices
Devices in the field can be reached remotely, which makes secure communication and careful access essential. These decisions were part of the architecture from the start.
- Encrypted device communication. Devices communicate with the cloud over TLS.
- IAM-based access. AWS IAM controls which services and people can reach which resources.
- A device registry. AWS IoT Core keeps a registry of devices, rather than leaving them unaccounted for.
- Monitoring and diagnostics. AWS CloudWatch watches the system itself, so that problems in the platform can be found too.
- People act on alerts. Alerts tell the responsible teams about critical behavior. The response stays with them.
What was reported
The project delivered device connectivity, real-time data processing, dashboards and alerts on AWS. The effects below are what the project team reported; none of them is a measured result:
- Visibility. Teams were reported to have a clearer view of device performance and usage trends.
- Operations. Automation and central management were reported to reduce operational overhead. No figure is published here.
- Decisions. Data is intended to support decisions across maintenance and customer services.
- Scale. The design is intended to support large device numbers. No device count or load figures are published here.
What stays hard
Four challenges come with this kind of system, and any team building one should plan for them.
- Devices that drop off. Connections in the field are not always reliable, so the system has to cope with devices going quiet and coming back.
- Firmware updates at a distance. A bad update is hard to undo when nobody is standing next to the device.
- Data quality. Sensor data can be noisy or incomplete, and it has to be cleaned before anyone relies on it.
- Useful alerts. If alerts fire too often, people stop reading them, so thresholds have to be set with care.