The problem with watching a network by hand
The client, a large telecom group that is not named publicly, needed to understand how data was moving across multiple communication channels. Network quality shows up in small measurements: how long a packet takes, how much that delay varies, and how many packets never arrive.
Tracking those measurements manually does not hold up. There is too much data, it never stops arriving, and a figure copied by hand can be wrong by the time someone reads it. Engineers can also spend their time gathering numbers rather than working out what the numbers mean.
The group wanted monitoring to happen automatically, with results presented in a form that engineers could read quickly when they needed to find a performance issue.
What we knew before we built
No before-and-after baseline is published for this project, so this page does not offer an improvement figure. What can be stated is what the scripts were set up to do.
- 4
- Objectives: automate monitoring, reduce manual tracking, give real-time insight and support decisions
- 5
- Parameters to record: packet flow, jitter, latency, volume and packet loss
- 2
- Directions to monitor on selected channels: incoming and outgoing packets
That set three requirements for the work. Capture had to run continuously without someone starting it. Measurements had to be stored in a form that could be queried later. And results had to be visual, so that a problem could be spotted without reading raw data.
The approach we chose
We kept the solution simple: modular Python scripts, each with a clear job, writing to a structured database. There was no need for a large platform to answer this problem.
- Modular scripts. Core Python scripts built for scalability and reuse.
- A database layer. Database interaction layers provide structured logging to PostgreSQL.
- Plotting. Python plotting libraries turn stored metrics into time-series graphs.
Monitoring data moves through four stages:
- Capture. Scripts continuously collect live transmission parameters on selected channels.
- Analyze. Delays, losses and jitter fluctuations are detected, and transmission efficiency and throughput are calculated.
- Store. Every metric is recorded in PostgreSQL, so that engineers can query history and trends.
- Report. Graphs are generated for daily, weekly and custom date ranges.
What we built
Our contribution was the automation scripts and the backend processing behind them.
Automated data capture
Scripts continuously collect live network transmission parameters, monitoring incoming and outgoing packets across selected network channels.
Performance analysis
The analysis detects packet delays, losses and jitter fluctuations, and calculates transmission efficiency and data throughput.
Graphical reporting
Raw data is converted into visual graphs that are simpler to interpret, with support for daily, weekly and custom-range reports.
Storage and retrieval
All recorded metrics are stored in a structured PostgreSQL database, which allows engineers to query historical and trend data easily.
Engineers should spend their time on what the numbers mean, not on collecting them.
The guardrails around monitoring data
People make decisions about a network based on this data, so it has to be consistent and easy to check. These choices were part of the build from the start.
- Structured logging. Database interaction layers write metrics in a consistent structure, rather than leaving them in loose files.
- Less manual entry. Automated capture removes hand-copied figures, a common source of error.
- History kept for checking. Stored metrics let engineers go back and check a trend rather than rely on a single reading.
- Engineers stay in charge. The scripts report what they measure. Decisions about the network remain with the engineers.
What was reported
The project delivered automated capture, analysis, storage and graphical reporting of network transmission metrics. The effects below are what the project team reported; none of them is a measured result:
- Workload. Manual workload and human error in network monitoring were reported to have reduced. No figure is published here.
- Issue detection. Real-time detection of issues was reported to support network reliability. No reliability figure is published here.
- Reporting. Reporting was reported to be more streamlined for technical teams and management.
- Decisions. Historical performance data gives teams more to work with when making decisions.
What stays hard
Four challenges come with this kind of monitoring, and any team building it should plan for them.
- Running continuously. Scripts that are expected never to stop have to be efficient, and any interruption leaves a gap in the record.
- Signal and noise. Jitter and latency fluctuate all the time, so the hard part is telling a real problem from normal variation.
- Growing history. Continuous metrics build up quickly, and historical queries have to stay fast as they do.
- Two audiences for one report. Engineers want detail and management wants a clear picture, and the same data has to serve both.