Home/Success Stories/From Raw JSON to Kibana: Automated Weather Data Pipeline in Brno
Environmental MonitoringOngoing engagementOpen Data Project
From Raw JSON to Kibana: Automated Weather Data Pipeline in Brno
Monitoring real-time environmental metrics requires a clean visualization layer backed by a resilient data pipeline. This project demonstrates how automated data collection, Elasticsearch indexing, and Kibana dashboarding come together to monitor meteorological conditions in Brno, Czech Republic.
ElasticsearchKibanaPythonCHMI APICron
Key Results
1h
Automated end-to-end ingestion cycle
4
Key meteorological metrics tracked in real time
10 min
Weather station measurement granularity
At a Glance
The Challenge
Raw hourly JSON feeds from the Czech Hydrometeorological Institute had no automated ingestion, no schema normalization, and no visualization layer.
The Pipeline
A custom Python script runs on a cron schedule, fetches the CHMI JSON payload, transforms it into a standardized format, and ingests it directly into Elasticsearch.
The Outcome
A fully automated Kibana dashboard delivering live temperature, precipitation, humidity, and wind metrics for Brno — updated every hour, hosted on Azure.
The Brno Weather Dashboard Overview
Monitoring real-time environmental metrics requires a clean visualization layer backed by a resilient data pipeline. This project demonstrates how automated data collection, Elasticsearch indexing, and Kibana dashboarding come together to monitor meteorological conditions in Brno.
At a glance, the top section of the Kibana dashboard offers a quick overview of current weather conditions in Brno. It tracks key meteorological phenomena, including temperature, precipitation, air humidity, and wind speed and direction.
The sidebar provides high-level summary metrics, such as minimum, maximum, and average values, calculated across all monitored variables. Below the overview, detailed time-series charts illustrate how these metrics evolve over time, while integrated benchmarks compare local measurements directly against the Czech national median.
The lower panels of the dashboard extend the view beyond temperature. The air humidity chart plots Brno's actual humidity against the Czech national median, revealing how local conditions diverge from the national baseline across the day. The wind speed panel follows the same comparative pattern — tracking both Brno's measured values and the national median in a shared time frame.
At the bottom, the pipeline health panel ('Stav datové pipeline') confirms the ingestion rate, showing a steady ~2,500 records per hour being processed reliably throughout the day. A flat, consistent line here is a signal of a healthy, uninterrupted pipeline — exactly what a production-grade observability stack should look like.
Dashboard Deep Dive: Detail Analysis Rainfall and Wind Dynamics
A second dashboard takes the analysis a level deeper moving beyond headline weather statistics to expose the structural relationships between rainfall and wind dynamics across a rolling 7-day window.
The centrepiece of this view is a dual-axis time-series chart that overlays 10-minute rainfall bursts against real-time wind speeds, making storm-front signatures immediately visible as correlated spikes across both axes. Alongside it, a wind direction donut chart breaks down dominant airflow patterns over the period — with prevailing westerly and northwesterly flows accounting for the majority of observations. A cross-metric scatter panel then links precipitation totals directly to specific wind direction categories and Beaufort force ranges, revealing a consistent pattern: measurable rainfall in Brno occurs predominantly during lighter wind conditions rather than at peak gust speeds.
The temporal dimension is covered by diurnal wind intensity heatmaps, which aggregate wind speed by hour-of-day across the full 7-day window. The resulting pattern is clear and repeatable: speed ramps up between 09:00 and 20:00 before settling into a calm overnight baseline. This kind of time-bucketed aggregation — applied here to meteorological data — is the same pattern RNX uses to build operational intelligence for enterprise infrastructure: converting raw, high-frequency event streams into structured, human-readable signals that expose what the raw numbers alone cannot.
Data Sourcing & Ingestion
The raw data is sourced from the Czech Hydrometeorological Institute (CHMI) website, which refreshes its public data feed every hour. On-site weather stations record measurements at finer 10-minute intervals.
To process this data automatically, a streamlined four-step pipeline runs without manual intervention:
1
Schedule
A custom Python script runs on an automated hourly schedule via a cron job.
2
Fetch
The script fetches the raw JSON payload directly from the CHMI source.
3
Transform
It transforms the unstructured data into a standardized format optimized for search and analytics.
4
Load & Display
The transformed records are ingested directly into an Elasticsearch database, while Kibana renders the interactive visualization layer.
Technical Foundations: The RNX Approach to Data Infrastructure
This project demonstrates the core technical principles, pipeline architectures, and visualization techniques engineered by RNX.
1
Unified Tech Stack — Elasticsearch & Kibana
Elasticsearch handles the indexing and rapid retrieval of all incoming weather records. RNX specializes in building, tuning, and scaling Elasticsearch clusters for production environments. Kibana transforms database records into actionable dashboards — RNX designs custom Kibana environments that turn complex metrics into clear, intuitive visual interfaces.
2
End-to-End Ingestion Pipelines
Automated JSON extraction, hands-off cron scheduling, schema normalization, and direct Elastic ingestion. This structure reflects the log and metric ingestion pipelines RNX designs for enterprise clients, ensuring system logs and business metrics are processed reliably without manual intervention.
3
Time-Series Aggregations & Metrics
The dashboard focuses on time-series tracking, parameter aggregation (min/max/average), and comparative baseline modeling. RNX applies these exact methodologies to enterprise infrastructure — converting raw system logs, APM traces, and operational metrics into actionable performance indicators.
Whether tracking localized meteorological phenomena or monitoring complex cloud infrastructure, the core engineering challenge remains the same: turning raw, distributed data into clear visual intelligence.
Summary
By combining lightweight ingestion scripts with the analytical power of Elasticsearch and Kibana, this project highlights how modern data pipelines bring clarity to complex time-series metrics. The Brno weather dashboard is a concrete example of the pipeline architectures and visualization patterns RNX engineers for production environments — scalable, automated, and built to run without interruption.
The same core patterns used here — automated ingestion, schema normalization, time-series aggregation, and comparative benchmarking — are directly transferable to enterprise log management, APM, and infrastructure monitoring use cases.