Home/Success Stories/ETL Transformation: From Zero Visibility to Enterprise-Grade Observability
Enterprise ITMulti-phase engagementEnterprise

ETL Transformation: From Zero Visibility to Enterprise-Grade Observability

A large enterprise had no centralised monitoring, no log management, and no insight into the state of its internal processes. We designed and delivered a full Elasticsearch platform on OpenShift — with 50+ Logstash pipelines, HA clustering across 4 TB of data, and alerting tunable in seconds.

ElasticsearchLogstashKibanaRed Hat OpenShiftECKCI/CDAPMSynthetics

Key Results

50+
Logstash pipelines in production
4 TB
Data covered across 3 HA clusters
50+
Alert rules, fully customisable
Problem

No centralised monitoring, no log management, and no visibility into critical business processes

The client was operating a large, distributed information system with no centralised tooling to understand its state. Log data was scattered, unstructured, and practically unusable. When something went wrong inside a business-critical process, the team had no reliable way to detect it, diagnose it, or act on it quickly.

Beyond the immediate operational risk, the absence of structured log management created compliance and legal exposure. There was no single view of the data estate, no auditability, and no baseline from which to measure the health of the organisation's core systems.

No centralised monitoring — each team had its own fragmented tooling or none at all

No log management system — logs existed but were unstructured and unsearchable

No visibility into internal process state — failures were discovered reactively, not proactively

Unclear scope: the size of the problem in terms of data volume and teams involved had never been assessed

Legal and compliance requirements unmet — no auditability or retention framework in place

Business continuity risk was unquantified and unmanaged

Solution

A CI/CD-driven Elasticsearch platform on OpenShift, built to grow with the organisation

RNX team designed and delivered the full platform from the ground up. Elasticsearch, orchestrated via ECK on Red Hat OpenShift, formed the core — selected after a rigorous analysis of sizing requirements, compliance needs, and long-term operational fit. On top of this, we built an ETL/ELT platform in Logstash with dozens of independent pipelines, each tailored to a specific log source or data stream.

The entire solution was built CI/CD-first: pipelines, configurations, and alert rules are version-controlled and deployable without manual intervention. This gave the client the ability to iterate, extend, and tune their observability stack independently — without relying on external consultants for every change.

1

Discovery & sizing

Thorough analysis of data volumes, log source inventory, team structure, and legal/compliance requirements to establish the correct platform architecture

2

Core platform delivery

Elasticsearch deployed on Red Hat OpenShift with ECK orchestration; 3-cluster HA topology covering 4+ TB of data

3

ETL/ELT pipeline build

50+ Logstash pipelines implemented gradually — covering relational DBs (DB2, MSSQL, PostgreSQL), 50+ distinct log types, and custom enrichment pipelines for process state and connection tracking

4

CI/CD integration

Full pipeline and configuration lifecycle managed through CI/CD — enabling rapid, safe deployment of changes without manual intervention

5

Alerting & access control

50+ alert rules created with fully customisable thresholds; role-based access control implemented across all dashboards and data streams

6

Visualisation & advanced monitoring

Public dashboards for company-wide IS health; dozens of private dashboards for data teams; APM and Synthetics monitoring added as the platform matured

Output

Instantaneous action on real-time data and a platform that keeps growing

From a standing start with no observability tooling, the client now operates an enterprise-grade platform capable of processing and acting on large data volumes in real time. Alert thresholds can be tuned in seconds. New log sources are onboarded through the existing pipeline framework without disruption. The platform has already expanded beyond its original scope — APM and Synthetics monitoring were adopted organically as teams discovered new value in the data.

50+
Logstash pipelines covering DBs, logs & enrichment
50+
Log types parsed and evaluated
50+
Alert rules with per-rule custom tuning
4 TB
Data under management across 3 HA clusters

Your environment

Ready to see similar results?

Let's talk about your observability challenges. Free consultation, no obligations.

Book a Free Consultation