On-Prem Cloud Native Data Engineering

tl;dr

• Industry: Energy
• A production-ready data platform in a high-security environment

Situation: Significant preliminary work, but no finalization

To reduce dependencies and accelerate delivery, a task force from the responsible team at the client company developed a staging environment for a data platform as a proof of concept. The platform runs solutions for transforming, processing, and analyzing data related to power grid operations. These solutions serve, among other things, for integration with other systems and external partners. The solutions run in an isolated OT environment.

Challenge: The Difficult Last Mile

The goal of the project was to combine the existing individual components into a production-ready environment.

Solution: Step by Step, Together

• Development of a production-ready setup using Ensemble Programming in MS Teams sessions.
• Design of a robust architecture using standard components with minimal customization.
• Application of cloud-native design patterns to on-premises installations.
• Development of a dedicated PKI for deploying certificates for end-to-end encryption and mutual trust between components (mTLS).
• Application of high-availability patterns (clusters, active-passive replication, application load balancers, network load balancers, virtual IP).
• Separation of the components into stateless services and persistence storage services.
• Setting up a geo-redundant installation across multiple on-premises data centers
• Professionalizing the deployment automation of the Ansible setup
• Creating technical documentation, including readiness runbooks and architecture decision records
• Conducting a team workshop for knowledge transfer with the entire team

Technologies

• Apache Airflow
• Python
• MinIO S3
• PostgreSQL
• Patroni
• etcd
• Redis
• RabbitMQ
• nginx
• haproxy
• squid
• VRRP
• ansible
• OpenSSL
• CNCF Distribution Registry
• GitLab
• Citrix
• Miro
• Jira
• Confluence
• DrawIO