Navigating the Cloud: Modernization of Healthcare Data Pipelines

By embracing a cloud-centric approach and optimizing data pipelines, NIX empowered healthcare and insurance providers with secure, and data-driven solutions.

Business Domain

Healthcare
Service

Data Engineering
Technologies

Python, Azure Data Factory, Azure Databricks

Business Overview

Business Overview

Our client, a prominent healthcare performance improvement company, offers SaaS solutions for acute and ambulatory care providers and academic medical centers. Collaborating with diverse healthcare entities, the company offers services in analytics, contract management, operational oversight, and supply chain management to enhance the delivery of high-value care, aligning cost, quality, and market performance.

The client’s on-premise solutions merge supply and clinical data for healthcare providers and insurance companies. This data covers diagnoses, procedures, medications, and medical equipment. It enables executives to enhance patient outcomes and reduce operational costs.

Their existing solutions were hard to maintain and required huge investments of resources to support. The client turned to NIX as a trusted vendor renowned for our extensive expertise in cloud services. The goal was to transition their on-premise system to a more scalable and cost-efficient architecture, while also ensuring a seamless migration of data pipelines.

Scope

Migrate existing system from on-premise to more scalable and cost-efficient cloud architecture that will increase flexibility and reduce costs

Ensure a smooth migration of data pipelines to the cloud architecture maintaining existing business logic and functionality

Improve the clinical data validation process for better accuracy, integrity, and compliance

Solution

The NIX team migrated the existing solutions from an on-premise Hadoop ecosystem to an Azure and Databricks technology stack. We also designed multi-tenancy architecture that is easy to scale, can resist peak loads, and ensures Infrastructure as Code (IaC) availability, while taking into account potential technical and business risks.

The NIX team opted to switch from the SAS programming language to Python, as Python is not only easier to maintain on Azure but also provides better cost-efficiency. The adoption of a single language for data engineering also proves to be more efficient. We integrated Octopus Deploy with Azure DevOps to provide a fully-automated build and deployment pipeline, improving resource utilization and ultimately leading to cost savings.

The migration concerned two solutions:

Financial reporting
The web platform collects statistical data on hospitals and allows financial departments to generate performance reports. This solution provides comprehensive reporting capabilities such as customizing the reports, comparing indicators with competitors, and identifying areas for further development.
Analytics for insurance calculations
Web application for clinics that connects procedural supply and clinical data to help healthcare providers understand the drivers of supply costs, utilization and quality of care. Clinicians enter data every time a patient comes for a visit and then this data is transferred for validation, processing and transformation. This solution aids in identifying opportunities from a variety of service lines and procedures to create actionable insights. Consequently, this data enables accurate insurance calculations and optimizes costs.

Data Pipelines Orchestration and Migration

Data Pipelines Orchestration and Migration

We orchestrated the migration of the client’s pipelines, shifting them from on-premises servers to the Azure platform and ensuring a smooth data exchange between databases and storages. At present, our efforts are focused on the transformation of PySpark scripts into Azure Databricks, alongside the development of pipelines within Azure Data Factory to execute these scripts effectively.

For orchestrating ETL/ELT pipelines and data enrichment processing, we leveraged Azure Data Factory and Databricks. Databricks Delta Lake was used as a main data warehouse and contains all the historical, procedural, clinical quality enrichment data. Azure Databricks serves as core for data processing and migration of existing Pyspark jobs.

On the Azure side, we also implemented table comparison service for comparing input/output data in Azure and on-premise by certain filter criteria to ensure uniformity.

Data Validation

Leveraging Apache Spark, we established and refined the processing and validation of hospital data, including patient personal information such as diagnoses, age, gender, payments, procedures, and prescription treatments.

Our team of data engineers devised validation rules to rectify discrepancies and implemented sophisticated data quality monitoring capabilities for clinics. This prevents the system from processing data files containing significant disparities, thereby ensuring superior accuracy, data integrity, and regulatory compliance.

Outcome

Outcome

Through the adoption of a cloud-centric infrastructure and the enhancement of data pipelines, NIX enabled the client to offer healthcare and insurance providers cost-effective, secure, and data-driven solutions, resulting in improved patient outcomes and reduced operational expenses.

The solutions leverage raw clinical and supply data to deliver valuable insights through advanced data visualization capabilities, all the while upholding robust data processing, strong security, and data integrity.