EDW Optimization for Insurance Company

Business Overview

Business Overview

The client is a full-service insurance and reinsurance company that offers different types of coverage to choose for third-party insurance agencies and platforms.

The company operates in several countries, and its enterprise data warehouse (EDW) stores historical data and reports from all of an organization’s business operations in a single, centralized platform acting as a backbone for data-driven decisions and operational efficiency. As more data was processed, the workload of the data hub infrastructure increased, and the need for scaling and optimization became a pressing matter.

The client approached NIX to join a distributed engineering team from different countries and redesign the architecture, making a difference using our proven experience with enterprise-level data solutions.

Challenge

Due to an outdated technology stack, the client’s EDW was barely scalable and sustainable, necessitating a large team, enormous funds, and computing power effort to maintain and support the large data flow coming from all operational servers. The engineering team spent almost all of their time maintaining the EDW rather than optimizing and developing new functionality.

The daily EDW ETL processes exceeded the available load window, and the processing of new data took more than 48 hours, thus operating with obsolete information. Furthermore, data governance was limited—the lack of foundational capabilities such as an enterprise data dictionary and data lineage hampered both delivery and consumption.

Project Scope

Our solution architect started by conducting an in-depth audit of the existing EDW architecture to develop the roadmap for enhancements. To help the client increase the reliability of data and build a single source of truth, the NIX project scope included the following activities:

1

Optimize and enhance the on-premise enterprise data warehouse (EDW) operating with up-to-date data
2

Reduce report generation time and set up regular automated report submission to streamline the department’s operations process

Solution

EDW Optimization
Report Management Enhancement

EDW Optimization

The NIX team managed to reduce the load of the EDW and stabilize system operations by reinforcing the data management and optimizing ETL process. To improve the overall efficiency of an ETL process, we loaded data incrementally, broke down large tables into small ones, and used parallel processing.

We used an online analytical processing (OLAP) cube—a data structure that is geared to handle multidimensional data coming from the data warehouse and derive insights.

Report Management Enhancement

We set up the process of automatic report generation by making necessary changes in OLAP cubes—subsets of the data warehouse in multi-dimensional cubes provide a source of fast ad-hoc analysis for self-service users via drag and drop functionality.

For analytics, we updated the existing Microsoft SQL Server’s Analysis Services to SSIS SSAS—an online analytical processing (OLAP) and data mining and reporting tool used in business intelligence.

The NIX team also pioneered improved report management with custom calculations, automation capabilities, and more, for various departments within the client’s company:

Pre-built reports—standard formatted reports regularly used by many consumers within the organization
Ad Hoc Reports—user-generated reports to answer a specific question or present an analysis’s results
Self-served analytic reports—dynamic and interactive process of analyzing one or more sets of data with the purpose of answering one or more questions

Outcome

As a result, the client received the ability to obtain all necessary reports without delay, thereby allowing business stakeholders to make data-driven decisions using up-to-date data.

The EDW was optimized to handle high loads and can now process new data in less than four hours, which is 12 times faster than before the system optimization. Moreover, enhanced capabilities of the reporting system streamlined operations between departments and allocated more time for higher-value activities.