Request a call
  • Hidden

Business Overview

Business Overview

Our client is a global education and technology company with 12 million students around the globe. Their portfolio comprises numerous educational platforms with various programs from elementary school to university students. 

With this number of users, all the platforms combined generated enormous amounts of data. The test submissions that needed to be processed and evaluated alone numbered millions daily. Hence the need to create a comprehensive solution that would automatically and in real-time assess student work, generate a variety of reports, and collect and store data in a data warehouse for future analytics and identifying business insights.

NIX had been developing software solutions for this client for many years. Knowing our experience in the education domain and our deep expertise in creating big data solutions, the client approached us with a request to develop and implement data as a service platform into their product ecosystem.

Data as a Service Platform for the Educational Ecosystem with 12 Million Students

Project Scope

This project consisted of three main components:

A data pipeline to export data both from the internal database and from external sources to the analytics system

The analytics core module, which directly checks and estimates a student’s work and then returns the result to the relevant education platform

A report processor that will generate numerous real-time and scheduled reports for students, teachers, administration, etc.

Solution

600h405 pic

Solution

Given that all of the client’s educational ecosystem had been deployed by AWS, the DaaS solution was also built on AWS, using the Spring framework, Spark jobs.

Our engineers created 155 different services for such a complex system, which ensured the functioning of all DaaS system components, namely data pipelines, analytics core, and report generating module.

33 Spring services
30 Spark Jobs
59 AWS Lambda functions
33 Amazon API Gateway

Data Pipelines

One of the fundamental phases of the project was to build data pipelines to collect the data for several purposes. First is the creation of historical storage. The second is providing an analytical  BI solution built on Snowflake and Looker with the required data.

Collecting data on user activity

This pipelining allows us to analyze the behavioral patterns of users when interacting with platform features and identify opportunities to improve them. To collect the data, our team used the Gateways API and Kinesis Firehose, which allowed us to build a pipeline with an average throughput of 40 GB per day, which is a solid number.

One more crucial pipeline performs data Export from MongoDB and S3 Bucket to the data warehouse. This pipeline works based on Spark, which exports data from the database and cloud storage and sends it either to create reports or to Snowflake, from where it goes for analysis and visualization.

Data as a Service Platform for the Educational Ecosystem with 12 Million Students

Analytics Core

Report Generation

The developed solution can generate 17 reports for different types of users (teachers, students, and school and university administrators), which allows for evaluating students’ success, identifying topics where students have the most significant difficulties, and providing insights to improve the process and the quality of learning.

 

Various reports require different amounts of information to be processed. For example, it may be a report on a single student or the progress of an entire class. Also, for some users, the relevance of the information is critical. Considering these factors, our engineers used several approaches to generate the reports. This also allowed us to optimize the cost and overall load on the system.

  • Real-time Reports
  • Scheduled Reports
  • Custom Report Models

Data Orchestration

As you can see, we used Jenkins for data orchestration and deployment of the entire environment. Long story short, our deployment process looks like this: 

AWS Configuration by Jenkins Analytics AWS tool to AWS Cloud

We developed custom templates for AWS Configuration, ensuring that only updated configurations and nothing else is deployed and Spark jobs run at specific times. Moreover, we created custom templates for services like S3, Lambda, Firehose, SQS, SNS, and many more.

Outcome

The developed AWS-based data analytics solution on top of Snowflake and Looker helped the client compose a holistic picture of its products and services. Moreover, it dramatically increased the value that the system provides to users. For example, real-time reports increase student user experience, which has become one of the competitive edges among educational platforms. In regards to other users: business insights from Big Data is a crucial tool for administrations for discovering even more options for service improvements; teachers using analytics can enhance course content i.e. use a more personalized approach to each student.

Data as a Service Platform for the Educational Ecosystem with 12 Million Students

Team:

7 experts (4 Java/Scala data engineers, System  analyst, UI designer,  Project manager)

Tech Stack:

AWS, Snowflake, MongoDB, Jenkins, AWS Kinesis, Apache Camel, AMQ, Elastic Beanstalk, S3, Kinesis Data Firehose, EMR, Fargate, CloudFront, Spring, Spark

Contact Us

Latest Insights

DevOps Automation: Best Practices and Benefits

Are you considering DevOps automation for your company? Let’s find out whether you need it or not. Check its benefits, steps, and best practices.

Offshore Development: Guide to Successful Collaboration

Offshore development allows businesses to leverage outsourcing to build high-quality solutions. Let’s explore how offshoring can empower you to create better apps.

How to Cut Costs Without Layoffs in the Face of Global Crisis

This article describes how your business can reduce operational and IT costs with help of tech solutions during a global crisis while maintaining business growth.

Data Migration: Strategy, Process, Types, and Key Steps

Data migration is the transfer from one operating environment to another, a process associated closely with infrastructure upgrades.

MongoDB Clusters: Strategies and Architecture

Clustering strategies help make data processing sustainable. Read and learn how to optimize the performance of a database system with MongoDB clusters.

Digital Marketing for Small Businesses: Benefits and Challenges

Not all business owners believe that digital marketing can help them improve ROI and target the right audience. Learn more about the reasons to invest in digital marketing campaigns from this post.

Multi-cloud Architecture: What Is The Best Strategy to Implement?

Do you consider the implementation of a multi-cloud architecture? Read this article to find out the best practices for it.

What is Salesforce Commerce Cloud and Why Might Your Business Need it?

Does your company need Salesforce Commerce Cloud? What are its specifics? Learn what benefits Salesforce Commerce Cloud can bring to your business.

Web Design Trends in 2023: Glimpses Into the Future

Keep updated on the 25 latest trends in website design that reign supreme today and are going to carry the day in the niche in 2023.