Scalable big data analytics platform for leading industrial supply company

Being a large-scale industrial supply company, our client needed to efficiently manage large amounts of data, including the data on inventory-related costs. Therefore, the company decided to extend its data warehouse solution which collects data from multiple departments.

For the development and support of the solution, our client hired on-site consultants. However, that caused significant overhead, and the company was looking to reduce these operational costs. Also, our client wanted to migrate the solution to the cloud to make it more scalable and cost-efficient.

Therefore, to boost their engineering capabilities, further support the solution, and optimize costs, the company needed a reliable offshore development partner with long-standing big data expertise. As a result of the rigorous selection process, our client chose N-iX.

To migrate from on-premise Hadoop Hortonworks cluster to AWS and allow processing additional data in AWS, the N-iX team built an AWS-based big data platform from scratch. Also, we have been involved in extending and supporting the existing Teradata solution. Teradata is used to collect data from other systems and further generate reports with Business Object and Tableau. The data sources are MS SQL, Oracle, and SAP.

To choose the data warehouse design and the tech stack that fit our client’s business needs, our specialists created a proof of concept. We compared Amazon Redshift with Snowflake and preferred Snowflake as it met the client’s approach of cloud neutrality: it can easily scale up and down any amount of computing power for any number of workloads and across any combination of clouds.

The whole development process is cloud-agnostic and is designed to ensure that the client can easily change the cloud provider in the future. For example, we use Terraform as it is compatible with all cloud vendors - AWS, Azure, and Google Cloud.

The project is in the development phase. Currently, our specialists are in the process of building the environment that will be able to process large datasets. The use of Snowflake and Airflow technologies allows us to automate the data extraction process. Also, Snowflake minimizes data duplication by checking whether the ingested files have been already processed or not.

After half of the year of working on the data pipeline unification, we managed to integrate more than 100 different data sources into a unified data platform. This includes daily data loads, along with a backfill of historical data. We are working with TBs of data tables, and the size is growing.