Automation, cloud migration, and cost optimization for a global tech company

Our client required efficient tools for handling vast amounts of data that was generated by their equipment. Virtually all of the company’s siloed data was managed on-premises, which incurred significant costs and made scalability quite problematic. Their sprawling IT resources caused issues with data accessibility, leading to inefficient data governance, storage, organization, etc. This data was essential for generating Monthly Service Reviews (MSRs) - equipment performance reports that were sent to clients every month. As a result, this process was inefficient and cost ineffective: it required lots of manual work and needed a paid third-party tool.

Together with the client's team, our experts have migrated the on-premise infrastructure from MS SQL Server to a unified data warehouse that we have built on GCP. Over 70 operational data sources, 4 data warehouses, and 1 data lake have been migrated and consolidated to Google Cloud. Over 20 servers have been decommissioned in the process, leading to significant savings on the client’s side. Additionally, our experts have ensured high data quality in the warehouse by locating and eliminating anomalies.

Our client has over 40 data centers around the world, managing hundreds of thousands of customers' virtual machines. Every few minutes they receive data from each piece of equipment (servers, routers, switches, etc.), such as CPU usage, memory and packet throughputs, etc. The client used a third-party tool to aggregate all of this information and manually create MSRs for their customers. To make this process more efficient and cost-effective, the client decided to replace the third-party tool with their own solution. The new solution would utilize GCP since the client is an official Google partner.

First, our team investigated the workflow of the third-party tool to learn how the data is retrieved, interpreted, and transformed. After a thorough analysis, we built a new solution that uses Google Cloud Dataflow to transfer data from Kafka to BigQuery. Since the client used different vendors on their servers to collect information (Cisco, IBM, etc.), we also needed to standardize and unify all data. Therefore, instead of using the standard ETL (Extract-Transform-Load) approach, we used the ELT method, where data is first extracted from Kafka, loaded into BigQuery, and only then transformed. This has enabled the solution to create comprehensible reporting tables with information about the performance of each server instance.

Our team has also helped the client streamline the generation of MSRs. The process involved lots of manual tasks and consumed nearly 17,000 work hours per year. N-iX has helped the client simplify this process by consolidating information to BigQuerry as a single location. Once the data was consistently available across the customer base, our team automated the manipulation of raw data into MSRs.

Together with the client's team, we have built a unified data warehouse in GCP, allowing the client to elevate data management across the business. Our team has created various new solutions on GCP and maintained the data quality of the migrated data warehouse, allowing the client to benefit in a variety of ways:

• Optimized costs by migrating from MS SQL Server to GCP and decommissioning over 20 servers;
• Streamlined data collection, management, and accessibility by centralizing and unifying all data in Google Cloud;
• Significantly reduced expenses by automating the MSR generation process that consumed nearly 17,000 working hours per year and required the use of an expensive third-party tool.