End-to-end Development of a Big Data Platform for Gogo – a Leading Inflight Internet Provider

Client background

Gogo has been a North American market leader in in-flight connectivity for over 20 years. It serves more than 2,900 commercial aircraft and over 6,600 business aircraft. The company is constantly working on enhancing its technology and customer experience.

Business challenge

Gogo needed a strong team of software engineers with expertise in Big Data, Cloud Solutions, Data Science, and BI. The client wanted to migrate their not scalable expensive on-premise data solutions to the cloud, make them scalable, and improve the customer experience based on the big data analysis results.

Transformation

N-iX team has performed a complete transition of Gogo solutions to the cloud and built a unified data platform. We’ve helped the client to optimize operations, completely rebuilt the solution using open-source technologies, such as Spark, streamlined the system of predicting failures and replacing devices, and built several highly efficient analytics platforms.

Value delivered by N-iX

  • Expanding the client’s development capabilities by gathering a strong team of big data engineers and data scientists;
  • End-to-end development of the cloud-based platform which collects data from more than 20 different sources;
  • Analyzing and finding reasons for ill-performance and equipment failures, reducing the number of not-fault-founds by 8 times.
Location
United States, Chicago
Partnership period
January 2017 - present
Team size
30
Team location
Ukraine, Lviv
Expertise delivered
Business Intelligence Data Science Big Data Data Engineering
Technologies
Scala Python AWS, Spark, Hadoop, Tableau, R, Pig, Hive, SQL

PRODUCT OVERVIEW

Client’s goals

More than 2,900 commercial aircraft and 6,600 business aircraft are equipped with Gogo’s Wi-Fi services. To ensure flawless operation of all the systems and the best speed of the in-flight Internet, the company needed to collect and analyze huge amounts of data. This includes structured data like system uptime and latency and unstructured data like the number of concurrent Wi-Fi sessions and video views during a flight. However, the data they gathered was stored on-premise in many different sources. So the client’s main goal was to increase the capabilities of their in-house engineering team to build a unified data platform in the cloud. This would enable them to process their big data more efficiently and enhance customer experience based on the analysis results.

Gogo partnered with N-iX to perform the following tasks:

  • Transition multiple data solutions to a new cloud-based data platform;
  • Build the data warehouse system for storing and processing significant amounts of data;
  • Perform Data Science and present insights as actionable BI reports;
  • Improve operations and customer experience by applying machine learning and data science to user behaviour and flight information;
  • Provide 24/7 operational team and enable uninterrupted performance of the client’s product (since the team is located in Ukraine, we are able to provide support during our working day while it’s night in the US).
Challenges
  • Changing ETL tools and MapReduce code with new more efficient Spark applications;
  • No documentation of the source code and processes so we needed to analyze the old code and rewrite it;
  • A need to understand the very specifics of the equipment, how it interoperates with other devices and whether it works well on a specific type of planes or under certain conditions;
  • Processing and analyzing vast amounts of geospatial data;
  • Developing the solution for log parsing to process various kinds of data streamed from aircraft in real time;
  • Defining the relationship among myriads of different devices (modems, antennas, WAP points) and finding a reason for a particular technical hitch.
Implementation

Migration to the cloud, development of a big data platform

To enable the processing and storage of a considerable amount of data, we’ve performed a complete migration of the solution to the cloud (AWS) and built a data warehouse system. Our engineers have made an end-to-end delivery pipeline: from the moment when the logs come from the equipment to the moment when they are entirely analyzed, processed, and stored in the data warehouse. The system is centralized, and all the data is taken from one source.

To implement the solution, our team used such technologies as AWS, EMR, Hadoop, Apache Spark, Apache AirFlow, Kinesis, AWS Lambda, and others.

One of the project challenges was working with the source code. Yet the team was able to understand how the old code worked and delivered great product despite the lack of requirements.

Processing and analyzing geospatial data

In the scope of developing the data platform, we delivered a solution for processing geospatial data. It allows monitoring the quality of connection on different aircraft in real time. As the planes are flying in different parts of the world and at a different altitude, the data is streamed in real time and we can see which satellites they communicate with and all the related metrics. After processing geospatial data, the team built a solution for the maintenance engineering team, which provides a comprehensive picture of flight information and enables them to react to issues as soon as possible.

Log parsing using Spark

Another part of the platform our team has been working on is the solution for log parsing. Since a lot of data is streamed in real time from the planes, we take the logs from different equipment, parse them using Scala, and then use for data analytics.

Full integration of an extended team

Ever since we started our cooperation with Gogo, our team has grown from three to more than thirty people. The dedicated team is tightly integrated with the client’s internal team and works as a part of Gogo’s operational excellence department which ensures the quality of service. Our specialists have regular business trips to the USA to get a more profound domain knowledge, study the devices, learn what kind of data the onboard equipment generates, etc.

Value delivered by N-iX

N-iX successfully gathered and later scaled a strong dedicated development team of Big Data software engineers, data scientists, BI developers, QA specialists, and a Scrum Master to cover a large share Gogo’s Big Data analytics needs. Our engineers both supported the original environment and developed a new one – a unified data platform in the cloud. For one thing, this enabled our client to cut infrastructure costs. For another, the solution provided a single point of truth and unified huge amounts of data in one place. Therefore, it allowed our partner to process and analyze their data more effectively, run data science models, and ultimately, provide insightful reports to airlines and the company decision makers.

Summing up, we have achieved the following results:

  • Development of a data platform based on AWS that aggregates all the data from different sources using Spark;
  • Building a Data Lake for collecting data from all sources in one place;
  • Delivering a solution for measuring availability and in-cabin performance of various devices as well as analyzing the operation of WAP points and other equipment;
  • Development of a platform for analyzing and finding reasons for equipment failures or ill-performance of specific maintenance teams;
  • Improving the system of timely replacing devices and reducing the number of no-fault-founds by eight times since the new platform was implemented.

RELATED SERVICES

Connect with our experts
Get in touch