Our team has used AWS DMS to retrieve data from the client’s on-premise Oracle database in near-real-time mode. This has allowed our experts to receive CDC data (Change Data Capture) as soon as possible and build upstream tables based on all occurred changes. They have also used the already implemented Dynamics 365 as the second source of data, which is retrieved with the help of the Dynamics 365 REST API.
All data is transferred to AWS S3, where it moves through 3 layers: bronze (raw data ingestion), silver (filtered, cleaned, and transformed data), and gold (business level aggregates). The movement of data between these layers is attained through the use of Apache Spark, an analytics engine for data processing. Data from the business layer is also moved to Snowflake, a data warehouse, where it can be served to Power BI. Our experts can then use their extensive knowledge in Power BI to build informative Business Intelligence reports for the client’s customers.
N-iX team has also helped the client significantly improve their data security by establishing effective data governance. This data governance is separated into two parts, one being based on AWS, and another one - on Snowflake.