A unified data management platform
From days to minutes: one of the world’s top-five insurance companies has improved its end-to-end delivery of data thanks to cloud services
Many sub-companies based on different data management systems, heterogeneous technologies and several days of delay to deliver data to the business.
Centralization of data information, while reducing Process Cycle Time.
Amazon Cloud Services
Significant reduction of the end-to-end delivery of data from several days to minutes (the original pipelines based on batch processes) through AWS services.
* * *
When one of the world’s top-five insurance groups asked Agile Lab to contribute in designing and develop the new data management platform, it was clear the complexity that it was going to be achieved starting from the initial situation: many sub-companies based on different data management systems, heterogeneous technologies and several days of delay to deliver data to the business.
Agile Lab provided its technical skills in Cloud and Big Data technologies, collaborating with the internal team of the customer to compose a unique and top-notch team. Given the complexity of this challenge, the big point was picking the right composition of services to address all the requirements.
Data from any source
AWS provides a plethora of tools for data ingestion and integration. With the will to collect data for near real-time scenarios, Amazon MSK has represented a baseline for any kind of data ingestion, enabling the data platform to gather data from legacy operational systems through CDC and be open to other interfaces, such as data APIs and batches, easily reaching TB of data volumes.
Data exploration and business models
Building a landing area on top of AWS S3 has provided the opportunity for data exploration by means of AWS Athena. On the one hand, data exploration enables data analysts to understand and analyze data and build business models. On the other hand, AWS Glue fits the need for industrialization of business models, since big data engineer can build Spark applications based on data analysts' results as specifications. Further, AWS Glue provides great and transparent horizontal scaling, zeroing the time for operations.
Master Data and KPI
This initiative has dealt with many challenges, among the others: how to centralize master information keeping the near-real-time requirement in mind. Having AWS MSK as an entry point, it is possible to expose a mechanism that unifies data streams into a company’s standard format that can be stored to AWS Aurora. This addresses data access in terms of performance and provides interesting integrations with AWS Lambda to extend the scope of a database transaction. For instance, this case has considered the opportunity to integrate AWS Elasticsearch to empower search capabilities. Thus, having centralized and standardize data, it is easier to summarize into analytics and KPIs what is required for such an insurance company for strategic purposes.
The big achievement can be experienced through a significant reduction of the end-to-end delivery of data from several days (the original pipelines based on batch processes) to minutes through AWS services. This meaning that a customer can see the results of his/her operations as soon as they have been requested rather than struggling several days to get the evidence of those operations.