CONSORTIUM FOR INFORMATION SYSTEM

Client Context

Our Client is a Consortium of Public Entities operating in the ICT field as Innovation & Technological Partner for over 100 public institutions. 

It’s one of the main players in Italy’s Digital Transformation, certified by AGID as a Cloud Service Provider to offer Digital Services to the Public Administrations. 

Two notable moments in its long history are 1985, when it opened its AI and Advanced Statistics Laboratories, where the first DWH/BI applications for Public Institutions in Italy were created and 1991, when it became one of the few Italian Super-Computing Centers. 

Its mission is focused on the Social Value of Innovation; that is, to implement innovative services for public administrations – thereby helping them to be more efficient and modern, to reduce public spending and to simplify the lives of Citizens and Companies. With 1200 employees and over 40 years of experience, it operates in a wide range of public innovation areas, among which: Smart City, e-Mobility, Digital HealthCare, Smart Energy, Territory, Environment, Real Estate, Production Systems, e-Government, Smart Education, Cultural Heritage, and further ones. 

In order to promote the technical innovation of its Members, the Consortium performs a wide range of R&D activities in Italy and abroad. Working with many organizations, it achieves economies of scale, implying significant cost savings for theAdministrations, which can always rely upon high level professional skills.

The Consortium fosters, along with its specific Services and Projects, the adoption of the most innovative technology solutions, networks, services and methodologies. It promotes the Open Innovation paradigms and in particular Open Data, encouraging and supporting its Members in developing data-driven new services and public policies based on data-driven analysis. 

Challenge

Our Client has designed and implemented an open source, modular “Smart Data Platform”, compliant to IoT and Big Data reference standards, aimed to offer public and private actors a variety of tools to experiment and create innovative data-driven solutions and services. 

It’s a “self-service” PaaS enabling the development of applications based on Big Data, allowing aggregation of open or shared data (public or private) from the Internet of Things, Internet of People (i.e. social networks) and from public and private systems. It is managed through its own Cloud infrastructure, released in Open Source with EUPL (European Union Public License) and accessible on GitHub repository for reuse and/or new implementations. 

The platform provides selective sharing of the information collected (private/shared/open data), allows for self-service processing of data, enables the exposure of information (data and metadata) via API and provides Big Data Analytical tools. 

The information is collected in a “Big Data Storage” in compliance to Security and GDPR privacy standards. 

The platform is dedicated to all public and private entities interested in developing new applications based on the data available on the platform itself, in particular: 

• Companies, Start-ups, Research Centers, Universities and Developers in the world of Open & Big Data;  

• Public Institutions for the implementation of innovative services for data enhancement, integration of heterogeneous data and creation of new knowledge tools for decision support; 
• Public/Private Actors using the open/shared data available on the platform within their applications, e.g. experimenting algorithms to create new knowledge, new data and new services. 
   

The Smart Data Platform can support multiple Digital Ecosystems, in terms of Communities of public / private actors interacting and exchanging information and services in order to develop their business objectives by exploiting the Value of Data. 

Each Actor – depending on the initiative and context – can manage its own tenant work area where it is possible to operate on all resources made available by the Platform, such as Smart Objects, APIs and Data.  Based on EU standards – like Homer and Open DAI – to share Open Data Sets and API Open Services, it is natively integrated with the AGID National Open Data Platform www.dati.gov.it and federated with EU Open Data Portals; moreover, the platform is used in the regional context to manage the data of all the regional entities that subscribed the Open Data dedicated policy. 

Once the Data Platform had been created, the further challenge for our Client was to make it “Smarter” by fully enabling it to manage ingestion, processing and monitoring of Streaming Data, including all those IoT and multimedia Big Data coming in real-time from all Smart Objects / Devices that can be connected to the open platform to increase the common data asset base, and necessary to exploit its full potential for new Smart City end-to-end applications in specific domains. 

Solution

The Solution provided by Agile Lab is based on the adoption of its highly scalable Witboost platform, with all its features working in a real Big Data context.

Witboost is a modular platform aimed to build complex Big Data applications, in particular for Streaming Data in real time; it relies on a Kappa / Lambda architecture mainly leveraging Kafka and Spark technologies.

Agile Lab has embedded Witboost within the Client’s Smart Data Platform, in order to enable it to ingest and process a huge amount of very heterogeneous data, and consequently to analyze it through dedicated data pipelines.

Witboost and its specific components enable Streaming Analytics for real-time data ingestion, processing, publishing and real-time monitoring of multiple Big Data Streams:

• the IoT Big Data / Events coming in real time streaming or near real-time from the wide network of Sensors, Gateways and Smart Objects registered within the Smart Data Platform ecosystem
• the IoP (Internet of People) Streaming Data coming from Social Networks
• the Real time data coming from the many public and private systems connected to the Platform for specific cases 

Moreover, the solution makes it possible to provide the Smart Data Platform Users with Witboost’s “pipe graph” features, aimed to design and manage specific data pipelines, as its data set / data flow management provides further tools e.g. for enrichment of the Open Data available on the platform.

Agile Lab’s solution enables the collection and the “self-service” processing of data in real-time for, amongst other, CEP (Complex Event Processing), Real Time Alerting based on IoT events, Discovery, experimental Artificial Intelligence algorithms, and so on, for multiple use cases. The information can now be collected and analyzed, allowing users to even deal with complex physical and social phenomena that can only be managed in real-time (for example, to follow the evolution of a natural disaster).

Achieved results & benefits

• The Smart Data Platform is now able to combine the Internet of Things (IoT) and Open Data without any kind of limitation in terms of scaling. 
  
  
• The new IoT Streaming capabilities enabled our Client to participate to the current EU Smart City Initiative “Big IoT” (Bridging the Interoperability Gap of the Internet of Things) in order to capitalize its Data platform as a Smart City Platform together with further EU platforms, with the integration of new data from sensors monitoring, for instance, air quality, traffic, availability of parking places and availability of bike sharing slots.
  
  With many Projects taking place across Europe, the BiG IoT Initiative is aimed to achieve interoperability among different IoT-based Smart City platforms, to promote the development of new services and Applications for Citizens and Public Entities, leveraging the great quantity of public and private data from interconnected mobile devices and sensors, interacting in real time with the external environment. 
  
  
• With the progressive uptake of 5G, the higher bandwidths will allow more devices to connect; this, in turn, means that there will be greater data generation and processing capacity, allowing public entities and companies to benefit from Big Data even more – in particular from IoT data – and from Artificial Intelligence applied in real-time to these data, especially for the further development of Smart Cities.