Big Data Test Infrastructure: The EU’s approach to enhancing public sector efficiency

Located within the European Commission’s Directorate-General for Digital Services (DIGIT), the Big Data Test Infrastructure (BDTI) team explains how the BDTI can help European public administration to overcome socio-economic challenges through the analysis of open data. 

The European public sector faces increasingly complex socioeconomic problems that require innovative solutions. Big data and data-informed decision-making offer a pathway to better solutions, greater transparency, and enhanced trust in public administration. However, implementing these technologies requires skills, experience, and appropriate digital tools, which are often lacking. 

Functionality and benefits

To tackle these challenges, the European Commission launched the Big Data Test Infrastructure under the Digital Europe Programme in 2019. The BDTI aims to promote the reuse of public sector data and facilitate a data-informed public sector across EU member states. It provides public administrations with a comprehensive set of mainstream open-source tools for data storage, processing, and analytics, all hosted in the cloud and free of charge. 

This service empowers civil servants at all government levels to conduct pilot projects independently, demonstrating the value of data in policymaking and administration. Public administrations can utilise the BDTI to prototype solutions before implementing them in live environments. The platform comprises open-source tools and necessary cloud infrastructure like virtual machines, analytics clusters, storage facilities, and networking capabilities. After completing a pilot project, users can transfer the source code and data to continue their work on their preferred cloud or other resources. 

The BDTI offers a variety of tools to facilitate data-driven projects, including data storage and processing tools for handling large datasets, analytics tools for analysis and visualisation, and cloud infrastructure for running data-intensive tasks. Its open-source nature ensures flexibility, allowing public administrations to explore different solutions cost-effectively. 

Use cases

Several public administrations have utilised and continue to use the BDTI to enhance decision-making processes. For example, the health service in the city of Valencia, Spain health service previously employed BDTI to apply text mining techniques for extracting valuable insights from health data. Similarly, the Norwegian Digitalisation Agency (Digdir) used BDTI to optimise public procurement processes through data analytics, while the city of Florence, Italy utilised BDTI to understand the impact of Covid-19 on the city through data analytics. 

Currently, numerous public administrations are leveraging BDTI. For instance, the University of Macedonia in Thessaloniki, Greece transformed data gathered through the Greek National Registry of Administrative Public Service (MITOS API) into linked data aligned with the EU standard model to support more public services. The city of Bochum, Germany is using BDTI to analyse the health of trees, providing valuable insights for urban planning and environmental management. Additionally, the City of Naples is employing BDTI to collect data related to public spaces and mobility, streamlining citizen participation, and creating a transportation dataset tailored for public administration in compliance with the open data directive. 

Additionally, the BDTI website highlights various use cases, such as the search analytics project based on the CORDIS open dataset, which shows how data can be used to enhance search functionalities. Another example is the low code analytics project that utilises the EMHIRES dataset, focusing on solar power generation, to illustrate the application of low code solutions in data analytics. 

“The BDTI represents a significant step towards a datainformed public sector in the EU.”

Resources

To further support public administrations, the BDTI team has developed a BDTI Essentials course, available for free. This course consists of five one-hour webinars, targeting public administration officials to help them better understand how to use the BDTI with concrete examples. This educational resource aims to build the necessary skills and knowledge to effectively leverage the BDTI for data-driven projects.  

Additionally, the BDTI team has recently launched a new online course in collaboration with the interoperability academy, to help public sector actors advance their data literacy journey and understand how BDTI can help shortcut their data maturity efforts. 

In line with their commitment to inclusivity, the BDTI team has also taken the initiative to translate key BDTI materials into the 24 European official languages. This effort allows individuals from diverse linguistic backgrounds to fully understand and engage with the content.  

Public administrations interested in using the BDTI can subscribe to the join up page or to The BDTI Kitchen: Baking Data Stories newsletter for updates on future events and initiatives. 

Impact and promise

The BDTI represents a significant step towards a data-informed public sector in the EU, offering public administrations the tools and infrastructure needed to harness the power of big data. 

Through pilot projects and real-life use cases, the BDTI demonstrates the transformative potential of data analytics in improving public services and policymaking. By enabling experimentation and innovation in a cost-effective manner, the BDTI helps public administrations overcome data challenges, fostering a more transparent, efficient, and trustworthy public sector. 

Are you a civil servant interested in developing your own use case on the BDTI or a student and researcher eager to support a public administration? Apply for a BDTI pilot project here or reach out to the team via EC-BDTI-PILOTS@ec.europa.eu.