The Pacific Data Hub: creating a sustainable solution to store and access Pacific-related data

About the Pacific Community

The Pacific Community (SPC) is the principal scientific and technical organisation supporting development in the Pacific region. Created in 1947, its 27 members are the 22 Pacific nations and territories and their five metropolitan partners: Australia, the United States, France, the United Kingdom, and New Zealand. The SPC’s core areas of work include fisheries, public health and education, good governance, trade, disaster management, employment and education, and cross-cutting issues such as human rights, food systems, gender equality, and environmental sustainability and climate change. 

Amongst its other important functions, the SPC is mandated by its members to provide leadership in data and statistics. The SPC is the custodian of over seven decades of Pacific-related data, including population census, household survey, and geospatial data. It works to make this easily available to government decision-makers and related Pacific civil society and business organisations, who require it for evidence-based research and policy-making. This critical work includes enabling Pacific states to undertake the compilation, updating, and dissemination of the Pacific Sustainable Development Indicators, which, in turn, assist countries in meeting their reporting requirements in relation to the United Nations Sustainable Development Goals (SDG). The SDGs create significant statistical challenges for Pacific states, particularly the smaller ones, which do not otherwise have the capacity to undertake the necessary data collection. 

Pacific Data Hub objectives

Pacific Data Hub is an open data portal that serves as a gateway to the most comprehensive data collection of relevance to the 22 Pacific Island nations. In addition to Pacific states, the Hub also pulls in related data from SPC’s metropolitan members and the organisations such as the World Bank, the United Nations, and the Organisation for Economic Cooperation and Development. 

The Hub is the SPC’s answer to the challenges and opportunities it faces as a data steward for the Pacific region. According to Sioeli Tonga, the SPC’s program manager, the Hub’s inception arose in response to the following questions:

How do we ensure that the data under our stewardship is protected, shared responsibly, and utilised ethically? And how do we ensure that it continues to be accessible beyond project funding horizons? And, also, how do we ensure that data can be used to inform policy developments and decision-making that results in positive outcomes for Pacific people?

Prior to the Hub, there was no regional digital infrastructure for the long-term storage and use of data amassed by SPC. Pacific-related data and statistics were difficult to locate, fragmented, incomplete, and held by various stakeholders in multiple formats across numerous geographic and administrative locations. This inhibited the ability of government, researchers, civil society organisations, and business to obtain insights from the region’s data sources and share and reuse data. 

The Hub acts as a digital platform on which any government, donor, academic, or member of the community can both store and access high-quality, reliable, up-to-date data. It is a sustainable digital public good and open data infrastructure for member states, many of whom lack the resources to establish and maintain one on their own; one that goes beyond the shorter funding horizons of most development assistance programs and aid projects. In this way the Hub also seeks to contribute towards:

• Providing timely, high-quality data to enable better policy-making and the creation of more informed and ‘Pacific-centric’ solutions.
• Strengthening data collection and data governance and management practices throughout the region.
• Building partnerships to enhance coordination, strengthen capacity, and improve data literacy.

The Hub also plays a crucial function in building trust in the role of the SPC and its members as data stewards. In Tonga’s words, the Hub ‘provides a governance framework that ensures that the data assets under our care can be used and reused to not only maximise value and impact but also to ensure that the data is used responsibly and ethically.’

Pacific Data Hub functionality

The Hub seamlessly combines CKAN—the Comprehensive Knowledge Archive Network –  with Drupal – a free and open-source web content management system—into an open data platform solution that is both attractive and highly functional. This includes being fully usable for desktop, mobile, and tablet devices. Data can be searched by country or subject and presented in different ways, including:

• CSV, spreadsheets, JSON
• As charts, graphs, and diagrams
• Geospatially in map forms within the CKAN catalogue using PacificMap 

The Hub has the capacity to store data via the CKAN data catalogue. This includes material for which the external source links may not be available or are unstable or temporary. The Hub can also take users to external sites and data repositories. It utilises a variety of harvesters to pull data on a regular basis from external sources. If the external portal or data catalogue has an application programming interface or web service, the Hub can link to it and enable smooth, simple access to datasets by end users.

The Hub took Link Digital two years to mature, from conception to approval of proof of concept and to gathering a critical mass of data, features, and context into the catalogue. 

The technical aspects of the project were complex, not only pulling in and presenting data collected over seven decades and stored in different tools and platforms, but also data held by other organisations stored in their own individual platforms and systems. It underscores Link Digital’s expertise in successfully helping SPC to bring together a deep ecosystem of open source technical elements related to open data dissemination in one comprehensive portal design. Link Digital assisted SPC with the broader complexities involved in sourcing and organising data sets from across multiple geographic and administrative jurisdictions while factoring in linguistic and cultural considerations and questions of data sovereignty and where the data is stored. An important part of the challenges involved in this was establishing the trust of stakeholders, in other words, obtaining the social license. As Tonga puts it:

That initial step of developing a prototype that we could use to demonstrate value quickly was really important. This got us the buy-in we needed to continue the journey. I can’t stress enough how important that partnership with Link Digital for us was to make that happen, because we certainly did not have the capacity or capability internally to do that ourselves quickly.

Since it was established in late 2020, Link Digital has collaborated with SPC on improvements to the Hub’s functionality. This includes building a dashboard that functions as a regional dissemination platform for monitoring the Pacific region’s progress towards meeting the UN SDGs. This is one of ten topic-specific dashboards hosted by the Hub, which can be viewed here, which nearly all use Hub data hosted on CKAN as their source. Link Digital has also worked with SPC on the re-indexing of datasets to align with topics and categories of particular importance to the Pacific, thus improving its ability to assist with evidence-informed research, policy, and planning, and recently upgraded the site to CKAN 2.11.

Benefits of the Pacific Data Hub

The Hub demonstrates how data management platforms based on open source technology such as CKAN have the potential to provide real-world benefits over the long term as digital public infrastructure. The Hub plays a vital role in supporting evidence-based research and more informed policy development on the part of Pacific stakeholders through:

• Improved data discovery by helping to streamline and increase data publishing, discovery, sharing, and re-usage. 
• Improved data quality.
• Improved collaboration by empowering researchers working on similar projects or topics to connect and share resources. 

Examples that clearly demonstrate the Hub’s impact include:

Improving census data and its use and re-use

The Hub’s Microdata Library has become a central repository of the Pacific region’s survey, census, and administrative-based microdata, metadata, and documentation. A free service, populated and maintained by the SPC’s Statistics and Development Division, it allows users to search, access, and download a wide variety of quality data, organised in compliance with international standards. The Microdata Library has led to significant advancements in the standardisation of census and survey methodologies and instruments by Pacific Island governments, including the adoption of clear and transparent legal and data governance frameworks and utilised tools to reduce statistical disclosure risk.

Improving data on sustainable food systems. 

The Hub’s Food Trade Database is a regional data collaboration between regional commodity and country experts, the SPC, and researchers from the University of Wollongong in Australia. In addition to making quality data on healthy, equitable, and sustainable food systems more easily discoverable and shareable, it delivers trade data on food and beverage commodities across the Pacific. According to a 2023 article, the Food Trade Database enables country and regional decision-makers to ‘review trade trends for specific foods, or groups of foods, to better understand economic and health impacts, and identify what future trade trends are likely to be.’ The database also includes an interactive dashboard on the food and beverage trade in the Pacific and a detailed methods paper, made available as an SPC publication.

Meeting the demand for high-quality population data

The Hub provides reliable population data, essential for everything from disaster risk assessment and response to infectious disease control to coastal fisheries management. This includes data that is available to view in population maps that take actual or projected population data and represent the spatial distribution on a continuous grid surface. The Hub also hosts the first detailed estimates of the coastal proximity of populations in 22 Pacific island countries and territories. The data, a collaborative project between SPC, the University of Wollongong and the not-for-profit research organisation WorldFish, is vital data in terms of assessing vulnerability to natural disasters, economic and livelihood issues, and mitigating the impacts of climate change.

Building your data portal?

Get in touch and one of our CKAN and Drupal experts will reach out.