A national OECD government
The portal is the Government’s central open catalogue for the discovery of public data, featuring tabular, spatial and relational data with hosted APIs and the option for agencies to link data and services hosted by other government sources at a national, state, and local level. The portal has been operating for over a decade. As the national data ecosystem evolved and the portal transitioned from one statutory authority to another, the government engaged Link Digital to securely transition the portal exclusively to the Comprehensive Knowledge Archive Network (CKAN), as opposed to its previous mixed software architecture in which CKAN was only one part of the tech stack. The client wanted the new solution to continue to harvest existing data from many sources, meet strict accessibility guidelines, and feature a highly customisable front end to support supplementary content and changing requirements.
This project demonstrates Link Digital’s CKAN expertise and our ability to deliver secure, national-level high-performance open data infrastructure for national government.
Link Digital was engaged late 2024, and the portal went live mid-2025.
Link Digital has been involved in improving the quantity and quality of the government’s open data and the portal’s tech stack for much of the period since it was first established in the beginning of the twenty-tens. Early on we were engaged as the primary technical partner to manage, host, and maintain the early CKAN infrastructure on Amazon Web Services, and provided 24/7 infrastructure monitoring, security patching, customised extension development, and the deployment of dataset APIs (DataStore). A different government agency took charge of the portal in the early 2010s. While CKAN remained as the core data management technology, another open source software was integrated into the platform to administer its front-facing search features and federated registry layer.
When the portal was transferred to a new agency Link Digital was engaged to help consolidate the portal to a new all-CKAN environment and install Drupal as the site’s content management software.
While the client now hosts the portal in-house, Link Digital continues to provide managed support services.
The portal is the Government’s official national portal, supporting its policy of requiring all non-sensitive government data to be open by default and encouraging the sharing and re-use of the data for economic growth, improved service delivery outcomes and a more informed citizenry.
The portal currently hosts over 100,000 data sets. A proportion of those are native to the site, while the remainder have been harvested from other sites and republished on the portal, and that volume expands continuously.
Data on the platform is available in machine-readable and open standard formats to allow automated querying and API access. The data set library spans broad subject classifications and data from a wide range of formats, including:
Data is syndicated from over a thousand organisations. These include:
The portal also hosts publicly funded research data and datasets from private institutions that are in the public interest, and provides metadata and other information regarding unpublished data or datasets available for purchase.
Anyone can freely search, view, and download publicly listed datasets without registering or providing personal information. To publish data on the portal, users must have an account with a government email address at a national, state or local level. In rare instances a data set hosted by an external organisation linked by the national open data portal may require a separate login, registration, or agreement to specific usage terms.
The client was keen to transition the portal to a single, unified CKAN architecture to make it simpler to administer and reduce maintenance costs. The new portal utilises CKAN for all back-end data management, and Drupal for front-end content. Link Digital’s work involved the following:
The migration to an all-CKAN environment was the most complex part of the service engagement. Maintaining all the datasets native to the portal while reintegrating the harvester process seamlessly back into CKAN was challenging due to the outdated and insecure versions of CKAN being used by the previous authority overseeing the portal. We worked with the new owner, who had no CKAN experience, to fully upgrade CKAN and engineer the migration of the site to a new domain, while ensuring total data integrity and continuity. We smoothly transitioned the architecture and seamlessly harvested all existing harvest sites to ensure total data integrity and continuity.
While the portal was being migrated there was only a brief period of a few days during which new datasets could not be uploaded into the portal, but otherwise it remained fully online and accessible to the public.
Implementing custom extension plugins and maintaining hosted APIs (such as tabular, spatial, and relational data query endpoints). The search and filter functions now allow users to search across a massive library of datasets and filter them by tags, organizations, and file formats (such as CSV, XLSX, HTML, WMS, and ESRI MapServer).
The new data federal architecture includes a harvesting mechanism designed to identify and communicate the original source of data sets to users, effectively managing potential duplicate data from other government open data portals. This metadata-driven approach ensures users can navigate to the original publisher when necessary.
We developed a new, dynamic front page using Drupal, designed to be highly customisable so it can meet evolving requirements. This included the implementation of a “Toolkit” feature, empowering administrators to easily create and manage supplementary pages (e.g., About, Privacy, and user toolkits).
Metadata structures conform to open standards, like the Data Catalog Vocabulary. This enables seamless data harvesting and cross-catalog interoperability between national, state and local portals.
The new version of the portal was audited to adhere to strict accessibility standards. All identified issues were subsequently addressed by the team to ensure the latest version of the portal met government accessibility mandates and ensured inclusive public access.
Link Digital is providing end-to-end service transition, custom development, and ongoing managed infrastructure. This covers day-to-day technical support, proactive system maintenance, critical security updates, and minor application development tasks to continuously improve the platform.
The portal continues to serve as a single discovery platform where anyone – citizens, researchers, developers, and businesses – can access anonymised real-world public data published by government agencies at all levels, and civil servants can upload datasets.
The updating and migration of the CKAN instance has improved security and ensured full compliance with government accessibility standards. Development continues on a secure platform for internal data sharing between government departments.
The project demonstrates Link Digital’s ability to manage large-scale, national-level open data infrastructure. By combining CKAN data management with a flexible Drupal front end, strict accessibility compliance, and reliable monthly managed services, we have established a clear blueprint for delivering secure, high-performance CKAN ecosystems for national governments, with minimal service disruption or downtime. The project is also a textbook case study in the adaptability and flexibility of CKAN, including its ability in terms of working with other software components.