Helping the New South Wales Government share large-scale climate data efficiently and securely
New South Wales Department of Climate Change, Energy, the Environment and Water (the department)
New South Wales Climate Data Portal (the Portal)
Access to reliable, high-resolution and up-to-date climate data is important for government, industry and communities. It is vital to support building and infrastructure planning, water catchment management, disaster risk assessment and preparedness, and the development of climate change adaptation and resilience strategies. Utilising the Comprehensive Knowledge Archive Network (CKAN) – Link Digital assisted the department to implement an innovative and cost-effective solution to make their regional scale climate datasets more discoverable and shareable to a broad range of stakeholders and users, in a highly secure environment.
The NSW Department of Climate Change, Energy, the Environment and Water is the key agency responsible for protecting the state’s environment and heritage. It conserves and protects the state’s natural environment, ensures sustainable management of water resources, and leads the way on climate action by driving the sustainable transition to a net zero economy.
A ‘beta version’ of the NSW Climate Data Portal began in 2020 and it went live in early 2021. Work on the current Portal began at that time and went live in September 2025. In this time frame, the project worked through the technical design to implement a complex operation system that could be managed and hosted the external and internal portals.
The Portal has been designed to provide a simple and straightforward way to share important but also highly complex and very large climate datasets with end users experienced with using climate datasets. The climate projections come from the NSW and Australian Regional Climate Modelling (NARCliM) program.
Led by the NSW Government, with support from other Australian state governments, universities, and research institutions, the latest generation of NARCliM, version 2.0, delivers high-resolution regional climate projections for 1951 to 2100 for NSW and south-eastern Australia on a 4km grid. It also covers the broader Australasian region, including New Zealand and parts of Indonesia, and extends out into the Indian Ocean on a 20km grid. It uses the global climate models and greenhouse gas emissions scenarios from the Coupled Model Intercomparison Project Phase 6 (CMIP6), as used by the United Nations Intergovernmental Panel on Climate Change.
The department downscales this data using the Weather Research and Forecasting (WRF) model to ensure it represents the Australian context, factoring in regional and local atmospheric, climatic and land conditions to make it more relevant for local users in these areas. The NARCliM project generates historical and future climate and atmospheric variables and indices. The scientific design of NARCliM2.0 is described more in detail in this 2025 peer-reviewed scientific journal article. The department’s desire to give stakeholders access to this data required an application that takes the complex data structure of NARCliM data and organises data in an easy to understand and navigate interface.
The project comprises two CKAN instances: a public facing portal and an internal portal, linked by a secure API.
On the public portal, users can search for NARCliM data, displayed as key metadata fields, refine their data selection and request the datasets. The dataset request and its parameters are communicated via the secure API to the CKAN internal portal, connected to the department’s secure internal data management system where data requests are packaged and then uploaded to the Amazon cloud storage for users to download. Publicly available data (represented by its metadata) is first registered in the internal portal and then pushed to the AWS S3 bucket.
Using a suite of CKAN filtering tools, users can search by time scale/frequency, scenario, spatial domain (extent) and type of data, such as climate indices or climate variables. The Portal provides both bias adjusted and post-processed data. Through the filtering tools users have fast and easy access to a subset of datasets. Additionally, once users identify the dataset they want with the filtering tools, the Portal offers a selection of tools to refine the dataset choice spatially, by selecting an area or a point location, and temporally by selecting a date range and then selecting the desired output format. Once this refined selection is added to the data cart, users can review and request a download.
While anyone can browse the metadata on the public portal, users must register to access and download datasets. This enables the department’s staff to keep a record of what is downloaded, which helps if users have questions or need assistance. It also enables staff to track needs and feedback from user cohorts and data usage more effectively. It is also important to stakeholder engagement, enabling the department to communicate with users about Portal features and to notify them of new datasets. To aid with discoverability, the public portal home page hosts a glossary of terms, a dictionary of variables and additional support information to assist users make informed data selection choices.
The Climate Data Portal is intended to support a range of data users, including researchers, scientists, climate risk analysts, and specialists across government, community and the private sectors. Through communication activities, the department provides the public with Portal information to build awareness of NARCliM climate projections and access.
As of this case study’s publication date, users can search metadata for over 7000 climatic datasets and collections on the Portal site. The Portals provide data in three output formats:
Users can pick or enter a point on a map and export it to CSV format, to generate time series information for a location. Users can also export areal data to NetCDF, a self-describing file format and library for storing and accessing large, multidimensional scientific data, or as a GeoTIFF of an area location for use in a Geographic Information Systems or other geospatial application.
The Climate Data Portal follows the guidelines for metadata formatting and file naming conventions of the Coordinated Regional Climate Downscaling Experiment (CORDEX). CORDEX is a global initiative by the World Climate Research Programme, aimed at improving and coordinating regional climate downscaling projects. Priority CORDEX metadata fields are displayed with each dataset. NARCliM2.0 is compliant with CORDEX conventions.
The Climate Data Portal is fully DCAT compliant.
The Climate Data Portal is different to many CKAN-powered portals Link Digital has worked on, in that only the metadata, not the datasets, are hosted and managed in the CKAN instances. The department specifies that data be stored securely and managed at its data centre which hosts data for research, modelling and analysis.
Link Digital had to integrate the two CKAN instances with the department’s internal data storage and workflow processes in such a way as to ensure that the data was secure and unable to be compromised. The most challenging part of the job was integrating the CKAN instances into their secure API, because it had to be architected in such a way that there was a complete separation between the public/external data portals and the back-end infrastructure of the department’s data farm. DCCEEW provided all the metadata and Link Digital had to map it onto their specified architecture.
The client’s desired configuration also meant engineering a solution that anticipated the potential pain points for the various links in the chain and fault proof each of them. This required an in-depth understanding of the Portal’s data workflow, which is as follows:
Why did the department want the Climate Data Portal configured like this?
AWS cloud storage is an appropriate resource for the temporary storage of requested datasets, until it can be downloaded by the user. Maintaining the data storage on internal systems keeps the data secure and ensures it can exist as a single source of truth. This way the department’s staff only need to manage the source of truth datasets, which brings maintenance and cost efficiencies.
The department selected CKAN for three main reasons:
The Portal increases exposure to important climate data delivery gaps for practicing and experienced users – people who use localised climate data in their day-to-day occupation and understand and can analyse it.
Now that the site is live, users will be able to provide feedback to support future improvements. With its design, implementation and operational procedures, the NSW Climate Data Portal provides users with an opportunity to enhance awareness and access to localised climate projections data.
Link Digital is ready to start a conversation with your team on your open data needs. Contact us and tell us about your project.