The New South Wales Department of Climate Change, Energy, the Environment and Water (DCCEEW) is responsible for ensuring a sustainable NSW, including working to protect the state’s natural environment and heritage, water resources, its natural park estate, and driving the transition to a net zero economy. Funding for the Data4me project was provided by the DCCEEW’s Biodiversity, Conservations and Science Group, as part of efforts to examine how it can help transform the DCCEEW’s service delivery and improve customer service and environmental outcomes.
The Data4me project was first conceptualised by Tom Barrett, Senior Scientist in the Science, Economics and Insights Division of DCCEEW. Barrett has worked on many large-scale planning activities across all tenures and regions of NSW and is now focused on developing tools that support others to better manage and share knowledge about data and its application. Link Digital partnered with Barrett and his team to make Data4me a reality.
The main driver of Data4me was Barrett’s personal experience, developing and applying Graphic Information Systems (GIS) to support conservation and natural resource management and planning decisions since 1995. “It’s a culmination of thirty years of real frustration and anguish associated with trying to use data to undertake projects,” Barrett said in a February 2024 presentation on the project to Link Digital’s Monthly Forum series. “And I discovered that this is also shared by a lot of other data users.”
The core problem Data4me seeks to solve is the challenge of finding fit-for-purpose data from the growing volume of data in open data portals at NSW state and local levels, but also nationally. This rendered the portals little better than data registries and made it difficult not only to find the right data, but also to share knowledge about it and manage data for specific projects, from the beginning and all the way through the project cycle.
Data4me is an internal facing data portal that harvests data from two NSW government portals, which Link Digital also developed: DCCEEW’s internal asset register (IAR) and a public-facing repository of environment data, Sharing and Enabling Environmental Data or SEED. Date4me is designed to identify and smooth over the pain points that occur over the workflow involved in using data. It does this in the following ways:
Registering a project: Users register a new project in Data4me. This project can be made public, so that everyone in the portal can see it, or private. Users can flag others involved in a project (if these people also registered with the site) or give them the option of being included.
Searching for data: Users can search for data and metadata by keyword or topic, registered projects or people, and organisation. The search returns a list of records with minimal summary info and users can click on the URL metadata statement to review the data in much more detail. The user also has the option to select and add datasets to their project and, thus, consolidate what they find on their own project page.
Rate and review data: When data is downloaded and used in a project, the user usually goes through a discovery phase of reviewing the data to see if it is fit for purpose, before deciding whether to use it or not. Data4me allows the user to flag datasets as A (not used), B (used), or C (generated by project), and rate them 1-5 (with 5 being the highest) in terms of their quality, strengths, and weaknesses, etc. Users can also use a window provided to record more detailed comments of their experience of using a dataset.
Reporting: Users can generate a report that contains a list of citations or references of the datasets they have used, which they can upload to their project site, thus completing the data life cycle. Users can also link their registered project to other information, such as a final project report or paper they have written and other data outputs, or material they have found useful.
Link Digital was selected to undertake the Data4me project on the basis of our extensive knowledge of the Comprehensive Knowledge Archive Network (CKAN), a powerful open-source software that can be configured and set up to function as an open data platform once it is deployed and hosted on a web server. CKAN was used in the implementation of DCCEEW’s IAR and SEED portals, and a decision was taken that Link Digital could use its in-depth knowledge of these two portals to better integrate them with Data4me.
In addition to building a working prototype of the portal, Link Digital’s expertise included:
While it is envisaged that the primary use of Data4me will be on a desktop PC, the portal is ready to support mobile devices with minimum additional effort by utilising a mobile-first approach.
Link Digital currently undertakes managed hosting of the portal on Amazon Web services and are also providing ongoing support with maintenance and troubleshooting.
“The team at Link Digital have been fantastic to work with,” Barrett said. “It has been great. Nothing has seemed too much trouble.”
In the early stages of the open data portals, their use by the government focused more on how to deliver greater transparency. While this is still a vital consideration, governments are placing increasing emphasis on knowledge management; how data is shared, internally and externally, how it is catalogued, and made discoverable for a range of human and non-human actors. It is in this context that the benefits of the Data4me portal become particularly apparent.
While the dataset extension capabilities of Data4me are not technically new, its innovative configuration delivers new functionality. In traditional data portals, users locate a data set and download and use the data outside the portal. They may, if they have the time and resources, feed their results of their research back into the portal.
But, by registering projects, Data4me enables users to search for and work with data inside the actual portal itself. Projects have their own list of the data they are using, centralized in the one place, and this can be supplemented by adding external research material and documentation. And if a user wants to add additional people to the research team or hand the project over to someone else, that person knows exactly what data is being used and its lineage.
Data4me has been running since 2023, so the portal is still in the early stages of its development. Even so, Link Digital believes it is a fascinating example of what is possible in relation to data portal configuration.
Future directions for Data4me include:
As part of the ongoing effort to promote and monitor the portal, the decision has been made to use it to coordinate the process around the upcoming NSW State of Environment Report. The report occurs every three years and is managed by the state’s Environment Protection Authority. Projects will be set up in Data4me for each of the report’s chapters, which will be used to house all the data related to that chapter. “It is a challenging and logistically complex project,” said Barrett. ‘Involving a lot of experts and a lot of data and the challenge will be to see if Data4me can help coordinate the completion of the report more easily. It is also quite a rigorous process, and the hope is that using Data4me for it will give the site added credibility and encourage more people to use it.”
Do you have a data project that you need assistance with?