Data4me: An internal data portal project by the New South Wales Government

This case study was published in May 2024.

About the client

The New South Wales Department of Climate Change, Energy, the Environment and Water (DCCEEW) is responsible for ensuring a sustainable NSW, including working to protect the state’s natural environment and heritage, water resources, its natural park estate, and driving the transition to a net zero economy. Funding for the Data4me project was provided by the DCCEEW’s Biodiversity, Conservations and Science Group, as part of efforts to examine how it can help transform the DCCEEW’s service delivery and improve customer service and environmental outcomes.

The Data4me project was first conceptualised by Tom Barrett, Senior Scientist in the Science, Economics and Insights Division of DCCEEW. Barrett has worked on many large-scale planning activities across all tenures and regions of NSW and is now focused on developing tools that support others to better manage and share knowledge about data and its application. Link Digital partnered with Barrett and his team to make Data4me a reality.

Objectives of Data4me

The main driver of Data4me was Barrett’s personal experience, developing and applying Graphic Information Systems (GIS) to support conservation and natural resource management and planning decisions since 1995. “It’s a culmination of thirty years of real frustration and anguish associated with trying to use data to undertake projects,” Barrett said in a February 2024 presentation on the project to Link Digital’s Monthly Forum series. “And I discovered that this is also shared by a lot of other data users.”

The core problem Data4me seeks to solve is the challenge of finding fit-for-purpose data from the growing volume of data in open data portals at NSW state and local levels, but also nationally. This rendered the portals little better than data registries and made it difficult not only to find the right data, but also to share knowledge about it and manage data for specific projects, from the beginning and all the way through the project cycle.

Data4me is an internal facing data portal that harvests data from two NSW government portals, which Link Digital also developed: DCCEEW’s internal asset register (IAR) and a public-facing repository of environment data, Sharing and Enabling Environmental Data or SEED. Date4me is designed to identify and smooth over the pain points that occur over the workflow involved in using data. It does this in the following ways:

Registering a project: Users register a new project in Data4me. This project can be made public, so that everyone in the portal can see it, or private. Users can flag others involved in a project (if these people also registered with the site) or give them the option of being included.

Searching for data: Users can search for data and metadata by keyword or topic, registered projects or people, and organisation. The search returns a list of records with minimal summary info and users can click on the URL metadata statement to review the data in much more detail. The user also has the option to select and add datasets to their project and, thus, consolidate what they find on their own project page.

Rate and review data: When data is downloaded and used in a project, the user usually goes through a discovery phase of reviewing the data to see if it is fit for purpose, before deciding whether to use it or not. Data4me allows the user to flag datasets as A (not used), B (used), or C (generated by project), and rate them 1-5 (with 5 being the highest) in terms of their quality, strengths, and weaknesses, etc. Users can also use a window provided to record more detailed comments of their experience of using a dataset.

Reporting: Users can generate a report that contains a list of citations or references of the datasets they have used, which they can upload to their project site, thus completing the data life cycle. Users can also link their registered project to other information, such as a final project report or paper they have written and other data outputs, or material they have found useful.

Link Digital’s role in Data4me

Link Digital was selected to undertake the Data4me project on the basis of our extensive knowledge of the Comprehensive Knowledge Archive Network (CKAN), a powerful open-source software that can be configured and set up to function as an open data platform once it is deployed and hosted on a web server. CKAN was used in the implementation of DCCEEW’s IAR and SEED portals, and a decision was taken that Link Digital could use its in-depth knowledge of these two portals to better integrate them with Data4me.

In addition to building a working prototype of the portal, Link Digital’s expertise included:

• Enabling Data4me to harvest data from the DCCEEW IAR and SEED while filtering out duplicates from metadata. Data is automatically harvested via the Data4me application programming interface from both portals once a day.
• Visual styling of the portal, which had to strictly follow NSW government design standards.
• Designing Data4me to deliver a similar user experience to the DCCEEW IAR and SEED portals.
• Working with client team to identify project metadata properties to be implemented.
• Customizing Data4me with various plug-ins. One of these was CKAN extension for managing datasets relationships. This is normally used to link related datasets, giving people the opportunity to explore what data might be relevant to them. But Link Digital used the functionality to give the application the ability to link up these datasets to projects that are registered on Data4me.

While it is envisaged that the primary use of Data4me will be on a desktop PC, the portal is ready to support mobile devices with minimum additional effort by utilising a mobile-first approach.

Link Digital currently undertakes managed hosting of the portal on Amazon Web services and are also providing ongoing support with maintenance and troubleshooting.

“The team at Link Digital have been fantastic to work with,” Barrett said. “It has been great. Nothing has seemed too much trouble.”

Benefits of Data4me

In the early stages of the open data portals, their use by the government focused more on how to deliver greater transparency. While this is still a vital consideration, governments are placing increasing emphasis on knowledge management; how data is shared, internally and externally, how it is catalogued, and made discoverable for a range of human and non-human actors. It is in this context that the benefits of the Data4me portal become particularly apparent.

While the dataset extension capabilities of Data4me are not technically new, its innovative configuration delivers new functionality. In traditional data portals, users locate a data set and download and use the data outside the portal. They may, if they have the time and resources, feed their results of their research back into the portal.

But, by registering projects, Data4me enables users to search for and work with data inside the actual portal itself. Projects have their own list of the data they are using, centralized in the one place, and this can be supplemented by adding external research material and documentation. And if a user wants to add additional people to the research team or hand the project over to someone else, that person knows exactly what data is being used and its lineage.

Other benefits of the project include:

• Richer metadata: Metadata often contains only the bare minimum of information. Data4me enables users to improve and build on metadata with the knowledge they have gleaned from using it – or not – and understanding what its strengths and weaknesses are, which in turn improves data comprehension and re-use. This is further enhanced by the ability to add supporting documentation that can be accessed with the dataset and is context-sensitive.
• Improved efficiency: Data4me can save users time by enabling them to join the dots more easily between projects, documentation, people, themes, organisations, and metadata. This is done via the portal’s unique configuration and by enabling users to leave feedback on datasets that others can see, enabling datasets to be previewed for far more immediate comprehension and sense making. Enabling users to view projects and how they have used data can also potentially avoid reinventing the wheel, in terms of the research they are doing.
• Encouraging collaboration: Enabling users to see what others are doing breaks down silos and enhances collaboration within and between different parts of the same government department or agency.

Future plans

Data4me has been running since 2023, so the portal is still in the early stages of its development. Even so, Link Digital believes it is a fascinating example of what is possible in relation to data portal configuration.

Future directions for Data4me include:

• Adding new features to help with useability.
• Linking to more portals for data harvesting. In the first instance, this includes Data.NSW, NSW Climate Data Portal, and the Commonwealth open data portal, data.gov.au, all of which use CKAN. This will allow users to search over 100,000 metadata records from NSW and Nationally.
• Working to achieve a critical mass of users and encourage those who do use the site to leave feedback and rate datasets. This is in many respects the biggest challenge. It requires cultural change within DCCEEW to make Data4me regarded as just another tool staff use when they start a project.
• Developing a new feature that will allow the Date4me team to better track adoption and user engagement.
• Implementing email alerts sent to data creators to inform them their data is being used by Data4me users when it is added to projects.

As part of the ongoing effort to promote and monitor the portal, the decision has been made to use it to coordinate the process around the upcoming NSW State of Environment Report. The report occurs every three years and is managed by the state’s Environment Protection Authority. Projects will be set up in Data4me for each of the report’s chapters, which will be used to house all the data related to that chapter. “It is a challenging and logistically complex project,” said Barrett. ‘Involving a lot of experts and a lot of data and the challenge will be to see if Data4me can help coordinate the completion of the report more easily. It is also quite a rigorous process, and the hope is that using Data4me for it will give the site added credibility and encourage more people to use it.”

---

Do you have a data project that you need assistance with?