It’s no secret that as the open data movement enters its third decade, traditional data platforms are dealing with a profound structural shift. For the past twenty years, open data portals operated primarily under a static storage model. Organisations collected data, structured it into standard tabular or geospatial formats, assigned basic administrative metadata, and uploaded it to public portals for humans to download and share.
However, the exponential growth of machine learning (ML) and autonomous AI agents exposes vulnerabilities in these traditional repositories. When autonomous AI agents or ML models ingest raw data files that have been stripped of their original operational context, consent conditions, and creation intent, the risk of data drift, privacy violations, and model hallucinations increases dramatically. And without explicit, machine-readable provenance, modern AI models can end up training on unverified or altered inputs.
Link Digital is working to improve this with the introduction of verifiable honesty – upgrading traditional data catalogues into active, verifiable data infrastructure tailored specifically for machine learning pipelines. Or, to put it another way, rather than asking users to trust the data from a system you cannot see into, a clear Statement of Intent is published up front, and what happens can be checked against it.
This is the intent behind the Provenance Sandbox, an initiative being led by Link Digital’s Executive Director Steven De Costa.
Once data is in and it has been used for machine learning, how would we correct things that might be completely wrong or bad?
Steven said during a live demonstration of the Sandbox and the ideas behind it at the most recent CKAN Monthly Live in August.
If we can’t verify the lineage and intent behind our data, we’re building a future of AI on foundations of shifting sand. We might publish something right now and know the topology of it, but as things change, we won’t be able to hold onto it.
Here’s a brief recap of Steven’s demonstration of the Sandbox and the stack behind it, which you can watch in full at the link below.
Moving beyond passive data storage
In traditional data management, open data portals function as passive storage endpoints. Data providers upload assets in various formats, alongside high-level descriptive tags. Downstream data consumers, including automated ETL (Extract, Load, and Transform) pipelines and ML systems, download the data under the implicit assumption that it is accurate, complete, and legally usable.
But in an AI-driven environment, this model starts to experience problems. Large language models and predictive algorithms possess no intrinsic awareness of the organisation that generated a given dataset or the purpose for which it was developed. And they cannot inherently determine whether a dataset was compiled with strict regulatory consent, whether its fields were modified post-publication, or what operational hypothesis guided its collection.
Steven discussed how the Provenance Sandbox can help meet these challenges. Rather than asking users to trust the data from a system they cannot see into, a clear Statement of Intent is published up front and integrated directed into the metadata model, and what happens can be checked against it. Instead of treating a dataset as an isolated and static artifact, datasets are wrapped in a continuous, machine-readable chain of evidence, changing data repositories from simple download distribution hubs into active infrastructure capable of participating in agentic workflows.
One of the key things that we really want to understand about data is, who stands behind it and how did it come to be and what does the data consent to? It’s consent from the custodian of the measurement, the thing that it was sampled from. If it’s a place, if it’s a time, how do you hold those things and how do you have the provenance available?
The Provenance Sandbox stack
The architecture introduced by Steven combines open-source metadata standards, cryptographic logging, and runtime verification protocols layered directly over the core CKAN engine.
Machine learning integration via MLCommons Croissant
Data discovery adopts Croissant, an open metadata standard developed under the umbrella of MLCommons. In the Provenance Sandbox architecture, MLCommons Croissant serves as the declarative discovery and structural layer for machine learning workflows. It provides:
- A standardised metadata format (using JSON-LD) that describes what a dataset contains and how its columns, files, and resources are structured. It provides these in a standardised, machine-readable format for datasets, making datasets instantly discoverable and loadable by leading machine learning frameworks.
- Direct Pipeline Ingestion: Datasets can be loaded directly into popular machine learning tools and frameworks without requiring custom ingestion scripts.
But while Croissant solves the structural ingestion problem – explaining how a dataset is formatted and where its columns map – it does not inherently evaluate the operational rules, business logic, or goal-setting criteria that governed the data’s production.
Goal-setting logic via the Objective Observer Initiative
To address the operational logic missing from standard metadata formats, the Provenance Sandbox introduces a framework being developed as part of another capability Link Digital is working on, the Objective Observer Initiative (OOI). This establishes three distinct functional domains:
- This domain captures the initial Statement of Intent, the explicit grant, scope, intended utility, and legal constraints agreed upon by data creators during collaborative phases.
- It anchors transformation scripts, processing algorithms, and model parameters directly to the declared intent.
- It captures the observer experience, acting as an automated observer, recording live execution outputs, verifying schema integrity, and publishing state registries back to the data portal.
The OOI has the potential “to transform abstract AI ethics into machine readable, verifiable records of trust,” said Steven. The idea of what trust is is kind of a synthetic agreement. How we agree about how we are going to operate in civic contexts, in social norms is very difficult to adjudicate and find justice in unless that synthetic agreement is somehow tangible, somehow constructed and readable. And, if we are going to be using a lot of AI, and supporting technologies, it really has to be machine readable and actionable as well.”
While Croissant handles data discovery and structural description for ML engineers, it acts as a complement to the OOI. Croissant defines the dataset’s layout for machines, whereas OOI defines the prior goal-setting logic, operational rules, and ‘game’ behind the data.
CKAN as the readable surface for the data
The Comprehensive Knowledge Archive Network (CKAN) is used as the foundational data catalogue and storage layer. CKAN acts as the physical repository where the raw data files, datasets, and initial resources are uploaded, stored, and managed, and provides the portal user experience, search features, access controls, and API endpoints that allow users and applications to browse, discover, and download datasets.
“CKAN is actually a really cool tool for doing this,” De Costa maintained. “It has a lot of features. It is very API happy to work with other tools.”
Rather than replacing CKAN, the Provenance Sandbox extends it. CKAN acts as the reliable storage core, while the external frameworks (MLCommons Croissant for structural ML schemas and OOI for intention sealing/witnessing) layer on top to turn CKAN from a passive repository into active ML infrastructure.
Also to read:The Objective Observer Initiative and the challenge of being fully open
Establishing verifiable trust in the AI era
The architecture takes a different approach to guaranteeing data integrity. Rather than checking for data tampering after a dataset has already been processed, the system anchors parameters before computational runs take place:
- A data steward or automated pipeline registers a Statement of Intent outlining the experimental design, geographic boundaries, and processing constraints.
- The declared parameters, raw source references, and transformation scripts are hashed, locking the initial state into the platform’s ledger.
- As live transformations run, the OOI observer continuously conducts ‘witness checks,’ testing runtime outputs against the sealed parameters.
- If incoming data points deviate from the declared parameters – due to sensor drift, code changes, or altered upstream sources – the deviation is not silently overwritten. Instead, the drift is appended to an immutable, PROV-O compliant ledger.
We have to create data that is for the dynamics that we want to improve,” Steven said. “That’s where I’m starting. I am starting before the data, with the statement of intent. And I am saying… Croissant is the standard way to express that at the moment doing R&D in this space.
The aim is to replace subjective assumptions about data reliability with objective proof. Rather than asking whether a dataset is abstractly accurate, machine learning models can verify whether the dataset strictly conforms to its witnessed lineage and declared purpose.
The transition from static data portals to active, verifiable infrastructure represents a vital evolution for open data ecosystems. As automated tools, autonomous agents, and machine learning models become the primary consumers of public data, traditional metadata tagging models are no longer sufficient to guarantee data safety, attribution, and reliability.
Working in this way allows you to know what is verifiably honest… The records really matter. They come through a process that has a declared intent at the start, so an identifying entity. It operates authentically to that identity and, as it operates, it encodes itself. It makes itself observable, communicable and effectual, and it leaves records that are also observable, communicable and effectual.
Steven finished his demonstration with a call to the open-data community to move away from passive data consumption and actively participate in building active, verifiable infrastructure.
Creating the right data sets for the right future of AI is a big responsibility. And I think we really have to try and catch that responsibility and hold onto it.
He stressed that the Sandbox is open to “folks wanting to do other stuff, working with data sets that can be more actionable, ready for machine learning, ready for some of these challenges that we are facing. Let us know if you want to play with it. If you want an account.”
If you want to know more, you can watch Steven’s Provenance Sandbox demonstration, including a potential use case example and the discussion that followed it, on our YouTube site here.
You can also download Steven’s presentation here.
If you are looking to participate in this experiment, you can join by simply registering here
