Apache Sourcelume
Verifiable provenance for AI training data
Apache Sourcelume is open-source instrumentation for AI training-data provenance — a metadata specification, a reference registry, and tooling that let dataset curators and model producers publish signed, independently verifiable records of where their data came from and what terms it carries.
Most training corpora carry licensing and provenance information that is missing, wrong, or unverifiable — a 2024 audit found license-omission rates above 70% across popular dataset-hosting sites. Sourcelume doesn't adjudicate whether a dataset's stated terms are accurate; it gives producers a shared, neutral way to document custody and licensing so that claims can be checked independently.
High-level Apache Sourcelume architecture
graph LR
SPEC[Specification] -->|shapes| ATTEST[Attestation]
SPEC -->|shapes| REG[Registry]
PROD[Dataset producers<br/>& curators] -->|sign with| ATTEST
ATTEST -->|signed records| REG[Registry<br/>Apache Atlas]
REG -->|REST / GraphQL| USERS[Trainers & auditors]
Specification
A versioned, JSON-LD metadata schema for dataset origin, custody chain, and licensing — built to align with OTDI, the DPI annotation taxonomy, Croissant, and the SPDX AI Profile rather than compete with them.
Registry
A reference implementation built on Apache Atlas, exposing REST and GraphQL APIs so trainers and auditors can search, filter, and query provenance records at scale.
Attestation
A signing and verification library — C2PA-compatible where applicable — that lets producers cryptographically sign what they assert, and lets anyone independently check the record's provenance.
Our North Star
graph TD
SPEC[Specification<br/>JSON-LD, JSON Schema, SHACL] -->|crosswalks to| CROISSANT[MLCommons Croissant]
SPEC -->|crosswalks to| SPDX[SPDX AI Profile]
SPEC -->|crosswalks to| OTDI[OTDI]
PROD[Dataset producers<br/>& curators] -->|cli ingest| ATTEST[Attestation<br/>sign & verify]
SPEC -->|defines shape of| ATTEST
ATTEST -->|signed ProvenanceRecord<br/>JSON-LD| REG[Registry<br/>Apache Atlas]
SPEC -->|defines shape of| REG
REG -->|REST / GraphQL| EXPLORER[Explorer<br/>search & provenance cards]
REG -->|REST / GraphQL| MACHINE[Machine consumers<br/>trainers, auditors]
Where to go next
| Page | What you'll find |
|---|---|
| About | About Apache Sourcelume |
| Get involved | Mailing list, chat, and how to contribute |
| FAQ | Short answers to common questions |