Registry
The Apache Sourcelume Registry is the central repository for signed provenance records. It provides a stable, queryable endpoint for dataset consumers and auditors to verify provenance claims.
Current Implementation vs. Target Architecture
The Registry is evolving. The current implementation is a Quarkus service that validates, stores, and resolves provenance records through a REST API. Storage is pluggable through a storage SPI: the PostgreSQL distribution is the default backend, and an Apache Atlas distribution is also available. The high-level architecture below describes the target enterprise-grade system, and the Simple Demo remains a lightweight way to try the core registry workflows without a real backend.
Quick Start: Simple Demo
Want to see the registry in action without any infrastructure? The simple-demo provides a lightweight, standalone implementation of the core registry concepts.
- Purpose: Demonstrates the fundamental REST API workflows for provenance.
- Ideal for: Rapid prototyping, understanding basic data flow, and local testing.
- Key Difference: Uses an in-memory store instead of a persistent backend.
View the Simple Demo on GitHub →
To see the whole stack — training a small model and publishing its provenance records — visit the Sample.
API Reference
The REST API of the current registry implementation is described by an OpenAPI document that is kept in sync with the code:
- OpenAPI document (
docs/openapi.yaml) — the endpoints, request and response schemas, and status codes of themainbranch. - Interactive documentation — a locally running registry serves the same document at
/q/openapiand an interactive Swagger UI at/q/swagger-ui(for examplehttp://localhost:8082/q/swagger-ui).
The API is still evolving. The API Design section below sketches the intended design of the registry; the OpenAPI document describes what the registry implements today.
High-Level Architecture (Target State)
The Registry serves as the authoritative store for AI training data provenance records. It receives signed attestations from data producers, stores them with full lineage tracking, and exposes them through modern APIs for verification and audit.
graph LR
A[Data Producers] -->|Signed Attestations| B[Attestation Pipeline]
B -->|Ingest| C[Registry Core]
C -->|Store| D[Metadata store]
D -->|Lineage & Metadata| C
C -->|REST API| E[Explorer UI]
C -->|GraphQL API| F[Machine Consumers]
C -->|REST API| G[Auditors]
Architecture Flow:
- Data Producers → Signed Attestations → Attestation Pipeline
- Attestation Pipeline → Ingest → Registry Core
- Registry Core ↔ Store/Retrieve ↔ Metadata store
- Registry Core → REST API → Explorer UI
- Registry Core → GraphQL API → Machine Consumers
- Registry Core → REST API → Auditors
Key Components:
- Metadata store — Pluggable through the storage SPI; PostgreSQL is the default distribution today, and the Apache Atlas distribution adds lineage tracking and graph-based querying
- Attestation Ingestion — Validates and processes signed provenance records
- REST & GraphQL APIs — Dual interface for human-readable exploration and programmatic access
- Verification Engine — Cryptographic signature validation and chain-of-custody verification
Detailed Architecture
Technology Stack
The Registry is built with:
- Java 21 — Modern Java with performance improvements and language features
- CDI (Contexts and Dependency Injection) — Enterprise dependency injection and lifecycle management
- Storage SPI — Pluggable metadata backends; PostgreSQL (default distribution) and Apache Atlas
- REST & GraphQL — Dual API layer for flexible access patterns
Core Modules
graph TD
A[API Layer] --> B[Service Layer]
B --> C[Attestation Validator]
B --> D[Lineage Tracker]
B --> E[Query Engine]
C --> F[Storage Client]
D --> F
E --> F
F --> G[Metadata store]
The Registry architecture consists of several layered modules:
- API Layer — REST and GraphQL endpoints with authentication and rate limiting
- Service Layer — Business logic that coordinates attestation processing, verification, and querying
- Attestation Validator — Cryptographic signature verification and schema validation
- Lineage Tracker — Manages provenance chains and dataset relationships in the metadata store
- Query Engine — Optimized queries against the metadata store for common access patterns
- Storage Client — Integration layer with the pluggable storage backend
- Metadata store — Underlying repository for records and lineage (PostgreSQL or Apache Atlas today)
Data Flow
-
Ingestion Workflow
- Attestation arrives via REST endpoint with signature
- Signature validated against known public keys
- Schema validated against Sourcelume specification
- Metadata extracted and normalized
- Record stored in the metadata store with lineage relationships
- Confirmation returned to producer
-
Query Workflow
- Consumer requests provenance for dataset identifier
- Query engine translates to a store-specific lineage query
- Results filtered and formatted
- Signature proofs included in response
- Response returned via REST or GraphQL
-
Verification Workflow
- Auditor submits attestation for verification
- Registry checks signature validity
- Lineage chain reconstructed from the metadata store
- Chain-of-custody validated
- Verification report generated
API Design
REST Endpoints:
POST /api/v1/attestations— Submit new attestationGET /api/v1/attestations/{id}— Retrieve attestation by IDGET /api/v1/datasets/{id}/provenance— Get full provenance chainPOST /api/v1/verify— Verify attestation signature and chain
GraphQL Schema:
type Attestation {
id: ID!
datasetId: String!
producer: Producer!
signature: Signature!
metadata: JSON!
lineage: [Attestation!]!
}
type Query {
attestation(id: ID!): Attestation
datasetProvenance(datasetId: String!): [Attestation!]!
verifyAttestation(attestationId: ID!): VerificationResult!
}
Developer Setup
To contribute to the Registry or run it locally for testing, follow these steps.
Prerequisites
- Java 21 — Install via SDKMAN! or your package manager
sdk install java 21.0.1-tem
- Maven — Build tool
- Docker — For running the storage backend locally
- Git — Version control
Local Development Environment
- Clone the repository
git clone https://github.com/apache/sourcelume-registry.git
cd sourcelume-registry
-
Start a storage backend
The
dev-supportdirectory contains Docker Compose files for the supported backends. For the default PostgreSQL distribution:
docker compose -f dev-support/docker-compose.postgres.yml up -d --wait
For the Apache Atlas distribution, start a local Atlas 2.5.0 stack instead (Atlas
runs on localhost:21000, default credentials admin / atlasR0cks!).
- Build the project
mvn clean install
-
Run the Registry
The repository ships two distributions — one per storage plugin:
mvn -pl sourcelume-registry-distribution-postgres quarkus:dev
# or
mvn -pl sourcelume-registry-distribution-atlas quarkus:dev
The Registry API will be available at http://localhost:8082, with OpenAPI at
/q/openapi, Swagger UI at /q/swagger-ui, and health at /q/health/ready.
See
dev-support/README.md
for the full matrix of backend and distribution combinations.
Testing Against Local Registry
The dev-support directory includes Docker Compose configurations for the supported storage backends.
Run integration tests:
mvn verify -Pintegration-tests -pl sourcelume-registry-distribution-atlas
Publish a record:
curl -X POST http://localhost:8082/records \
-H "Content-Type: application/ld+json" \
--data-binary @examples/simple-demo/sample-provenance-record.jsonld
Read it back (references resolved on read):
curl http://localhost:8082/records/<url-encoded-record-id>
Query the dataset views:
curl "http://localhost:8082/api/v1/datasets?qualifiedName=<record-id>"
Development Tips
- Hot Reload —
quarkus:devreloads Java changes automatically in dev mode - Atlas UI — When using the Atlas distribution,
http://localhost:21000exposes the Atlas UI for browsing metadata and lineage - Logging — Set
quarkus.log.level=DEBUGfor verbose output during development
Registry Repository
The Sourcelume Registry is actively developed in its own repository:
Visit the repository to:
- View current implementation status
- Report issues or propose features
- Contribute code or documentation
- Join technical discussions