Geospatial Schema Standardization & Mapping Automation

Validate DCAT-AP Metadata with SHACL Shapes

DCAT-AP is a contract: a national data portal that harvests your catalogue expects every record to carry a publisher, a resolvable license, at least one distribution, and a well-typed spatial extent. “Looks right” is not enough — the portal validates against the official DCAT-AP SHACL shapes and rejects records that fail, often with a terse error days after submission. Validating locally with the same shapes turns that late rejection into an immediate, actionable CI failure. This procedure models a record as RDF, runs it through pyshacl, and gates publication on the result. It is the validation half of Dataset Metadata & JSON-LD Publishing, which owns the manifest and the engine that emits the DCAT-AP node this page checks.

The steps follow the configure → execute → validate → log rhythm: load the record and the shapes, run the validator, interpret the report, and gate the build. SHACL validation is deterministic and offline once the shapes are pinned, so it runs in every pull request without network flakiness.

Prerequisites checklist

Confirm the toolchain and the pinned shapes before validating. The most common source of non-reproducible results is an unpinned shapes graph fetched live from the web, so vendor the shapes into the repository first.

Step 1: Load the DCAT-AP record as an RDF graph

SHACL validates RDF, not JSON, so first parse the metadata into an rdflib.Graph. rdflib reads JSON-LD directly when the json-ld parser plugin is available, which lets you validate the exact document the publishing engine emits without a separate conversion step.

python
# load_graph.py  — rdflib >=7.0
from __future__ import annotations
import logging
from pathlib import Path

from rdflib import Graph
from rdflib.namespace import DCAT, DCTERMS

logger = logging.getLogger(__name__)


def load_record(record_path: Path) -> Graph:
    """Parse a DCAT-AP record (JSON-LD or Turtle) into an RDF graph."""
    g = Graph()
    fmt = "json-ld" if record_path.suffix in {".json", ".jsonld"} else "turtle"
    g.parse(record_path, format=fmt)

    # Fail fast if the record contains no dcat:Dataset at all.
    datasets = list(g.subjects(predicate=None, object=DCAT.Dataset))
    if not any(g.triples((s, None, DCAT.Dataset)) for s in g.subjects()):
        # Fall back to an explicit type check.
        typed = list(g.subjects(object=DCAT.Dataset))
        if not typed:
            raise ValueError(
                f"{record_path.name} contains no dcat:Dataset node; "
                "check the @type and namespace mapping in the emitter."
            )
    logger.info("Loaded %d triples from %s", len(g), record_path.name)
    return g

If parsing raises or the graph has zero dcat:Dataset subjects, the problem is upstream in the emitter’s @context — a namespace typo that JSON-LD expansion would have caught. Validate expansion first, as described in the parent page’s CI section, then bring the record here.

Step 2: Load the official DCAT-AP SHACL shapes

The DCAT-AP shapes are published by the European Commission’s SEMIC programme as a set of Turtle files. Vendor a specific version into the repository and load it as its own graph; pinning is what makes validation reproducible, because the shapes evolve between DCAT-AP releases and a live fetch would silently change your acceptance criteria.

python
# load_shapes.py  — rdflib >=7.0
from pathlib import Path
from rdflib import Graph


def load_shapes(shapes_path: Path) -> Graph:
    """Load the pinned DCAT-AP SHACL shapes graph."""
    if not shapes_path.exists():
        raise FileNotFoundError(
            f"DCAT-AP shapes not vendored at {shapes_path}. "
            "Download the pinned release and commit it; do not fetch live."
        )
    sg = Graph()
    sg.parse(shapes_path, format="turtle")
    return sg

Record the exact shapes version in the repository (in a SHAPES_VERSION constant or the filename) so the validation report can state which contract a record was judged against. A record that conforms to DCAT-AP 2.1.1 may fail DCAT-AP 3.0; the version is part of the compliance claim.

Step 3: Run pyshacl and capture the report

pyshacl.validate takes the data graph and the shapes graph and returns a triple: a boolean conforms, the report as its own RDF graph, and a human-readable text report. Capture all three — the boolean gates the build, the report graph is the machine-readable audit record, and the text is for the engineer reading the CI log.

python
# validate.py  — pyshacl >=0.25, rdflib >=7.0
from __future__ import annotations
import logging
from dataclasses import dataclass
from pathlib import Path

from pyshacl import validate
from rdflib import Graph

logger = logging.getLogger(__name__)


@dataclass
class ValidationOutcome:
    conforms: bool
    report_graph: Graph
    report_text: str


def validate_record(data_graph: Graph, shapes_graph: Graph) -> ValidationOutcome:
    """Validate a DCAT-AP record against the pinned SHACL shapes."""
    conforms, report_graph, report_text = validate(
        data_graph,
        shacl_graph=shapes_graph,
        inference="rdfs",       # apply RDFS entailment so subclass types resolve
        abort_on_first=False,   # collect every violation, not just the first
        meta_shacl=False,       # shapes are pre-validated; skip meta-validation
        advanced=True,          # enable SHACL-AF features used by DCAT-AP shapes
    )
    if conforms:
        logger.info("Record conforms to DCAT-AP shapes.")
    else:
        logger.error("Record does NOT conform; %d-char report follows.", len(report_text))
    return ValidationOutcome(conforms, report_graph, report_text)

Set abort_on_first=False so a single run surfaces every problem; aborting on the first violation forces a slow fix-one-rerun loop. Keep inference="rdfs" on, because the DCAT-AP shapes target dcat:Dataset and your record may declare a more specific subclass that only resolves under RDFS entailment.

Step 4: Interpret and triage violations

A non-conforming report is an RDF graph of sh:ValidationResult nodes. Each result names a sh:focusNode (the offending subject, usually your dataset URI), a sh:resultPath (the property at fault, such as dcterms:license), a sh:resultSeverity, and a sh:resultMessage. Query these into a flat list so the CI log points straight at the field to fix.

python
# interpret.py  — rdflib >=7.0
from rdflib import Graph
from rdflib.namespace import SH


def summarize_violations(report_graph: Graph) -> list[dict[str, str]]:
    """Flatten sh:ValidationResult nodes into an actionable list."""
    rows: list[dict[str, str]] = []
    for result in report_graph.subjects(predicate=None, object=SH.ValidationResult):
        rows.append(
            {
                "focus_node": str(report_graph.value(result, SH.focusNode) or ""),
                "path": str(report_graph.value(result, SH.resultPath) or ""),
                "severity": str(report_graph.value(result, SH.resultSeverity) or "").rsplit("#", 1)[-1],
                "message": str(report_graph.value(result, SH.resultMessage) or ""),
            }
        )
    # Sort Violations before Warnings so the blocker is first in the log.
    rows.sort(key=lambda r: (r["severity"] != "Violation", r["path"]))
    return rows

Distinguish severities. A sh:Violation means the record breaks the DCAT-AP contract and a portal will reject it — this must block publication. A sh:Warning (for example a missing recommended dcat:keyword) signals a quality gap that should be tracked but does not have to block a merge. Mapping each violation back to a manifest field is exactly the field-name reconciliation discipline that Field Renaming & Type Coercion Rules applies to attribute data.

Step 5: Gate publication on conformance

Wire the outcome into a single gate that fails the build on any Violation and writes the report to an auditable path. This is the point where DCAT-AP validation becomes a hard precondition for publishing rather than advisory.

python
# gate.py  — pyshacl >=0.25, rdflib >=7.0
import json
import sys
from pathlib import Path

from load_graph import load_record
from load_shapes import load_shapes
from validate import validate_record
from interpret import summarize_violations


def gate(record_path: Path, shapes_path: Path, report_path: Path) -> int:
    data = load_record(record_path)
    shapes = load_shapes(shapes_path)
    outcome = validate_record(data, shapes)
    rows = summarize_violations(outcome.report_graph)

    report_path.write_text(json.dumps(rows, indent=2), encoding="utf-8")
    violations = [r for r in rows if r["severity"] == "Violation"]
    if violations:
        for r in violations:
            print(f"VIOLATION {r['path']} on {r['focus_node']}: {r['message']}")
        return 1
    print("DCAT-AP conformance: PASS")
    return 0


if __name__ == "__main__":
    sys.exit(gate(Path(sys.argv[1]), Path(sys.argv[2]), Path("dist/dcat-report.json")))
yaml
# .github/workflows/dcat-ap-gate.yml
name: DCAT-AP Conformance Gate
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install "rdflib>=7.0" "pyshacl>=0.25"
      - name: Validate DCAT-AP record against pinned shapes
        run: python -m gate dist/parcels-2024.jsonld shapes/dcat-ap-3.0-shacl.ttl

Because the shapes are pinned and the record is deterministic, this gate is stable across runs — it fails only when the metadata genuinely breaks the contract. Aggregating this pass/fail into a repository-wide conformance signal, and blocking merges on drift over time, is the job of the CI Validation Scorecards cluster, which consumes the JSON report this gate writes.

Verification

Confirm the gate distinguishes a good record from a bad one — a validator that passes everything is worse than none. Prove it with a deliberately broken fixture:

python
# test_gate.py  — pytest >=7, rdflib >=7.0, pyshacl >=0.25
from pathlib import Path
from load_graph import load_record
from load_shapes import load_shapes
from validate import validate_record
from interpret import summarize_violations

SHAPES = load_shapes(Path("shapes/dcat-ap-3.0-shacl.ttl"))


def test_valid_record_conforms():
    outcome = validate_record(load_record(Path("tests/fixtures/valid.jsonld")), SHAPES)
    assert outcome.conforms, outcome.report_text


def test_missing_license_is_a_violation():
    """A record with no dcterms:license must fail with a Violation."""
    outcome = validate_record(load_record(Path("tests/fixtures/no_license.jsonld")), SHAPES)
    assert not outcome.conforms
    paths = {r["path"] for r in summarize_violations(outcome.report_graph)}
    assert any("license" in p for p in paths), "license violation not reported"

A healthy run passes test_valid_record_conforms and fails the broken fixture with a license violation. On the CLI, python -m gate dist/parcels-2024.jsonld shapes/dcat-ap-3.0-shacl.ttl prints DCAT-AP conformance: PASS and exits 0 for a good record, and prints one VIOLATION line per broken field and exits 1 otherwise.

Troubleshooting

Symptom Likely cause Fix
conforms is True but the portal still rejects the record Validated against an older shapes version than the portal uses Pin the shapes version the portal declares; re-run Step 2 with the matching release.
Every dataset reports a sh:Violation on type dcat:Dataset type not asserted because of a @context namespace typo Fix the DCAT namespace in the emitter; confirm JSON-LD expansion first, then reload in Step 1.
pyshacl raises ReportableRuntimeError on advanced features advanced=True omitted while the DCAT-AP shapes use SHACL-AF Pass advanced=True to validate as in Step 3.
License violation despite a license being present License emitted as a literal string, not an IRI node Emit dcterms:license as an @id object, not a plain string; re-check the manifest URI.
Validation passes locally, fails in CI Shapes fetched live locally but vendored (stale) in CI, or vice versa Vendor a single pinned shapes file and reference only that path everywhere.
Spatial extent flagged as invalid dcterms:spatial missing the expected dcat:bbox or locn:geometry typing Emit the location node with the geometry serialization the pinned shapes require.