Publish Machine-Readable Dataset Metadata as JSON-LD Jump to heading#

A geospatial dataset that no crawler can parse is invisible to open-data portals and useless as audit evidence. The fix is a deterministic metadata document, emitted alongside every published dataset, that states what the data covers, where it came from, and under what license it may be used — in a form both Google Dataset Search and a compliance auditor can read without a human in the loop. That document is JSON-LD: schema.org Dataset for discovery, DCAT and DCAT-AP for interoperability with national data catalogues. This stage operates at the core of Geospatial Compliance Reporting & Audit Trails, the practice area that turns each transform and validation event into queryable, regulator-ready evidence.

This page owns the metadata contract and the engine that serializes it. It does not own the transform records themselves — those are produced upstream by Lineage Manifest Generation, and this stage only references them through provenance links. It also stops short of the merge-gating scorecards published by CI Validation Scorecards; here CI merely proves the metadata document itself is well-formed. Two focused walkthroughs sit beneath this page: building a portal-ready schema.org document in emitting schema.org Dataset JSON-LD for open data portals, and shape-checking DCAT-AP in validating DCAT-AP metadata with SHACL shapes.

Scope: what a dataset metadata document must assert Jump to heading#

Machine-readable metadata is not free-text documentation. Every published dataset must assert, in a machine-checkable form, five categories of fact:

• Identity — a stable identifier (a DOI or a persistent portal URI), a title, and a description that survive re-publication.
• Coverage — the spatial extent as a bounding box or polygon, the CRS that box is expressed in, and the temporal window the observations cover.
• Access — one or more distributions, each a concrete downloadable representation with a media type, byte size, and access URL.
• Rights — a license expressed as a resolvable URI, not a prose sentence, so that reuse conditions are unambiguous.
• Provenance — a wasDerivedFrom link to the source that produced the dataset, which for pipelines in this practice area is the append-only lineage manifest.

schema.org Dataset covers identity, coverage, access, and rights directly. DCAT-AP, the European application profile of DCAT, adds the catalogue-record semantics that national portals harvest — dcat:distribution, dcterms:license, dcterms:spatial, dcterms:temporal, and dcat:theme. The two vocabularies overlap heavily; the engine below builds both from one source of truth so they can never drift.

Declarative metadata manifest Jump to heading#

The parts of a dataset’s metadata that a machine cannot infer — who published it, under which license, which theme it belongs to, and which lineage manifest proves how it was built — belong in a version-controlled YAML manifest. Everything a machine can infer (bounding box, CRS, row count, temporal extent) is extracted from the dataset itself in the next section, never hand-entered, so it cannot fall out of sync with the data.

# dataset_metadata.yaml  — pyyaml >=6.0
dataset:
  identifier: "https://data.example.gov/id/parcels-2024"   # MANDATORY: stable URI or DOI
  name: "Municipal Parcel Boundaries 2024"                  # MANDATORY: human title
  description: >                                            # MANDATORY: 50-5000 chars
    Authoritative cadastral parcel boundaries for the municipality,
    normalized to EPSG:4326 and reconciled against the county roll.
  publisher:
    name: "City GIS Office"                                 # MANDATORY
    uri: "https://data.example.gov/org/city-gis"            # MANDATORY: publisher URI
  license: "https://creativecommons.org/licenses/by/4.0/"  # MANDATORY: license URI, not prose
  themes:                                                   # MANDATORY: min 1 DCAT-AP theme
    - "http://publications.europa.eu/resource/authority/data-theme/REGI"
  keywords: ["cadastre", "parcels", "boundaries"]          # OPTIONAL
  contact_point:                                            # OPTIONAL but required by DCAT-AP
    name: "GIS Data Steward"
    email: "[email protected]"
  provenance:
    lineage_manifest_uri: "s3://audit/parcels-2024/lineage.parquet"  # MANDATORY: wasDerivedFrom
    derived_from_name: "County Assessor Roll Q1 2024"      # OPTIONAL: human label for source
  distributions:                                            # MANDATORY: min 1 entry
    - access_url: "https://data.example.gov/dl/parcels-2024.gpkg"
      media_type: "application/geopackage+sqlite3"
      format: "GPKG"
    - access_url: "https://data.example.gov/dl/parcels-2024.parquet"
      media_type: "application/vnd.apache.parquet"
      format: "GEOPARQUET"
  dataset_path: "/data/parcels-2024.gpkg"                   # MANDATORY: local path for coverage extraction

Mandatory vs optional field reference:

Treat this manifest as part of the schema contract. Renaming identifier after publication breaks every catalogue that has already harvested the record, so the identifier is immutable once a dataset is public. The consistency rules — matching field names against the lineage record, using resolvable license URIs — mirror the naming discipline enforced by Field Renaming & Type Coercion Rules in the ETL pillar.

Preprocessing: extracting coverage from the dataset Jump to heading#

Coverage facts must be derived from the data, not typed by a human, because the whole point of the metadata is to make an auditable claim about what the dataset actually contains. Extract the bounding box, the CRS, and the temporal window directly, and reproject the bounding box to EPSG:4326 so the emitted GeoShape is expressed in the WGS84 lon/lat that both schema.org and DCAT-AP expect.

# coverage.py  — geopandas >=0.14, pyproj >=3.6
from __future__ import annotations
import logging
from dataclasses import dataclass
from pathlib import Path

import geopandas as gpd
from pyproj import CRS, Transformer
from pyproj.exceptions import CRSError

logger = logging.getLogger(__name__)


@dataclass
class Coverage:
    west: float
    south: float
    east: float
    north: float
    source_epsg: int | None
    time_start: str | None   # ISO 8601 date
    time_end: str | None


def extract_coverage(
    dataset_path: Path,
    time_field: str | None = None,
) -> Coverage:
    """
    Derive spatial and temporal coverage from the dataset itself.
    Bounding box is always returned in EPSG:4326 (lon/lat) for JSON-LD.
    """
    gdf = gpd.read_file(dataset_path)
    if gdf.crs is None:
        raise ValueError(
            f"{dataset_path.name} has no CRS; coverage cannot be trusted. "
            "Normalize CRS before publishing metadata."
        )

    try:
        src = CRS.from_user_input(gdf.crs)
    except CRSError as exc:
        logger.critical("Unreadable CRS on %s: %s", dataset_path.name, exc)
        raise

    minx, miny, maxx, maxy = gdf.total_bounds
    if src.to_epsg() != 4326:
        # always_xy=True: emit (lon, lat), never authority lat-first order.
        tf = Transformer.from_crs(src, CRS.from_epsg(4326), always_xy=True)
        minx, miny = tf.transform(minx, miny)
        maxx, maxy = tf.transform(maxx, maxy)

    t_start = t_end = None
    if time_field and time_field in gdf.columns:
        ts = gpd.pd.to_datetime(gdf[time_field], errors="coerce").dropna()
        if not ts.empty:
            t_start = ts.min().date().isoformat()
            t_end = ts.max().date().isoformat()

    return Coverage(
        west=round(minx, 6), south=round(miny, 6),
        east=round(maxx, 6), north=round(maxy, 6),
        source_epsg=src.to_epsg(),
        time_start=t_start, time_end=t_end,
    )

The reproject to EPSG:4326 with always_xy=True is not optional. A bounding box emitted in the dataset’s native projected CRS, or in authority latitude-first order, is silently wrong to any harvester that assumes WGS84 lon/lat — the same axis-order failure mode documented in Projection Normalization Workflows. Rounding to six decimal places (roughly 0.1 m at the equator) keeps the emitted document byte-stable across runs so that an unchanged dataset produces an unchanged metadata file — a prerequisite for the CI diffing described below.

Execution engine: building and serializing the JSON-LD Jump to heading#

With the manifest loaded and coverage extracted, the engine assembles two nodes into one JSON-LD document and serializes them with the standard-library json module. Using json.dumps with sort_keys=True and a fixed separator produces a deterministic, diffable document; never hand-format JSON-LD with f-strings, which is how invalid @context values and unescaped strings reach production.

# emit_jsonld.py  — Python 3.10+, uses only the standard library json module
from __future__ import annotations
import json
import logging
from pathlib import Path
from typing import Any

from coverage import Coverage

logger = logging.getLogger(__name__)

# A single @context maps both schema.org and DCAT-AP terms. Order is irrelevant
# to a JSON-LD processor but fixed here so the serialized bytes are stable.
CONTEXT: dict[str, Any] = {
    "@vocab": "https://schema.org/",
    "dcat": "http://www.w3.org/ns/dcat#",
    "dcterms": "http://purl.org/dc/terms/",
    "prov": "http://www.w3.org/ns/prov#",
}


def build_dataset_node(meta: dict[str, Any], cov: Coverage) -> dict[str, Any]:
    """Assemble the schema.org Dataset node with spatial/temporal coverage."""
    node: dict[str, Any] = {
        "@type": "Dataset",
        "@id": meta["identifier"],
        "identifier": meta["identifier"],
        "name": meta["name"],
        "description": meta["description"].strip(),
        "license": meta["license"],
        "publisher": {
            "@type": "Organization",
            "@id": meta["publisher"]["uri"],
            "name": meta["publisher"]["name"],
        },
        "spatialCoverage": {
            "@type": "Place",
            "geo": {
                "@type": "GeoShape",
                # schema.org box order: "south west north east"
                "box": f"{cov.south} {cov.west} {cov.north} {cov.east}",
            },
        },
        "distribution": [
            {
                "@type": "DataDownload",
                "contentUrl": d["access_url"],
                "encodingFormat": d["media_type"],
            }
            for d in meta["distributions"]
        ],
        "prov:wasDerivedFrom": {"@id": meta["provenance"]["lineage_manifest_uri"]},
    }
    if cov.time_start and cov.time_end:
        node["temporalCoverage"] = f"{cov.time_start}/{cov.time_end}"
    if meta.get("keywords"):
        node["keywords"] = meta["keywords"]
    return node


def build_dcat_node(meta: dict[str, Any], cov: Coverage) -> dict[str, Any]:
    """Assemble the DCAT-AP dcat:Dataset node harvested by national portals."""
    return {
        "@type": "dcat:Dataset",
        "@id": meta["identifier"],
        "dcterms:title": meta["name"],
        "dcterms:description": meta["description"].strip(),
        "dcterms:publisher": {"@id": meta["publisher"]["uri"]},
        "dcterms:license": {"@id": meta["license"]},
        "dcat:theme": [{"@id": t} for t in meta["themes"]],
        "dcterms:spatial": {
            "@type": "dcterms:Location",
            "dcat:bbox": f"{cov.west},{cov.south},{cov.east},{cov.north}",
        },
        "dcat:distribution": [
            {
                "@type": "dcat:Distribution",
                "dcat:accessURL": {"@id": d["access_url"]},
                "dcat:mediaType": d["media_type"],
            }
            for d in meta["distributions"]
        ],
        "prov:wasDerivedFrom": {"@id": meta["provenance"]["lineage_manifest_uri"]},
    }


def emit_document(meta: dict[str, Any], cov: Coverage, out_path: Path) -> Path:
    """Serialize the combined graph deterministically."""
    document = {
        "@context": CONTEXT,
        "@graph": [build_dataset_node(meta, cov), build_dcat_node(meta, cov)],
    }
    text = json.dumps(document, indent=2, sort_keys=True, ensure_ascii=False)
    out_path.write_text(text + "\n", encoding="utf-8")
    logger.info("Wrote %d-byte JSON-LD document to %s", len(text), out_path)
    return out_path

Two details carry most of the correctness burden. First, the box string in a schema.org GeoShape is ordered south west north east (latitude first), while DCAT’s dcat:bbox is ordered west south east north (longitude first) — mixing them up is the single most common metadata bug, and the two builders deliberately use different orderings against the same Coverage values. Second, prov:wasDerivedFrom carries an @id pointing at the lineage manifest URI, which is what makes the published metadata an entry point into the audit trail rather than a dead-end description. The detailed schema.org build, including creator and includedInDataCatalog, is walked through in emitting schema.org Dataset JSON-LD for open data portals.

Failure modes Jump to heading#

Metadata failures are quiet: a malformed document does not crash the pipeline, it just fails to be harvested weeks later, or worse, publishes a false claim about coverage. Enumerate the failure modes and give each a deterministic guard.

The wasDerivedFrom reachability check deserves emphasis. A provenance link that resolves to nothing is worse than no link, because it presents the appearance of an audit trail while providing none. Before publishing, confirm the referenced lineage manifest exists — the format and generation of that manifest is owned by writing append-only lineage manifests to Parquet.

Compliance reporting output Jump to heading#

The emitted JSON-LD is itself a compliance artifact, but the publication event must also be logged to the audit trail, exactly as CRS transforms and attribute coercions are. Every metadata emission writes a structured record so an auditor can reconstruct which version of the metadata was live at any point in time.

# metadata_audit.py  — pyarrow >=14
import datetime
import hashlib
import json
from pathlib import Path


def write_publication_record(audit_path: Path, meta: dict, doc_path: Path) -> None:
    """Append one publication-event record to the metadata audit NDJSON."""
    doc_bytes = doc_path.read_bytes()
    record = {
        "identifier": meta["identifier"],
        "doc_sha256": hashlib.sha256(doc_bytes).hexdigest(),
        "doc_bytes": len(doc_bytes),
        "license": meta["license"],
        "lineage_manifest_uri": meta["provenance"]["lineage_manifest_uri"],
        "n_distributions": len(meta["distributions"]),
        "published_utc": datetime.datetime.now(datetime.UTC).isoformat(),
        "outcome": "PUBLISHED",
    }
    with audit_path.open("a") as fh:
        fh.write(json.dumps(record) + "\n")

The record’s doc_sha256 is the load-bearing field: it lets an auditor prove that the metadata harvested by a portal on a given date is byte-identical to what the pipeline generated, closing the gap between “we published metadata” and “we can prove exactly what we published.” Retain these records for the full statutory window — INSPIRE obligations run to six years, and ISO 19115 lineage records are expected to persist for the life of the dataset. This publication log slots into the same audit store as the lineage rows generated upstream, and its per-event schema mirrors the LI_Lineage statements produced by generating ISO 19115 lineage statements automatically.

CI integration Jump to heading#

The JSON-LD document must be validated on every build, before it can reach a portal. Validation has three layers: it must be syntactically valid JSON, it must expand cleanly as a JSON-LD graph (which catches every @context and namespace error), and its DCAT-AP node must satisfy the official SHACL shapes. The first two run in milliseconds with no network; the third is detailed in validating DCAT-AP metadata with SHACL shapes.

# tests/test_jsonld.py  — pytest >=7, pyld >=2.0
import json
from pathlib import Path

import pytest
from pyld import jsonld

DOC = Path("dist/parcels-2024.jsonld")


def test_document_is_valid_json():
    json.loads(DOC.read_text())  # raises on malformed JSON


def test_document_expands_without_context_errors():
    """Expansion resolves every @context term; a bad namespace raises here."""
    doc = json.loads(DOC.read_text())
    expanded = jsonld.expand(doc)
    assert expanded, "expansion produced an empty graph"


def test_geoshape_box_is_south_west_north_east():
    doc = json.loads(DOC.read_text())
    dataset = next(n for n in doc["@graph"] if n["@type"] == "Dataset")
    box = dataset["spatialCoverage"]["geo"]["box"].split()
    assert len(box) == 4, "GeoShape box must have four ordinates"
    south, west, north, east = map(float, box)
    assert south <= north and west <= east, "box ordinates are transposed"
    assert -90 <= south <= 90 and -180 <= west <= 180, "box not in EPSG:4326"

# .github/workflows/validate-metadata.yml
name: Validate Dataset Metadata
on: [push, pull_request]

jobs:
  validate-jsonld:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install "geopandas>=0.14" "pyproj>=3.6" "pyld>=2.0" "pyshacl>=0.25" "pytest>=7"
      - name: Build metadata document
        run: python -m emit_jsonld dataset_metadata.yaml dist/parcels-2024.jsonld
      - name: Validate JSON-LD expansion and coverage
        run: pytest tests/test_jsonld.py -v --tb=short
      - name: Validate DCAT-AP against SHACL shapes
        run: python -m validate_dcat_ap dist/parcels-2024.jsonld shapes/dcat-ap-shacl.ttl

Gate this workflow on any change to the metadata manifest, the coverage extractor, or the emit engine. Because the emitted document is deterministic, a second CI step can diff the freshly built document against the committed one and fail if they differ, guaranteeing the published metadata always matches its source. Whether that gate blocks a merge — as opposed to merely reporting — is the concern of CI Validation Scorecards, which aggregates this check with schema-conformance checks into a single published scorecard.

Frequently Asked Questions Jump to heading#

Why emit both schema.org Dataset and DCAT-AP instead of picking one? They serve different harvesters. Google Dataset Search reads schema.org Dataset embedded in the landing page; European national portals and the EU data portal harvest DCAT-AP. Emitting only one strands half your audience. Because the engine builds both from a single manifest and a single extracted coverage, the marginal cost of the second vocabulary is near zero, and there is no risk of the two records making contradictory claims.

Should the JSON-LD be embedded in the HTML page or served as a standalone file? Both. Google Dataset Search discovers the schema.org block when it is embedded in the dataset landing page inside a <script type="application/ld+json"> tag, so that embedding is what earns discovery. A standalone .jsonld file at a stable URL is what CKAN and DCAT-AP catalogue harvesters fetch and what your CI validates. Generate one document and publish it in both places rather than maintaining two.

How does provenance in the metadata relate to the lineage manifest? The metadata’s wasDerivedFrom is a pointer, not a copy. It carries the URI of the append-only lineage manifest that records how the dataset was actually built — the CRS transforms, the rejected features, the tolerance decisions. Keeping the full lineage in the manifest and only a reference in the metadata keeps the published document small while preserving a verifiable path to the complete audit trail, which is generated by Lineage Manifest Generation.

What license format do portals actually require? A resolvable URI, not prose. https://creativecommons.org/licenses/by/4.0/ is machine-checkable; “CC-BY, see terms” is not, and it fails DCAT-AP SHACL validation. Store the license as a URI in the manifest and let the engine emit it as an @id. If your organization uses a custom license, publish that license at a stable URL and reference the URL.

Related Jump to heading#

• Lineage Manifest Generation — the append-only transform record that this stage references through wasDerivedFrom
• CI Validation Scorecards — turning the metadata validation gate into a merge-blocking published scorecard
• Emitting schema.org Dataset JSON-LD for Open Data Portals — the runnable build for Google Dataset Search and CKAN harvest
• Validating DCAT-AP Metadata with SHACL Shapes — shape-checking the DCAT-AP node before publication
• Geospatial Schema Architecture & Standards Mapping — the ISO 19115 and metadata-standard foundations the coverage and provenance fields map back to