Geospatial Schema Standardization & Mapping Automation

Publishing to CKAN with Automated DCAT Harvest Endpoints

CKAN is the most common home for government spatial data and its web form is the most common way data gets into it — which means the metadata a harvester sees is typically whatever someone typed while thinking about something else. Driving CKAN from the publication manifest instead makes the catalogue a derived artefact: reproducible, diffable, and identical whether the release was made by a person or by a scheduled job. This procedure implements the portal-facing half of open data portal publication workflows, inside Geospatial Compliance Reporting & Audit Trails.

The steps map to configure (Step 1, the object model), execute (Steps 2–3, dataset and resources), validate (Step 4, the DCAT extras), and log (Step 5, verification of what harvesters actually see).

Prerequisites checklist

Step 1: Decide the CKAN object model before writing code

CKAN’s model is organization → dataset (package) → resource, and DCAT-AP’s is catalog → dataset → distribution. The mapping that works and keeps citations stable is:

Manifest concept CKAN object Note
dataset_id package name The URL slug; permanent, never versioned
Release version package version field, plus a resource-level tag Not in the slug, or every release breaks the landing page
Distribution resource One per format, each with its own checksum
Conformance report resource with format: JSON A first-class artefact, not a link buried in the notes
Landing page package URL CKAN provides it; do not duplicate it in an extra
Manifest to CKAN Package to DCAT Distribution Three columns line up the same objects under three vocabularies. The manifest's dataset identifier becomes CKAN's package name and DCAT's dataset; the release version becomes CKAN's version field and appears in each resource label rather than in the URL; each distribution becomes one CKAN resource and one DCAT distribution carrying its media type and checksum; and the conformance report becomes a resource of its own rather than a link buried in the description. A note records that versioning at the resource level rather than in the package slug is what keeps citations and bookmarks resolving across releases. publication manifest CKAN DCAT-AP dataset_id package name (slug) dcat:Dataset permanent — never versioned release.version version field + resource label dcterms:hasVersion not in the slug — citations depend on it distribution + checksum resource (hash, mimetype, size) dcat:Distribution conformance report its own JSON resource dcterms:conformsTo

The temptation is to make each release its own CKAN dataset so the version appears in the URL. Resist it: consumers bookmark and cite the dataset, and a new slug every quarter fragments the audience and orphans every citation. Version at the resource level, keep old resources listed, and let the package remain the permanent identity.

Step 2: Upsert the dataset idempotently

python
# publish/ckan.py — requests >=2.31 — Python 3.10+
import logging

import requests

logger = logging.getLogger("publish.ckan")
TIMEOUT = 60


class Ckan:
    def __init__(self, base_url: str, token: str):
        self.base = base_url.rstrip("/")
        self.session = requests.Session()
        self.session.headers["Authorization"] = token

    def call(self, action: str, payload: dict) -> dict:
        response = self.session.post(f"{self.base}/api/3/action/{action}",
                                     json=payload, timeout=TIMEOUT)
        if response.status_code == 404 and action == "package_show":
            return {}
        response.raise_for_status()
        body = response.json()
        if not body.get("success"):
            raise RuntimeError(f"{action} failed: {body.get('error')}")
        return body["result"]


def upsert_dataset(ckan: Ckan, manifest: dict) -> dict:
    meta = manifest["metadata"]
    payload = {
        "name": manifest["dataset_id"],                 # stable slug
        "title": meta["title"],
        "notes": meta["description"],
        "license_id": meta["licence"],
        "owner_org": meta["organization"],
        "version": manifest["release"]["version"],
        "maintainer": meta["contact_point"]["name"],
        "maintainer_email": meta["contact_point"]["email"],
        "extras": dcat_extras(manifest),                # Step 4
    }
    existing = ckan.call("package_show", {"id": manifest["dataset_id"]})
    if existing:
        payload["id"] = existing["id"]
        result = ckan.call("package_update", payload)
        logger.info("updated dataset %s to version %s", payload["name"], payload["version"])
    else:
        result = ckan.call("package_create", payload)
        logger.info("created dataset %s", payload["name"])
    return result

package_update replaces the whole package, so any field omitted from the payload is cleared. That is a feature when the manifest is the source of truth — it means a field deleted from the manifest disappears from the portal, and the portal cannot drift — but it destroys anything a human added through the web interface. Say so explicitly in the manifest’s documentation: the portal record is generated, and manual edits will be overwritten on the next release.

Step 3: Create or update resources with their checksums

python
# publish/ckan.py — Python 3.10+
from pathlib import Path


def upsert_resources(ckan: Ckan, package: dict, manifest: dict) -> list[dict]:
    existing = {r["name"]: r for r in package.get("resources", [])}
    results = []

    for dist in manifest["distributions"]:
        path = Path(dist["path"])
        name = f"{dist['format']} — {manifest['release']['version']}"
        payload = {
            "package_id": package["id"],
            "name": name,
            "format": dist["format"],
            "mimetype": dist["media_type"],
            "size": path.stat().st_size,
            "hash": dist["checksum_value"],
            "hash_algorithm": "sha256",
            "description": dist.get("label", ""),
        }
        if name in existing:
            payload["id"] = existing[name]["id"]
            with path.open("rb") as handle:
                results.append(ckan.upload("resource_update", payload, handle))
        else:
            with path.open("rb") as handle:
                results.append(ckan.upload("resource_create", payload, handle))
        logger.info("resource %s (%s, %d bytes)", name, dist["checksum_value"][:12],
                    payload["size"])
    return results

Naming the resource with its release version is what makes the history legible on the portal page: a consumer can see that GeoPackage — 2026.2 is current and GeoPackage — 2026.1 is the previous release, without reading a changelog. Publishing hash and hash_algorithm gives a consumer the means to verify the download, and ckanext-dcat maps both into spdx:checksum in the RDF.

Step 4: Populate the extras that DCAT-AP harvesters actually read

python
# publish/ckan.py — Python 3.10+
import json


def dcat_extras(manifest: dict) -> list[dict]:
    meta = manifest["metadata"]
    minx, miny, maxx, maxy = meta["spatial_extent"]
    return [
        # ckanext-spatial reads this GeoJSON to build the spatial index and dct:spatial.
        {"key": "spatial", "value": json.dumps({
            "type": "Polygon",
            "coordinates": [[[minx, miny], [maxx, miny], [maxx, maxy], [minx, maxy], [minx, miny]]],
        })},
        {"key": "spatial_uri", "value": meta.get("spatial_uri", "")},
        {"key": "temporal_start", "value": meta["temporal_extent"][0]},
        {"key": "temporal_end", "value": meta["temporal_extent"][1]},
        {"key": "conforms_to", "value": manifest["metadata"]["conformance_report_uri"]},
        {"key": "access_rights", "value": meta.get("access_rights", "public")},
        {"key": "provenance", "value": meta["provenance_statement"]},
        {"key": "theme", "value": "http://publications.europa.eu/resource/authority/data-theme/REGI"},
    ]

Two of these are worth insisting on. The spatial extra must be a GeoJSON geometry in EPSG:4326 with longitude first — a bounding box typed in the projected CRS of the data, or with the axes swapped, produces a dataset that indexes at the wrong place on the map and never appears in a spatial search of its own area. And provenance is where the pipeline earns credit for everything upstream: a one-sentence statement naming the source, the transformation and the conformance report, generated from the lineage manifest rather than typed.

Step 5: Verify what a harvester will actually see

Publishing is not finished when the API returns 200. What matters is the RDF a harvester fetches.

python
# publish/verify.py — rdflib >=7.0, requests >=2.31 — Python 3.10+
from rdflib import Graph, Namespace, URIRef
from rdflib.namespace import DCAT, DCTERMS

def verify_dcat(base_url: str, dataset_id: str, manifest: dict) -> None:
    graph = Graph()
    graph.parse(f"{base_url}/dataset/{dataset_id}.rdf", format="xml")

    datasets = list(graph.subjects(predicate=None, object=DCAT.Dataset))
    assert datasets, "no dcat:Dataset in the published RDF"
    subject = datasets[0]

    distributions = list(graph.objects(subject, DCAT.distribution))
    assert len(distributions) == len(manifest["distributions"]), (
        f"{len(distributions)} distributions in RDF, "
        f"{len(manifest['distributions'])} in the manifest"
    )
    for predicate in (DCTERMS.title, DCTERMS.license, DCTERMS.spatial, DCTERMS.publisher):
        assert (subject, predicate, None) in graph, f"missing {predicate} in published RDF"

Run this as the last step of the release job, and fail the release if it fails. The two failures it catches most often are a distribution that uploaded but did not attach — leaving the RDF advertising fewer files than exist — and a licence that CKAN did not recognize, which silently produces a dataset with no dcterms:license and therefore no legal reuse terms for any harvester downstream.

Verification

bash
# The catalogue endpoint a harvester polls
curl -s https://data.example.gov/catalog.rdf | grep -c "dcat:Dataset"

# One dataset, as RDF and as DCAT-AP JSON-LD
curl -s https://data.example.gov/dataset/cadastral-parcels.rdf | head -40
curl -s https://data.example.gov/dataset/cadastral-parcels.jsonld | jq '.["@graph"][0].title'

A successful release logs the release version, every resource with a truncated hash, and the RDF assertion result:

text
INFO publish.ckan updated dataset cadastral-parcels to version 2026.2
INFO publish.ckan resource GeoPackage — 2026.2 (9f2a41c0b8d3, 481,203,776 bytes)
INFO publish.ckan resource GeoJSON — 2026.2 (2c7e08ab19f5, 1,204,882,001 bytes)
INFO publish.verify dcat: 3 distributions, licence and spatial present — ok

Troubleshooting

Symptom Likely cause Fix
Dataset does not appear in a spatial search spatial extra in the projected CRS, or axes swapped Emit GeoJSON in EPSG:4326, longitude first; ckanext-spatial does not reproject
Harvester reports no licence license_id not one of CKAN’s registered ids Use an id from the portal’s licence list, not a free-text name
Manual portal edits keep disappearing package_update replaces the whole package, as designed Move the field into the manifest; the portal record is generated
Duplicate resources after every release Resource matching by name failed because the name embeds a timestamp Match on a stable resource name and carry the version in the label
RDF shows fewer distributions than uploaded A resource upload failed after the dataset update Fail the release on the RDF assertion; re-run — the upsert is idempotent
Harvest is stale for days The harvester polls on its own schedule Publish the catalogue endpoint update in the same run and, where supported, trigger the harvest explicitly