Open Data Portal Publication Workflows Jump to heading#

Publication is where a pipeline meets the public, and it is usually the least engineered stage in the whole chain. A dataset that was validated deterministically, transformed reproducibly and audited to the feature is then uploaded by a person clicking through a portal form, with a description typed from memory and a date that is whatever day the upload happened. Everything upstream can be perfect and the published artefact will still misstate its own extent, contradict its metadata, or quietly replace last quarter’s file at the same URL so that a citation from a published report now resolves to different data. This stage makes publication a repeatable, audited step of the pipeline, inside Geospatial Compliance Reporting & Audit Trails.

The boundary against neighbouring topics is clean. Dataset metadata and JSON-LD publishing owns what the metadata says — the schema.org and DCAT-AP serializations and their validation. This page owns getting it and the data onto a portal repeatably: which artefacts constitute a release, how they are addressed, how a harvester finds them, and when they expire. Lineage manifest generation supplies the provenance that publication cites.

Declarative Configuration Manifest Jump to heading#

# publication.yaml — requests >=2.31, rdflib >=7.0
manifest_version: "2.2.0"        # MANDATORY
dataset_id: "cadastral-parcels"  # MANDATORY: stable across every release, forever
landing_page: "https://data.example.gov/datasets/cadastral-parcels"   # MANDATORY: never versioned
release:
  version: "2026.2"              # MANDATORY: the release identity
  versioned_url_template: "https://data.example.gov/datasets/cadastral-parcels/{version}"
  supersedes: "2026.1"           # OPTIONAL: recorded, and the predecessor stays resolvable
distributions:                   # MANDATORY: at least one
  - format: "GeoPackage"
    media_type: "application/geopackage+sqlite3"
    path: "build/parcels_2026_2.gpkg"
    checksum: sha256             # MANDATORY: computed at publish, published with the file
  - format: "GeoJSON"
    media_type: "application/geo+json"
    path: "build/parcels_2026_2.geojson"
    checksum: sha256
  - format: "CSV"                # attributes only; geometry as WKT is stated, not implied
    media_type: "text/csv"
    path: "build/parcels_2026_2.csv"
    checksum: sha256
    geometry_encoding: "WKT in column geom_wkt, EPSG:25832"
required_metadata:               # MANDATORY: refuse to publish without these
  - title
  - description
  - licence
  - spatial_extent
  - temporal_extent
  - contact_point
  - conformance_report_uri       # the quality report from the validation gates
harvest:
  dcat_endpoint: "https://data.example.gov/catalog.rdf"   # MANDATORY
  refresh_schedule: "weekly"
retention:
  keep_versions: 8               # MANDATORY: how many releases stay downloadable
  minimum_retention_years: 6     # MANDATORY: the governing records schedule
  expiry_requires_approval: true # MANDATORY: nothing is deleted by a cron job alone

The pairing of an unversioned landing page with versioned release URLs is the design decision that matters most, and it is the one portals push back on because their default is a single mutable URL per dataset. A report published in 2026 citing “the parcel layer” must still resolve in 2032 — to the 2026 data if the citation was versioned, and to a page explaining what changed if it was not. One mutable URL cannot do both.

Preprocessing Requirements Jump to heading#

Validation has passed and its report is addressable. The conformance_report_uri in the manifest points at the published quality report produced by the validation gates. Publishing data whose quality report is internal-only is publishing an assertion without evidence.

Distributions are built from one source, in one run. The GeoPackage, the GeoJSON and the CSV must be exports of the same dataset state, produced in the same job. Building them at different times is how a portal ends up with a GeoJSON of 214,338 features and a CSV of 214,201, which a user will find and report.

Checksums are computed after the final write, on the exact bytes. Compute after any compression, after any post-processing, on the file as it will be served — not on an intermediate. A checksum that does not match what a consumer downloads is worse than no checksum, because it manufactures distrust of correct data.

The extent and temporal coverage are derived, never typed. Read the bounding box from the data and the temporal range from the delivery, and let the manifest carry only what cannot be computed. Typed extents are wrong surprisingly often, usually by a coordinate order swap.

Execution Engine & Precision Guards Jump to heading#

# publish/portal.py — requests >=2.31 — Python 3.10+
import hashlib
import logging
from pathlib import Path

import requests

logger = logging.getLogger("publish.portal")
TIMEOUT = 60


def sha256_of(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def preflight(manifest: dict) -> None:
    """Everything that can be checked before a byte is uploaded."""
    missing = [f for f in manifest["required_metadata"] if not manifest.get("metadata", {}).get(f)]
    if missing:
        raise ValueError(f"refusing to publish: missing required metadata {missing}")

    for dist in manifest["distributions"]:
        path = Path(dist["path"])
        if not path.is_file():
            raise FileNotFoundError(f"distribution missing: {path}")
        if path.stat().st_size == 0:
            raise ValueError(f"distribution is empty: {path}")
        dist["checksum_value"] = sha256_of(path)

    version = manifest["release"]["version"]
    if release_exists(manifest["dataset_id"], version):
        raise ValueError(
            f"release {version} already published — bump the version rather than overwriting; "
            "published releases are immutable."
        )


def publish(manifest: dict, token: str) -> dict:
    preflight(manifest)
    session = requests.Session()
    session.headers["Authorization"] = f"Bearer {token}"

    created = []
    for dist in manifest["distributions"]:
        response = session.put(
            f"{manifest['release']['versioned_url_template'].format(**manifest['release'])}"
            f"/{Path(dist['path']).name}",
            data=Path(dist["path"]).read_bytes(),
            headers={"Content-Type": dist["media_type"],
                     "Digest": f"sha-256={dist['checksum_value']}"},
            timeout=TIMEOUT,
        )
        response.raise_for_status()
        created.append(response.headers.get("Location", ""))
        logger.info("published %s (%s)", dist["path"], dist["checksum_value"][:12])

    update_landing_page(manifest)         # points at the new release, keeps the archive list
    update_dcat_catalogue(manifest)       # same run: catalogue never lags the data
    return {"release": manifest["release"]["version"], "distributions": created}

Three guards are worth stating explicitly. Preflight validates everything cheap before uploading anything expensive, so a missing licence field fails in a second rather than after a 4 GB upload. A published release is immutable: re-publishing the same version is an error, not an overwrite, because a consumer who downloaded it yesterday must be able to reproduce their result. The catalogue is updated in the same run as the data, because the failure mode of a separate catalogue job is a harvester advertising a release whose files are not there yet.

Failure Modes & Fallback Routing Jump to heading#

Retention deserves the same care as publication. A sweep that removes old releases is the one automated operation in this stage that destroys something, and the manifest’s expiry_requires_approval means the sweep produces a proposal — these eight releases are outside the retention window and may be expired — which a person approves. Automated deletion of public data on a schedule is how a dataset cited in a court filing stops resolving.

What a Consumer Needs That Portals Do Not Ask For Jump to heading#

The portal form collects what the portal needs to render a page. It does not collect what a person needs to decide whether they can use the data, and the gap between those two sets is where most published spatial data becomes unusable without an email exchange.

A change note per release. Not a changelog of the pipeline — a sentence saying what a consumer will notice: 214 parcels split, boundary corrections in the north-west quarter, one new attribute. Anyone maintaining a derived product reads this first and nothing else.

The conformance report, linked and readable. Publishing the quality figures alongside the data is what lets a consumer decide whether the layer meets their threshold, which is frequently different from yours. A dataset with a stated 99.2% topological consistency is more useful than one with no figure, even though the second one sounds better.

Known limitations, stated plainly. The tiles that were not re-flown, the municipality that did not deliver, the attribute that is only populated for post-2018 records. These are the facts a consumer discovers three weeks into a project, and publishing them costs one paragraph.

A stable schema description. Field names, types, units and code list references, published as a machine-readable artefact rather than described in prose. This is the same data dictionary the pipeline already maintains; publishing it costs a file copy and removes the most common support question.

A contact who answers. A monitored address, not the department’s general enquiries form. The absence of one is read — correctly — as a signal that the dataset is unmaintained.

All five are generated rather than typed, which is what makes them survive staff changes:

# publish/release_notes.py — Python 3.10+
def render_release_notes(manifest: dict, delta_summary: dict, quality: dict) -> str:
    """Generated from the pipeline's own records; never hand-written."""
    release = manifest["release"]
    return "\n".join([
        f"# {manifest['metadata']['title']} — release {release['version']}",
        "",
        "## What changed",
        f"- {delta_summary['inserted']:,} features added",
        f"- {delta_summary['updated']:,} features modified",
        f"- {delta_summary['retired']:,} features retired",
        f"- superseded release: {release.get('supersedes', '(first release)')}",
        "",
        "## Quality",
        f"- features evaluated: {quality['population']:,}",
        f"- conformance: {quality['conformant'] / quality['population']:.3%}",
        f"- full report: {manifest['metadata']['conformance_report_uri']}",
        "",
        "## Known limitations",
        *[f"- {item}" for item in manifest["metadata"].get("limitations", [])],
        "",
        f"Questions: {manifest['metadata']['contact_point']['email']}",
    ])

Generating the notes from the delta summary and the quality report has a second benefit beyond saving effort: the numbers in the notes cannot disagree with the numbers in the audit trail, because they are the same numbers. Hand-written release notes drift from reality within about three releases, and the drift is always in the flattering direction.

Compliance Reporting Output Jump to heading#

# publish/audit.py — pyarrow >=14 — Python 3.10+
import pyarrow as pa

PUBLICATION_AUDIT_SCHEMA = pa.schema([
    ("dataset_id",       pa.string()),
    ("release_version",  pa.string()),
    ("published_at",     pa.timestamp("us", tz="UTC")),
    ("published_by",     pa.string()),      # the service account that ran the job
    ("manifest_version", pa.string()),
    ("distribution",     pa.string()),      # one row per distribution
    ("media_type",       pa.string()),
    ("byte_size",        pa.int64()),
    ("sha256",           pa.string()),
    ("feature_count",    pa.int64()),       # per distribution: catches the mismatch above
    ("conformance_report_uri", pa.string()),
    ("supersedes",       pa.string()),
    ("retention_until",  pa.date32()),      # computed from the policy at publish time
])

retention_until is computed once, at publication, from the policy then in force — not evaluated at sweep time against the current policy. That distinction protects against a policy change silently shortening the retention of data already published under a longer commitment. The rows join the lineage manifest by dataset and release, closing the chain from source delivery through transformation and validation to the public artefact.

CI Integration Jump to heading#

# tests/test_publication.py — pytest >=7
def test_preflight_refuses_missing_licence(manifest):
    manifest["metadata"].pop("licence")
    with pytest.raises(ValueError, match="missing required metadata"):
        preflight(manifest)


def test_distributions_agree_on_feature_count(built_distributions):
    counts = {d["format"]: feature_count(d["path"]) for d in built_distributions}
    assert len(set(counts.values())) == 1, f"distributions disagree: {counts}"


def test_extent_is_derived_not_typed(manifest, dataset):
    derived = dataset.total_bounds
    assert manifest["metadata"]["spatial_extent"] == pytest.approx(list(derived), abs=1e-6)


def test_republishing_a_version_is_refused(manifest, published_release):
    # Negative control: immutability must be enforced by the code, not by convention.
    with pytest.raises(ValueError, match="already published"):
        preflight(manifest)

Run these in the release workflow alongside the conformance scorecard, so a release that would publish incomplete metadata fails before anyone is paged about it.

Deeper Implementation Walkthroughs Jump to heading#

Publishing to CKAN with automated DCAT harvest endpoints implements the portal-facing half against a real API, including how CKAN’s resource model maps onto DCAT distributions. Scheduling metadata refresh and retention sweeps covers the recurring half — keeping metadata current without republishing data, and proposing expiries that a human approves.

Frequently Asked Questions Jump to heading#

Should every release get a DOI? Where a dataset is cited in research or in statutory reporting, yes — a DOI is the strongest available promise that a specific version stays resolvable, and minting one per release rather than per dataset is what makes a citation reproducible. For internal or high-frequency operational data the overhead outweighs the benefit; a stable versioned URL plus a checksum is enough.

How many formats should be published? Enough to cover the audience, and no more than can be built in one run and verified. In practice a modern container format such as GeoPackage, a web-friendly format such as GeoJSON, and a tabular export for non-GIS users cover almost everyone. Each additional format is another artefact that can silently disagree with the others, which is why the feature-count test above exists.

What belongs on the landing page that is not in the metadata? The things a person needs and a harvester does not: what changed since the last release, known limitations, the contact who answers questions, and a link to the conformance report. Portals treat these as optional; users treat their absence as evidence the data is unmaintained.

What happens when a published release turns out to be wrong? Publish a corrected release and mark the faulty one as superseded with a stated reason; never quietly replace the bytes at an existing release URL. Someone has already downloaded it, cited it, or built a derived product from it, and silently changing what that URL serves makes their result irreproducible without telling them. Where the error is serious enough that the data should not be used at all — a systematic coordinate error, a licence published in error — keep the release page, remove the distributions, and say on the page what was wrong. A withdrawn release that explains itself is far more useful than a URL that has quietly started returning different numbers.

Can the catalogue be regenerated from the audit table? It should be. If the DCAT catalogue can be rebuilt deterministically from the publication audit rows, then a portal that loses its index — which happens during migrations — is recoverable without archaeology. Treat the audit table as the source of truth and the catalogue as a derived artefact, exactly as JSON-LD dataset publishing treats its serializations.

Related Jump to heading#

• Geospatial Compliance Reporting & Audit Trails — the parent section and the evidence chain publication closes
• Publishing to CKAN with Automated DCAT Harvest Endpoints — the portal API implementation
• Scheduling Metadata Refresh and Retention Sweeps — keeping a catalogue current and expiring nothing by accident
• Dataset Metadata & JSON-LD Publishing — what the published metadata says and how it is validated
• Spatial Data Quality Validation & Geometry Integrity — the conformance report every release must cite