Geospatial Schema Standardization & Mapping Automation

Scheduling Metadata Refresh and Retention Sweeps

Two recurring jobs keep a published catalogue honest between releases, and both are easy to get wrong in ways that only show up later. The first is metadata refresh: a contact leaves, a licence is clarified, a conformance report moves — none of which changes the data, all of which must reach the catalogue. The second is the retention sweep, which decides what may stop being downloadable, and which is the only automated operation in the publication stage that destroys anything. This procedure implements both as scheduled work inside open data portal publication workflows, part of Geospatial Compliance Reporting & Audit Trails.

The steps map to configure (Step 1, separating the two kinds of change), execute (Step 2, the refresh), validate (Steps 3–4, candidates and approval), and log (Step 5, expiry and its record). The rule underneath all five: a scheduled job may publish metadata on its own authority and may never remove data on its own authority.

Prerequisites checklist

Step 1: Separate metadata changes from data releases

yaml
# publication.yaml — the two version fields are independent on purpose
release:
  version: "2026.2"            # data release: bumped only when the bytes change
  metadata_revision: 3         # metadata-only: bumped by the refresh job

A metadata correction that mints a new data release is worse than the error it fixes: it tells every consumer that the data changed, invalidates their caches, and buries the real releases in noise. Keeping two counters means the refresh job can say exactly what it did — release 2026.2, metadata revision 3 — and a consumer polling for data changes can ignore the second entirely.

The corollary is a rule about what the refresh job may touch: descriptive fields, contact details, licence identifiers, links to reports, and keywords. Never the distribution list, never a checksum, never an extent. Those describe bytes, and if the bytes changed it is a release.

Step 2: Detect what changed before writing anything

python
# publish/refresh.py — requests >=2.31 — Python 3.10+
import hashlib
import json
import logging

logger = logging.getLogger("publish.refresh")

REFRESHABLE = {"title", "notes", "license_id", "maintainer", "maintainer_email",
               "tags", "extras"}


def fingerprint(payload: dict) -> str:
    subset = {k: payload[k] for k in sorted(REFRESHABLE) if k in payload}
    return hashlib.sha256(json.dumps(subset, sort_keys=True).encode("utf-8")).hexdigest()


def refresh(ckan, manifest: dict) -> str:
    desired = build_metadata_payload(manifest)          # from the manifest, deterministically
    published = ckan.call("package_show", {"id": manifest["dataset_id"]})

    if fingerprint(desired) == fingerprint(published):
        logger.info("metadata unchanged for %s — no update issued", manifest["dataset_id"])
        return "unchanged"

    guard_immutable_fields(desired, published)          # raises if a distribution differs
    desired["id"] = published["id"]
    ckan.call("package_update", desired)
    logger.info("metadata revision %s published for %s",
                manifest["release"]["metadata_revision"], manifest["dataset_id"])
    return "updated"


def guard_immutable_fields(desired: dict, published: dict) -> None:
    """A metadata refresh must never alter what the release consists of."""
    desired_res = {(r["name"], r["hash"]) for r in desired.get("resources", [])}
    published_res = {(r["name"], r.get("hash", "")) for r in published.get("resources", [])}
    if desired_res and desired_res != published_res:
        raise ValueError("refresh would change distributions — that is a release, not a refresh")

The fingerprint comparison is what keeps a weekly job from producing fifty-two “modified” timestamps a year on a dataset nobody touched. Harvesters use dcterms:modified to decide what to re-fetch, so a job that updates unconditionally makes every downstream catalogue re-download the whole dataset every week.

guard_immutable_fields is the structural version of the Step 1 rule. Conventions are forgotten; an exception is not.

Step 3: Compute expiry candidates against both limits

python
# publish/retention.py — pyarrow >=14 — Python 3.10+
from datetime import date


def candidates(audit_rows: list[dict], policy: dict, today: date) -> list[dict]:
    """Releases outside the version window AND past their statutory floor."""
    by_dataset: dict[str, list[dict]] = {}
    for row in audit_rows:
        by_dataset.setdefault(row["dataset_id"], []).append(row)

    proposals = []
    for dataset_id, rows in by_dataset.items():
        rows.sort(key=lambda r: r["published_at"], reverse=True)
        keep = policy["retention"]["keep_versions"]

        for row in rows[keep:]:                     # older than the version window
            if row["retention_until"] > today:      # statutory floor wins, always
                continue
            proposals.append({
                "dataset_id": dataset_id,
                "release_version": row["release_version"],
                "published_at": row["published_at"],
                "retention_until": row["retention_until"],
                "downloads_last_year": downloads_for(dataset_id, row["release_version"]),
                "superseded_by": row.get("superseded_by", ""),
            })
    return proposals

The two limits are not interchangeable and the order matters. keep_versions is a housekeeping preference about how much history stays downloadable; retention_until is a records-management obligation computed at publish time from the policy that was in force then. A release inside its statutory floor is never a candidate, no matter how old or how unused — and because the floor was stamped at publication, a later policy change cannot retroactively shorten it.

downloads_last_year is not a criterion, it is context for the approver. A five-year-old release with two hundred downloads last month is being used by something, and the proposal should make that visible rather than let a version count decide.

Two Limits and an Approval Between a Release and Deletion A release passes through three tests before any bytes are removed. The first asks whether it falls outside the keep-versions window; releases inside the window are kept. The second asks whether today is past the retention_until date stamped at publication; releases still inside the statutory floor are kept regardless of age or use. Only releases failing both tests become proposals, and a proposal carries the release's download history to a named approver. Deletion happens only after approval, and even then the release page and its checksums remain published so that citations still resolve. a past release with its audit row outside keep_versions? housekeeping preference past retention_until? statutory floor, stamped at publish proposal → approver with download history no at either test → kept, and the sweep says nothing approval removes bytes only release page and checksums stay so citations still resolve

Step 4: Propose, never delete

yaml
# .github/workflows/retention-sweep.yml
name: retention-sweep
on:
  schedule:
    - cron: "0 6 1 * *"        # monthly, first of the month
  workflow_dispatch:

jobs:
  propose:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e .
      - run: python -m publish.retention --propose --out retention/PROPOSAL.md
      - uses: peter-evans/create-issue-from-file@v5
        with:
          title: "Retention sweep — releases eligible for expiry"
          content-filepath: retention/PROPOSAL.md
          assignees: data-custodian

The proposal is a document a person reads, so it should read like one:

text
## Retention sweep — 2027-03-01

cadastral-parcels 2024.1  published 2024-03-11  retention_until 2030-03-11  KEPT (statutory floor)
cadastral-parcels 2023.4  published 2023-11-02  retention_until 2026-11-02  ELIGIBLE
    downloads last 12 months: 0      superseded by 2024.1
address-points    2023.2  published 2023-06-14  retention_until 2026-06-14  ELIGIBLE
    downloads last 12 months: 412    superseded by 2024.2
    ⚠ still in active use — consider keeping

2 eligible · 1 kept · nothing has been deleted by this job

The last line is there for the reader’s benefit and the author’s. A sweep that has never deleted anything and says so is a sweep people trust enough to keep running.

Step 5: Expire the bytes, keep the record

python
# publish/retention.py — Python 3.10+
def expire(ckan, proposal: dict, approval: dict) -> dict:
    if not approval.get("approved_by"):
        raise PermissionError("expiry requires a named approver")

    # Remove the downloadable resources for this release …
    for resource_id in resources_for(proposal["dataset_id"], proposal["release_version"]):
        ckan.call("resource_delete", {"id": resource_id})

    # … but keep the release record itself resolvable.
    publish_tombstone(
        dataset_id=proposal["dataset_id"],
        version=proposal["release_version"],
        note=(f"Expired {approval['approved_at']} under the retention policy "
              f"(published {proposal['published_at']}, retained until "
              f"{proposal['retention_until']}). Checksums retained below."),
        checksums=checksums_for(proposal["dataset_id"], proposal["release_version"]),
    )
    return write_audit_row(proposal, approval)

The tombstone is the part that distinguishes retention from deletion. A citation to release 2023.4 must not return a 404 in 2031; it must return a page that says the release existed, when it was published, when and under what policy it was expired, and what its checksums were — so that a copy someone else holds can still be verified as authentic. Publishing the checksums after the bytes are gone costs nothing and answers the only question anyone will ask.

Verification

python
# tests/test_retention.py — pytest >=7
def test_release_inside_statutory_floor_is_never_a_candidate(audit_rows, policy):
    rows = [r for r in audit_rows if r["retention_until"] > date(2027, 3, 1)]
    proposals = candidates(rows, policy, today=date(2027, 3, 1))
    assert proposals == []


def test_sweep_never_deletes_without_approval(proposal):
    with pytest.raises(PermissionError):
        expire(ckan_stub, proposal, approval={})


def test_refresh_skips_when_metadata_is_unchanged(ckan_stub, manifest):
    assert refresh(ckan_stub, manifest) == "unchanged"
    assert ckan_stub.calls("package_update") == 0


def test_refresh_refuses_to_change_distributions(ckan_stub, manifest_with_new_file):
    # Negative control: a distribution change must be a release, not a refresh.
    with pytest.raises(ValueError, match="that is a release"):
        refresh(ckan_stub, manifest_with_new_file)

Troubleshooting

Symptom Likely cause Fix
Harvesters re-download everything weekly The refresh job updates unconditionally, moving dcterms:modified Compare the fingerprint first and skip when unchanged
A release disappeared and nobody approved it The sweep ran in delete mode, or expiry_requires_approval is unset Restore from backup, set the flag, and make approval a code path rather than a habit
Citations return 404 after a sweep Resources and the release page were removed together Keep the tombstone page; only the downloadable bytes may go
Proposals list the same releases every month Nobody is approving, or the approver is not reachable Reduce the frequency and name a specific person; an unread proposal is worse than none
A statutory floor looks wrong on old releases retention_until recomputed at sweep time from the current policy Stamp it at publish time and read it, never recompute
Metadata refresh clears fields set through the portal package_update replaces the whole package Expected — move those fields into the manifest; see publishing to CKAN