Geospatial Schema Standardization & Mapping Automation

Gating Pull Requests on Geospatial Schema Drift

A geospatial dataset’s schema is a contract: downstream joins, tile builds, and portal harvesters all assume the field named parcel_id stays a string and the layer stays in EPSG:2263. A pull request that renames a field, retypes a column, or reprojects a layer breaks that contract silently — the CI passes, the merge lands, and the break surfaces days later in a consumer. This guide builds a check that fails a pull request the moment its schema drifts from a golden snapshot, prints a diff a reviewer can read, and allows the drift through only with an explicit, recorded approval. The drift count it produces is the exact input to schema_drift_count on the CI Validation Scorecards — this page owns the detection; the scorecard owns the aggregate threshold.

Schema drift is not the same as a bad transform. Renaming a field or coercing a type can be entirely legitimate — it is drift without approval that must be blocked. The renaming and coercion rules that make a change intentional are specified in Field Renaming & Type Coercion Rules; when a change legitimately alters the contract, the new shape must be re-expressed for every downstream platform through Cross-Platform Schema Translation. This check is the gate that forces that work to be deliberate.

Prerequisites checklist

Step 1: Capture the golden schema snapshot

The golden snapshot is the authoritative shape the branch is measured against. Serialize field names, types, nullability, and the layer CRS into a stable, sorted JSON structure so that a diff is a pure content comparison and never a key-ordering artifact. Generate it once from the current production dataset and commit it; regenerating it is itself a schema change that must go through the override flow in Step 5.

python
# schema_snapshot.py  — geopandas >=0.14, pyproj >=3.6
from __future__ import annotations
import json
from pathlib import Path

import geopandas as gpd


def extract_schema(source: str | Path) -> dict:
    """Serialize a dataset's field/type/CRS contract into a stable dict."""
    gdf = gpd.read_file(source, rows=1)  # header only; do not load geometry rows
    fields = {
        name: {
            "dtype": str(dtype),
            "nullable": bool(gdf[name].isna().any()) if len(gdf) else True,
        }
        for name, dtype in gdf.dtypes.items()
        if name != gdf.geometry.name
    }
    crs = gdf.crs
    return {
        "fields": dict(sorted(fields.items())),
        "geometry_field": gdf.geometry.name,
        "geometry_type": str(gdf.geom_type.iloc[0]) if len(gdf) else None,
        "crs_epsg": crs.to_epsg() if crs else None,
    }


def write_golden(source: str | Path, out: Path = Path("schema/golden.json")) -> None:
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(extract_schema(source), indent=2, sort_keys=True))

Step 2: Extract the PR-branch schema

Run the identical extractor against the dataset the pull-request branch produces. Using the same function for both sides guarantees the two structures are directly comparable — any difference is real drift, not a difference in how the two schemas were read.

python
# in the CI job, on the PR branch
from schema_snapshot import extract_schema

pr_schema = extract_schema("build/output.gpkg")
golden = json.loads(Path("schema/golden.json").read_text())

Step 3: Diff the two schemas

Classify every difference into one of five drift kinds: a field added, a field removed, a field retyped, a nullability change, or a CRS change. Each kind carries a different blast radius — a removed field breaks consumers immediately, an added field is usually additive and safe, a CRS change silently offsets every coordinate — so the diff records the kind, not just the fact of a change.

python
# schema_drift.py  — Python 3.10+
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class DriftEntry:
    kind: str        # FIELD_ADDED | FIELD_REMOVED | FIELD_RETYPED | NULLABILITY | CRS_CHANGED
    field: str       # field name, or "<crs>" for a CRS change
    detail: str      # human-readable before -> after


def diff_schema(golden: dict, current: dict) -> list[DriftEntry]:
    drift: list[DriftEntry] = []
    g_fields, c_fields = golden["fields"], current["fields"]

    for name in c_fields.keys() - g_fields.keys():
        drift.append(DriftEntry("FIELD_ADDED", name, f"+ {c_fields[name]['dtype']}"))
    for name in g_fields.keys() - c_fields.keys():
        drift.append(DriftEntry("FIELD_REMOVED", name, f"- {g_fields[name]['dtype']}"))
    for name in g_fields.keys() & c_fields.keys():
        g, c = g_fields[name], c_fields[name]
        if g["dtype"] != c["dtype"]:
            drift.append(DriftEntry("FIELD_RETYPED", name, f"{g['dtype']} -> {c['dtype']}"))
        if g["nullable"] != c["nullable"]:
            drift.append(DriftEntry("NULLABILITY", name, f"{g['nullable']} -> {c['nullable']}"))

    if golden["crs_epsg"] != current["crs_epsg"]:
        drift.append(DriftEntry(
            "CRS_CHANGED", "<crs>",
            f"EPSG:{golden['crs_epsg']} -> EPSG:{current['crs_epsg']}",
        ))
    return drift


def count_drift(current: str, golden: str) -> int:
    """Convenience entry point used by the scorecard engine."""
    g = json.loads(Path(golden).read_text())
    c = json.loads(Path(current).read_text())
    return len(diff_schema(g, c))

Step 4: Gate and render the diff

Fail the check on any unapproved drift, and print a diff the reviewer can act on without downloading an artifact. The renderer groups entries by kind and writes GitHub-flavored Markdown to the job summary; the gate function returns the exit code.

python
# gate_drift.py  — Python 3.10+
import sys
from pathlib import Path

from schema_drift import DriftEntry


def render_diff(drift: list[DriftEntry], approved: set[str]) -> str:
    if not drift:
        return "## Schema drift check — **PASS**\n\nNo drift from the golden schema.\n"
    lines = ["## Schema drift check", "", "| Kind | Field | Change | Status |", "| --- | --- | --- | --- |"]
    for d in sorted(drift, key=lambda e: (e.kind, e.field)):
        status = "approved" if f"{d.kind}:{d.field}" in approved else "**BLOCKED**"
        lines.append(f"| `{d.kind}` | `{d.field}` | {d.detail} | {status} |")
    return "\n".join(lines) + "\n"


def gate(drift: list[DriftEntry], approved: set[str], summary: Path) -> int:
    unapproved = [d for d in drift if f"{d.kind}:{d.field}" not in approved]
    summary.write_text(render_diff(drift, approved))
    if unapproved:
        for d in unapproved:
            print(f"::error::unapproved schema drift {d.kind} on {d.field}: {d.detail}")
        return 1
    return 0
yaml
# .github/workflows/schema-drift.yml
name: Schema Drift Gate
on: [pull_request]

permissions:
  contents: read

jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install "geopandas>=0.14" "pyproj>=3.6" "pyarrow>=14" "pyyaml>=6.0"
      - name: Run schema drift gate
        run: python -m gate_drift --pr build/output.gpkg --golden schema/golden.json

Step 5: Allow an explicit override

Legitimate schema changes must be able to land — but only deliberately. Permit drift only when two independent signals agree: an approved-schema-change label applied to the pull request by a maintainer, and a matching entry in the committed schema/approved_changes.yaml allowlist. Requiring both prevents an accidental label from waving through a change nobody enumerated, and prevents a stale allowlist entry from applying to a pull request no maintainer reviewed. Every override is recorded in the diff table and, on merge, folded into a regenerated golden snapshot so the next pull request measures against the new contract.

yaml
# schema/approved_changes.yaml
approved:
  - "FIELD_ADDED:zoning_overlay"      # 2026-07: additive column for the new overlay layer
  - "CRS_CHANGED:<crs>"               # 2026-07: reproject 2263 -> 6539 (survey-foot correction)
python
# load the allowlist and combine with the PR label (passed via env from Actions)
import os
import yaml
from pathlib import Path

allow = yaml.safe_load(Path("schema/approved_changes.yaml").read_text())["approved"]
label_present = "approved-schema-change" in os.environ.get("PR_LABELS", "").split(",")
# Both must hold for any entry to count as approved.
approved = set(allow) if label_present else set()

Wire the label into the job with PR_LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }} in the step’s env. When the label is absent, approved is empty and every drift entry blocks — the allowlist alone can never open the gate.

Verification

Exercise all three outcomes — clean, blocked, and approved — and confirm the exit code and diff match each:

bash
# 1. No drift: gate passes, summary says PASS.
python -m gate_drift --pr golden_copy.gpkg --golden schema/golden.json
echo "exit=$?"   # expect 0

# 2. Unapproved drift: gate fails with an ::error:: line per entry.
python -m gate_drift --pr renamed_field.gpkg --golden schema/golden.json
echo "exit=$?"   # expect 1

# 3. Approved drift: label + allowlist entry both present -> passes.
PR_LABELS="approved-schema-change" \
  python -m gate_drift --pr added_column.gpkg --golden schema/golden.json
echo "exit=$?"   # expect 0

The job summary must render the drift table with one row per change and a BLOCKED status on every unapproved entry. Confirm count_drift() returns the same total the scorecard reports — the two must agree, because the scorecard’s schema_drift_count is literally this function’s return value.

Troubleshooting

Symptom Likely cause Fix
Gate fails on a field-order change with no real drift Golden snapshot not sorted, so key order differs Regenerate golden.json with sort_keys=True; the extractor already sorts fields — never hand-edit the snapshot.
CRS drift not detected to_epsg() returned None for a custom WKT CRS on one side Compare on WKT when to_epsg() is None; a custom CRS with no EPSG code still changes the contract.
Every PR blocks after a legitimate merge Golden snapshot not regenerated after the approved change landed Regenerate and commit golden.json as part of the approving merge so the next PR measures the new baseline.
Override ignored despite the label Allowlist entry key does not match KIND:field exactly Match the key format the diff emits (FIELD_ADDED:zoning_overlay); a typo silently leaves the entry unapproved.
Retype from int64 to float64 slips through Dtype strings compared loosely, or nullability masked the change Compare exact dtype strings; a widening coercion is still a contract change consumers must be told about.