Geospatial Schema Standardization & Mapping Automation

CI Validation Scorecards

Every stage of a geospatial pipeline already decides pass or fail: the CRS normalizer rejects unresolvable coordinate systems, the attribute ETL quarantines features that fail type coercion, the schema-compliance gate flags fields that drift from the target contract. Those decisions are scattered across log files, rejection queues, and exit codes. A CI validation scorecard collapses them into a single, reproducible conformance record per pipeline run — one artifact that says, in numbers a reviewer and an auditor can both read, whether this batch is fit to merge. This discipline operates at the core of Geospatial Compliance Reporting & Audit Trails, the practice area that turns transform and validation events into regulator-ready evidence.

This page covers what a scorecard is responsible for and where it stops. It does not re-run validation — it reduces the outputs that upstream gates already produced. The durable, append-only records it consumes are produced by Lineage Manifest Generation; a scorecard is a projection over those manifests, never a replacement for them. It also does not decide why a feature failed — the failure taxonomy belongs to each originating gate. Scope here is narrow: define the metrics, evaluate them against thresholds, and publish the badge, summary, and machine-readable artifact that a CI system uses to gate a merge. The two focused builds under this topic — publishing a schema conformance scorecard in GitHub Actions and gating pull requests on geospatial schema drift — implement the two hardest slices end to end.

Gate Results to Scorecard to Merge Gate Three upstream validation gates — CRS normalization, attribute ETL, and schema compliance — each write append-only lineage records. A scorecard engine reads all three sets of records for one pipeline run and reduces them into four metrics: conformance rate, degradation rate, quarantine breakdown, and schema-drift count. The engine emits three artifacts: a JSON scorecard, a Markdown summary, and an SVG badge. A threshold evaluator compares the metrics against the manifest and produces a pass or fail decision, which becomes the pull-request merge gate. Every path also appends to the run history that feeds the trend dashboard. CRS normalization gate lineage records Attribute ETL gate coercion + rejection log Schema compliance gate drift + field checks Scorecard engine • conformance rate • degradation rate • quarantine breakdown • schema-drift count reduce lineage manifests JSON scorecard Markdown summary SVG badge Threshold gate PR merge decision pass / fail + comment run history → trend dashboard evaluate metrics append run row data + decision flow historical append

Declarative Threshold Manifest

A scorecard is only as trustworthy as the thresholds it is judged against, and those thresholds must be reviewable in version control rather than buried in a script. Declare them in a YAML manifest that the scorecard engine loads at startup. The manifest names each metric, its comparison direction, the failing boundary, and whether breaching it blocks a merge or merely warns.

yaml
# scorecard_thresholds.yaml  — pyyaml >=6.0, pydantic >=2.0
scorecard:
  run_id_source: "git_sha"                 # MANDATORY: how each run is keyed
  min_features_for_rates: 50               # MANDATORY: below this, report counts only
  metrics:
    conformance_rate:                      # share of features passing every gate exactly / in tolerance
      min: 0.995                           # MANDATORY: floor; below it fails
      blocking: true                       # MANDATORY: breach blocks the merge
    degradation_rate:                      # share passing only via a fallback / lossy path
      max: 0.02                            # MANDATORY: ceiling
      blocking: true
    schema_drift_count:                    # count of fields added / removed / retyped vs golden schema
      max: 0                               # MANDATORY: any drift fails unless explicitly approved
      blocking: true
    quarantine_rate:                       # share routed to any rejection queue
      max: 0.01
      blocking: false                      # warn only; visible on the badge, does not block
  # --- OPTIONAL FIELDS ---
  baseline_run: "main"                     # branch/ref whose scorecard is the comparison baseline
  trend_window: 20                         # runs retained for the trend dashboard
  badge_style: "flat"                      # flat | flat-square — rendering hint only
  fail_on_missing_gate: true               # a gate that produced no manifest is treated as a hard fail

Mandatory vs optional field reference:

Field Required Type Notes
run_id_source Yes enum git_sha, ci_run_id, or manifest_hash; keys the scorecard to a reproducible run
min_features_for_rates Yes int > 0 Below this count, rates are statistically meaningless; the engine reports raw counts and suppresses pass/fail on rate metrics
metrics.*.min / .max Yes float Exactly one of min/max per metric, matching its comparison direction
metrics.*.blocking Yes bool true fails the CI run on breach; false renders on the badge but never blocks
baseline_run No string Ref whose scorecard the current run is diffed against for degradation and drift deltas
trend_window No int Number of historical runs kept for the trend series; defaults to 20
badge_style No enum Cosmetic only; does not affect thresholds
fail_on_missing_gate No bool When true, a silently absent gate manifest fails the run rather than scoring a false 100%

The last field matters more than it looks: the most dangerous scorecard is one that reads 100% because a gate crashed before writing any records. Treat a missing manifest as a hard failure, not as zero rejections. Validate the whole manifest with Pydantic before any run begins, and reject at parse time any metric that declares both min and max, or a conformance_rate.min outside [0, 1].

Preprocessing: Collecting Gate Outputs

The scorecard engine is a pure reducer — it must be handed a complete, well-formed set of gate outputs before it runs. Three preconditions have to hold:

  1. Every expected gate wrote a manifest. Each upstream gate emits an append-only lineage file for the run — the CRS stage’s routing records, the attribute ETL’s coercion and rejection log, the schema-compliance field report. Enumerate the gates the pipeline is supposed to have run and confirm each produced a file. A gate that ran but produced no records still writes an empty-but-present manifest with a header, which is how the engine distinguishes “nothing to reject” from “gate never executed.” The append-only guarantee itself is established by writing append-only lineage manifests to Parquet.

  2. Records share a common run key. Every record from every gate must carry the same run_id (the git_sha, CI run id, or manifest hash named in the threshold manifest). Records whose run key does not match the run under evaluation are dropped before reduction — this prevents a stale file left in the working directory from contaminating the scorecard.

  3. Outcome vocabularies are reconciled. Different gates use different outcome codes (PASS, TOLERANCE_EXCEEDED, COERCION_LOSSY, FIELD_DROPPED). Before reduction, each gate’s codes are mapped onto the three scorecard classes — conformant, degraded, quarantined — via a small, explicit lookup that lives beside the engine. An unmapped code is a hard error, never a silent bucket.

python
# preprocess.py  — pyarrow >=14
from pathlib import Path

# Explicit outcome -> scorecard class. An unmapped code raises.
OUTCOME_CLASS = {
    "PASS": "conformant",
    "EXACT": "conformant",
    "TOLERANCE_EXCEEDED": "quarantined",
    "NO_PATH": "quarantined",
    "INVALID_CRS": "quarantined",
    "COERCION_EXACT": "conformant",
    "COERCION_LOSSY": "degraded",
    "FALLBACK_PATH": "degraded",
    "FIELD_DROPPED": "quarantined",
    "GEOMETRY_INVALID": "quarantined",
}


def classify(outcome: str) -> str:
    try:
        return OUTCOME_CLASS[outcome]
    except KeyError as exc:
        raise ValueError(
            f"Unmapped gate outcome {outcome!r}. Add it to OUTCOME_CLASS "
            "before it can be scored — silent bucketing is forbidden."
        ) from exc


def expected_manifests(run_dir: Path, gates: list[str]) -> list[Path]:
    """Confirm every expected gate produced a manifest; raise if any is absent."""
    missing = [g for g in gates if not (run_dir / f"{g}.parquet").exists()]
    if missing:
        raise FileNotFoundError(
            f"Missing gate manifests: {missing}. A gate that produced no output "
            "must still write an empty header file; absence is a hard failure."
        )
    return [run_dir / f"{g}.parquet" for g in gates]

Execution Engine: Computing the Scorecard

With the manifests collected and outcomes classified, the engine reduces every record into the four headline metrics. It reads the append-only lineage manifests with pyarrow, projects only the columns it needs to keep memory flat across large batches, and computes rates only when the feature count clears min_features_for_rates.

python
# scorecard/engine.py  — pyarrow >=14, pydantic >=2.0
from __future__ import annotations
import json
import logging
from dataclasses import dataclass, asdict
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

from preprocess import classify, expected_manifests

logger = logging.getLogger(__name__)


@dataclass
class Scorecard:
    run_id: str
    total_features: int
    conformance_rate: float | None      # None when below min_features_for_rates
    degradation_rate: float | None
    quarantine_rate: float | None
    quarantine_breakdown: dict[str, int]
    schema_drift_count: int
    rates_suppressed: bool


def compute_scorecard(
    run_id: str,
    run_dir: Path,
    gates: list[str],
    min_features_for_rates: int,
    schema_drift_count: int,
) -> Scorecard:
    """Reduce per-gate lineage manifests into a single run scorecard."""
    manifests = expected_manifests(run_dir, gates)

    counts = {"conformant": 0, "degraded": 0, "quarantined": 0}
    quarantine_breakdown: dict[str, int] = {}

    for path in manifests:
        try:
            table = pq.read_table(path, columns=["feature_id", "outcome"])
        except (pa.ArrowInvalid, OSError) as exc:
            # A corrupt or unreadable manifest must never score as a pass.
            logger.critical("Unreadable gate manifest %s — %s", path, exc)
            raise

        for outcome in table.column("outcome").to_pylist():
            cls = classify(outcome)
            counts[cls] += 1
            if cls == "quarantined":
                quarantine_breakdown[outcome] = quarantine_breakdown.get(outcome, 0) + 1

    total = sum(counts.values())
    suppressed = total < min_features_for_rates
    if suppressed:
        logger.warning(
            "Run %s has %d features (< %d); reporting counts, suppressing rates.",
            run_id, total, min_features_for_rates,
        )

    def rate(n: int) -> float | None:
        return None if suppressed or total == 0 else round(n / total, 6)

    return Scorecard(
        run_id=run_id,
        total_features=total,
        conformance_rate=rate(counts["conformant"]),
        degradation_rate=rate(counts["degraded"]),
        quarantine_rate=rate(counts["quarantined"]),
        quarantine_breakdown=dict(sorted(quarantine_breakdown.items())),
        schema_drift_count=schema_drift_count,
        rates_suppressed=suppressed,
    )

The schema_drift_count is passed in rather than computed here because it is a comparison against a golden snapshot, not a reduction over this run’s records — that comparison is the subject of gating pull requests on geospatial schema drift. Keeping it as an input keeps the engine a pure reducer.

Once computed, the scorecard is serialized three ways from one source object so the three renderings can never disagree. The JSON form is the machine-readable record; the Markdown form goes to the CI job summary and the pull-request comment; the SVG badge is the at-a-glance signal embedded in a README or dashboard.

python
# scorecard/emit.py  — Python 3.10+
import json
from pathlib import Path

from scorecard.engine import Scorecard


def emit_json(sc: Scorecard, out: Path) -> None:
    out.write_text(json.dumps(asdict(sc), indent=2, sort_keys=True))


def emit_markdown(sc: Scorecard, passed: bool, out: Path) -> None:
    status = "PASS" if passed else "FAIL"
    lines = [
        f"## Conformance scorecard — `{sc.run_id}` — **{status}**",
        "",
        "| Metric | Value |",
        "| --- | --- |",
        f"| Total features | {sc.total_features} |",
        f"| Conformance rate | {_fmt(sc.conformance_rate)} |",
        f"| Degradation rate | {_fmt(sc.degradation_rate)} |",
        f"| Quarantine rate | {_fmt(sc.quarantine_rate)} |",
        f"| Schema-drift count | {sc.schema_drift_count} |",
    ]
    if sc.quarantine_breakdown:
        lines += ["", "### Quarantine breakdown", "", "| Reason | Count |", "| --- | --- |"]
        lines += [f"| `{k}` | {v} |" for k, v in sc.quarantine_breakdown.items()]
    out.write_text("\n".join(lines) + "\n")


def emit_badge(sc: Scorecard, passed: bool, out: Path) -> None:
    label, color = ("conformance", "#2e7d32") if passed else ("conformance", "#c62828")
    value = "n/a" if sc.conformance_rate is None else f"{sc.conformance_rate * 100:.2f}%"
    out.write_text(_render_badge_svg(label, value, color))


def _fmt(v: float | None) -> str:
    return "n/a (suppressed)" if v is None else f"{v * 100:.3f}%"

Failure Modes

A scorecard has one job that dwarfs all others: it must never report a passing grade for a run that did not actually pass. Most of its failure modes are variations on that theme — a signal lost, a file missing, a metric silently defaulting to a favorable value.

Failure type Likely cause Deterministic recovery action
Missing gate manifest Upstream gate crashed before writing any records With fail_on_missing_gate: true, fail the run; never score an absent gate as zero rejections
Corrupt / unreadable manifest Truncated Parquet from an interrupted write Raise on read; the scorecard is not produced, and the CI run fails hard
Unmapped outcome code A gate introduced a new outcome not in OUTCOME_CLASS Raise on classify(); require the code be mapped before the run can score
Rates computed on tiny batches Feature count below min_features_for_rates Suppress rates, report raw counts, and do not fail rate-based thresholds
Run-key mismatch Stale manifest from a prior run left in the working dir Drop non-matching records during preprocessing; log the count dropped
Badge disagrees with JSON Badge rendered from a separate computation Render all three artifacts from one Scorecard object so they cannot diverge
Drift count injected as zero Schema-drift comparison skipped or errored Treat a missing drift result as a hard failure, identical to a breached threshold

When the quarantine rate climbs across consecutive runs while conformance stays flat, that is the signature of an upstream input regression, not a scorecard problem — the fallback logic in Error Handling & Retry Logic is absorbing the damage and reporting it as degradation rather than outright failure. Watch the degradation rate as the leading indicator.

Compliance Reporting Output

The JSON scorecard is itself a compliance artifact: it is the merge decision record for the run, and it should be retained alongside the lineage manifests it summarizes. For datasets under INSPIRE or FGDC obligations, retain scorecards for the full audit window (INSPIRE’s six-year retention) so that any historical merge can be justified by the evidence that authorized it.

Each scorecard carries the fields an auditor needs to reconstruct the decision without re-running anything: the run_id that keys it to a specific commit and input batch, the total_features denominator, every rate with its suppression flag, the full quarantine_breakdown by reason code, and the schema_drift_count. Because the scorecard is derived purely from append-only lineage manifests, it inherits their provenance — a reviewer can always drill from a rate on the scorecard down to the individual feature records that produced it. That drill-down path is what distinguishes a genuine conformance record from a vanity metric.

Append every computed scorecard as one row to a run-history table (Parquet, keyed by run_id and timestamp). That table is the trend series: it lets you plot conformance and degradation over the last trend_window runs and detect slow erosion that no single run would trip. A run that passes every threshold but sits at the bottom of a two-week downward trend is a signal worth surfacing on the dashboard even though it does not block the merge.

CI Integration

The scorecard becomes a merge gate when a CI job computes it, compares each metric to the manifest, and exits non-zero on any blocking breach. The evaluation is deliberately trivial once the metrics exist — all the judgment lives in the manifest.

python
# scorecard/gate.py  — pydantic >=2.0
import sys
import logging

from scorecard.engine import Scorecard

logger = logging.getLogger(__name__)


def evaluate(sc: Scorecard, thresholds: dict) -> bool:
    """Return True if every blocking metric is within threshold, else False."""
    ok = True
    m = thresholds["metrics"]

    def check(name: str, value, bound, direction: str, blocking: bool) -> None:
        nonlocal ok
        if value is None:
            return  # suppressed rate: not evaluated
        breached = value < bound if direction == "min" else value > bound
        if breached:
            level = "BLOCKING" if blocking else "warn"
            logger.error("[%s] %s %s %.4f breaches %s %s",
                         sc.run_id, level, name, value,
                         direction, bound)
            if blocking:
                ok = False

    check("conformance_rate", sc.conformance_rate, m["conformance_rate"]["min"], "min", m["conformance_rate"]["blocking"])
    check("degradation_rate", sc.degradation_rate, m["degradation_rate"]["max"], "max", m["degradation_rate"]["blocking"])
    check("quarantine_rate", sc.quarantine_rate, m["quarantine_rate"]["max"], "max", m["quarantine_rate"]["blocking"])
    check("schema_drift_count", sc.schema_drift_count, m["schema_drift_count"]["max"], "max", m["schema_drift_count"]["blocking"])
    return ok


def main() -> None:
    # ... load thresholds, compute sc ...
    passed = evaluate(sc, thresholds)
    sys.exit(0 if passed else 1)

Wire this into pytest for local pre-merge checks and into a pre-commit hook for fast feedback, but the authoritative gate is the CI workflow. The end-to-end GitHub Actions build — running the validation suite, computing the scorecard, rendering the job summary, committing the badge, and posting the pull-request comment — is walked through in publishing a schema conformance scorecard in GitHub Actions. The narrower but sharper gate that fails a pull request the moment a field is added, removed, retyped, or its CRS changes — the input to schema_drift_count above — is built in gating pull requests on geospatial schema drift.

Frequently Asked Questions

What is the difference between conformance rate and degradation rate? Conformance rate is the share of features that passed every gate on an exact or in-tolerance path. Degradation rate is the share that passed only after a fallback or a lossy coercion — still conformant, but on a weaker path that an auditor should be able to see. They are reported separately because a run can hold conformance at 100% while degradation quietly climbs, and a single blended “pass rate” would hide exactly the erosion you most want to catch.

Should a scorecard be computed per run or per dataset? Per run. A scorecard describes one execution of the pipeline against a specific input batch and a specific manifest version, which makes it reproducible and diffable against the previous run on the same branch. Per-dataset or per-quarter rollups are derived afterward by aggregating run scorecards from the run-history table, never by re-scoring raw data.

Why compute the scorecard from lineage manifests rather than from live gate return codes? Return codes are ephemeral and lossy — a single exit status cannot express a quarantine breakdown or a degradation rate. The lineage manifests are the durable, append-only record every gate already writes for audit purposes, so reducing them keeps the scorecard perfectly consistent with the evidence a regulator would inspect, and it lets the scorecard be recomputed after the fact without re-running the pipeline.

Can a scorecard pass while an upstream gate is silently broken? Only if you let it. Set fail_on_missing_gate: true and enforce the empty-but-present manifest convention: a gate that ran with nothing to report still writes a header file, so a genuinely absent file is unambiguous evidence the gate did not run, and it fails the scorecard rather than scoring a false 100%.