Spatial Data Quality Validation as an Engineering Discipline Jump to heading#

A schema mapping can be perfectly correct and still publish garbage. Field names can land in the right columns, every value can carry the right type, the coordinate reference system can be canonical to the digit — and the polygon can still cross itself, the parcel fabric can still hold a 4 cm sliver between two neighbours, the land-use code can still be a value no code list has ever defined, and the same address point can still exist three times because three counties exported it. Schema conformance answers is this shaped correctly. Data quality answers is this true, and can I prove it. This section owns the second question.

The discipline matters most where the consequences are legal rather than aesthetic. A municipality publishing a parcel layer with overlapping boundaries is publishing two conflicting statements about who owns a strip of land. An agency reporting under INSPIRE with invalid geometries fails conformance testing on the geometry itself, not on the metadata. A utility whose asset layer contains duplicate features bills twice for one connection. In each case the defect is machine-detectable, and in each case the reason it reached production is the same: nothing in the pipeline was contractually obliged to look.

This section is written for the engineer who owns that obligation — the GIS data manager who signs off a publication, the government technology team accountable to an audit, and the Python ETL engineer who has to turn “the data should be clean” into an assertion a build server can fail on. Everything here is deterministic: the same input produces the same verdict, the same repair, and the same quality report, on every run, on every machine. Where a rule is judgement rather than fact — how many centimetres of overlap is a sliver, which of two duplicate records survives — the judgement is externalized into a version-controlled threshold rather than buried in code.

The Five-Gate Validation Pipeline Jump to heading#

Quality validation is not a single pass over the data. It is an ordered sequence of gates in which each gate assumes the previous one has already succeeded, because most quality checks are meaningless on input the earlier gate would have rejected. Testing whether two parcels overlap is meaningless if one of them is a self-intersecting bowtie whose interior is undefined. Testing whether a code list value is legal is meaningless if the record is a duplicate that should never have entered the dataset. Order is not a stylistic preference; it is what makes the results interpretable.

Two properties of this arrangement do the real work. The first is that every gate has exactly one failure exit, and that exit is a quarantine store rather than a deletion or a log line. A feature that fails is still addressable afterwards by its stable identifier, which is what allows a data steward to answer “what happened to parcel 14-023-9?” three months later. The second is that the gates share one vocabulary of measures. Whether a feature failed on a self-intersection or an undefined land-use code, it produces the same shape of record — a quality element, a measure, the threshold that was applied, and the observed value — which is why the rollup at the bottom can be a single ISO 19157 quality report instead of five incompatible logs.

Each gate has a page of its own. Gate 2 is geometry validity and repair rules. Gate 3 is topology rule enforcement. Gate 4 is attribute domain and code list validation. Gate 5 is duplicate detection and feature deduplication. Gate 1 is deliberately thin — it belongs to the reader stage and is covered where the readers live, in batch schema processing pipelines.

Standards Alignment Matrix Jump to heading#

Spatial quality is one of the few areas of geospatial engineering where the standards are unusually specific about what to measure while remaining silent about what value is acceptable. ISO 19157 defines the quality elements and the report structure; it does not tell you that a 0.05 m² sliver is tolerable. That number is yours, and the standards expect you to publish it.

The last row is the one teams most often miss. Positional accuracy is a quality statement, but it is produced by the CRS normalization stage, which knows whether a transformation used a grid-based path or a coarse fallback. If those two subsystems do not share a record format, the published dataset ends up claiming an accuracy that no stage actually measured.

Core Validation Pattern Jump to heading#

The canonical operation of this section is: test, attempt one deterministic repair, re-test, and account for what the repair changed. The accounting step is what distinguishes an engineering pipeline from a cleanup script. A repair that silently changes a parcel’s area by 3% has not fixed the data; it has replaced a detectable error with an undetectable one.

# quality/validity.py — shapely >=2.0, geopandas >=0.14 — Python 3.10+
from dataclasses import dataclass
from typing import Literal

from shapely import make_valid
from shapely.geometry.base import BaseGeometry
from shapely.validation import explain_validity

Verdict = Literal["clean", "repaired", "quarantined"]

# Repairs are accepted only when they move area by less than this fraction.
# Published in the quality report; not a magic number in code.
AREA_DELTA_BUDGET = 0.001  # 0.1 %


@dataclass(frozen=True)
class Result:
    verdict: Verdict
    geometry: BaseGeometry | None
    reason: str
    area_delta: float


def validate_and_repair(geom: BaseGeometry) -> Result:
    """Deterministic single-pass validity gate. No randomness, no retries."""
    if geom is None or geom.is_empty:
        return Result("quarantined", None, "empty-geometry", 0.0)

    if geom.is_valid:
        return Result("clean", geom, "", 0.0)

    reason = explain_validity(geom)          # e.g. "Self-intersection[512300 4210]"
    repaired = make_valid(geom)              # GEOS 3.10+ structured repair

    if repaired.is_empty or not repaired.is_valid:
        return Result("quarantined", None, f"unrepairable: {reason}", 0.0)

    # Type drift is a rejection, not a repair: a polygon that becomes a
    # collection has changed what the feature *means*.
    if repaired.geom_type != geom.geom_type:
        return Result("quarantined", None,
                      f"type-drift {geom.geom_type}->{repaired.geom_type}: {reason}", 0.0)

    before = abs(geom.buffer(0).area) or 1.0
    delta = abs(repaired.area - before) / before
    if delta > AREA_DELTA_BUDGET:
        return Result("quarantined", None,
                      f"area-delta {delta:.4%} over budget: {reason}", delta)

    return Result("repaired", repaired, reason, delta)

Three decisions in that function are worth stating explicitly, because they are the ones that get argued about in review. Type drift is a rejection. make_valid on a bowtie polygon returns a MultiPolygon or a GeometryCollection; accepting that silently turns one parcel into two and breaks every downstream join on the feature identifier. The area budget is a fraction, not an absolute. A 0.5 m² change is negligible on a county and catastrophic on a utility pole footprint. The original invalidity reason travels with the repaired feature, because a dataset in which 4% of polygons needed repair is telling you something about the upstream source that a clean output would hide.

Validation Gates & Thresholds Jump to heading#

Every threshold below is a published number with a defined unit and a defined failure action. They are defaults to argue with, not universals — but a pipeline that cannot state its numbers cannot claim conformance.

1. Ring closure — first and last position of every ring identical to full double precision. Failure: quarantine, code GEOM_RING_OPEN. No auto-close: an unclosed ring usually means truncated input, and closing it invents a boundary.
2. Coordinate range — every ordinate finite and inside the CRS area of use, expanded by 1%. Failure: quarantine, code GEOM_OUT_OF_RANGE. This catches the classic swapped latitude/longitude at import.
3. Simple-feature validity — is_valid true after at most one make_valid pass. Failure: quarantine, code GEOM_INVALID.
4. Repair area budget — ≤ 0.1% area change, and no change of geometry type. Failure: quarantine, code GEOM_REPAIR_DRIFT.
5. Vertex density — no two consecutive vertices closer than 1 mm in a projected CRS. Failure: deterministic vertex collapse, logged as a repair, code GEOM_DUP_VERTEX.
6. Sliver area — in a coverage, no gap or overlap polygon larger than 0.05 m² and no thinner than a 0.02 shape index. Failure: routed to the topology reconciliation queue, code TOPO_SLIVER.
7. Coverage closure — the union of a partitioned coverage differs from its convex boundary by no more than the sliver budget. Failure: dataset-level rejection, code TOPO_COVERAGE_OPEN.
8. Domain conformance — 100% of values in a coded field resolve in the pinned code list version. Failure: quarantine per feature, code ATTR_UNDEFINED_CODE. No “other” bucket; an unknown code is a fact worth surfacing.
9. Mandatory completeness — 100% non-null on fields the target schema marks mandatory. Failure: quarantine, code ATTR_MANDATORY_NULL.
10. Duplicate rate — after deduplication, zero pairs remain within the match radius that also agree on the identity attributes. Failure: dataset-level rejection, code DUP_UNRESOLVED.

Notice that only two of the ten fail the whole dataset. Feature-level defects quarantine features; structural defects — an open coverage, unresolved duplicates — mean the dataset as a whole is not a coherent statement about the world, and publishing 99% of it would be worse than publishing none.

Geometry Validity & Repair Rules Jump to heading#

The geometry validity and repair rules stage owns the single-feature question: is this geometry a legal simple feature, and if not, can it be made one without changing what it asserts? It covers the taxonomy of invalidity that GEOS actually reports — self-intersections, nested shells, hole-outside-shell, ring self-intersection — and it establishes the repair ladder, from the cheap and safe buffer(0) for a hole-orientation problem through the structured make_valid to outright rejection. Its two guides handle the case that dominates real municipal data, repairing self-intersecting polygons with make_valid, and the case that silently breaks web clients, ring orientation and winding order in GeoJSON.

Topology Rule Enforcement Jump to heading#

Where validity is a property of one feature, topology is a property of a set. Topology rule enforcement covers the rules that a parcel fabric, an administrative-boundary layer or a zoning coverage must satisfy collectively: must-not-overlap, must-not-have-gaps, boundaries must be coincident, and must-be-covered-by. It is the most computationally demanding gate in the section, because a naive implementation is quadratic in feature count, and the page treats spatial indexing and tiling as part of the rule rather than an optimization bolted on afterwards. Its guides cover detecting gaps and overlaps in parcel coverages and the delicate business of snapping shared boundaries without moving survey corners.

Attribute Domain & Code List Validation Jump to heading#

The attribute half of quality is domain and code list validation: numeric ranges, enumerations, mandatory-field rules, and the harder problem of code lists that are governed externally and versioned on someone else’s schedule. This is where the section meets the standards work directly, because an INSPIRE code list is a registry resource with a URI and a version, not a Python set, and a pipeline that hard-codes its members will pass today and fail silently the next time the register is amended. The guides cover validating code list values against INSPIRE registers and enforcing attribute domains with pandera schemas.

Duplicate Detection & Feature Deduplication Jump to heading#

The last gate, duplicate detection and feature deduplication, is the one where determinism is hardest to hold, because deciding that two records describe the same real-world thing is a matching problem with tunable behaviour. The page’s position is that the matching may be fuzzy but the decision must not be: a match rule is a declared combination of spatial predicate, distance threshold and attribute agreement, and the survivor of a matched group is chosen by a stated precedence order — never by whichever row the database happened to return first. Its guides work through deduplicating address points with spatial clustering and matching features across vintages with stable identifiers.

Compliance & Audit Requirements Jump to heading#

A quality result that exists only in a build log has no audit value. The requirement is that every published dataset carries a machine-readable statement of what was checked, against which thresholds, with what outcome — and that the statement is reproducible from stored evidence rather than re-derived by rerunning the pipeline against data that has since changed.

# quality/report.py — pyarrow >=14 — Python 3.10+
import pyarrow as pa

# One row per (dataset, quality measure, run). This is the ISO 19157 DQ_*
# rollup in columnar form; the XML/JSON serialization is generated from it.
QUALITY_REPORT_SCHEMA = pa.schema([
    ("dataset_id",      pa.string()),                    # stable dataset identity
    ("run_id",          pa.string()),                    # pipeline execution
    ("evaluated_at",    pa.timestamp("us", tz="UTC")),
    ("dq_element",      pa.string()),                    # e.g. topologicalConsistency
    ("measure_code",    pa.string()),                    # e.g. TOPO_SLIVER
    ("threshold",       pa.string()),                    # the published number, as declared
    ("units",           pa.string()),                    # m2, ratio, count
    ("population",      pa.int64()),                     # features evaluated
    ("conformant",      pa.int64()),                     # features passing
    ("nonconformant",   pa.int64()),                     # features failing
    ("result",          pa.string()),                    # pass | fail | not_evaluated
    ("evidence_uri",    pa.string()),                    # quarantine partition holding the failures
])

Four obligations follow from that schema and are worth treating as non-negotiable:

• Retain the population, not just the failures. A report saying “12 features failed” is uninterpretable without knowing whether the dataset held 200 features or 2 million. population and nonconformant together are what make a conformance rate meaningful.
• Record not_evaluated explicitly. A check that was skipped because a dependency was unavailable is not a check that passed. The single most common audit finding in geospatial CI is a green board produced by a gate that never ran.
• Point at the evidence. evidence_uri addresses the quarantine partition, so an auditor can inspect the actual failing features rather than trusting a count. This is the same append-only discipline described in writing append-only lineage manifests to Parquet.
• Version the thresholds with the code that applies them. When a threshold changes, historical reports must keep the number that was actually in force, which is why threshold is stored per row rather than looked up at read time.

The report then feeds two consumers that already exist in this reference: the lineage manifest, which carries it as provenance, and the CI validation scorecard, which turns it into a merge gate.

Maintenance & Regression Strategy Jump to heading#

Quality rules rot in a particular way: they keep passing. A rule that no longer matches the data it was written for, or one whose threshold was widened during an incident and never narrowed again, reports success indefinitely. Three practices keep a rule set honest.

Golden datasets with deliberate defects. Maintain a small fixture corpus — a few dozen features is enough — in which every defect class the gates detect is present exactly once: one bowtie, one hole outside shell, one 0.03 m² sliver, one undefined code, one duplicate pair. The suite asserts not only that the pipeline rejects them but that it rejects them with the expected code. A rule that starts reporting GEOM_INVALID where it used to report GEOM_REPAIR_DRIFT has changed behaviour even though the feature is still caught.

A negative-control run every build. Feed the gates input they must refuse — a coverage with a deliberate 1 m² overlap, a code value invented for the test — and fail the build if the gates pass it. An assertion that has never rejected anything proves nothing, and this is the cheapest way to prove the machinery is still armed.

Threshold drift review on a fixed cadence. Every threshold in the list above is a row in a manifest with an owner and a review date. Quarterly, the review asks two questions of each: what would break if this number were halved, and when did this rule last reject anything? A rule that has rejected nothing in twelve months is either unnecessary or broken, and finding out which is a five-minute job that nobody ever schedules unless the calendar does it for them.

Regression coverage is also the natural place to catch interaction bugs between this section and its neighbours. A change to field renaming and type coercion that starts emitting empty strings instead of nulls will not fail a casting test, but it will quietly convert a mandatory-field violation into a passing value — which is exactly the class of failure that a shared golden dataset, run end to end, catches on the day it appears.

Frequently Asked Questions Jump to heading#

Should invalid geometries be repaired automatically or always rejected? Repaired, but only within a published budget and only for defect classes where the repair is deterministic. make_valid on a self-intersecting polygon has a defined outcome, and the area-delta budget bounds how much the feature can change; that combination is auditable. What must never be automatic is a repair that changes geometry type or that exceeds the budget, because both change what the feature asserts about the world. Those go to quarantine with the original invalidity reason attached.

Why run topology checks after validity rather than together? Because topological predicates on an invalid geometry are undefined. GEOS will happily evaluate intersects on a bowtie, but the answer describes a shape whose interior is ambiguous, so a gap or overlap measured against it is not evidence of anything. Running validity first also cuts the cost of the expensive gate: features that will be quarantined anyway never enter the pairwise comparison.

Is buffer(0) still an acceptable repair? For one narrow case, yes — a polygon whose only defect is hole orientation — and for everything else, no. buffer(0) silently discards parts of self-intersecting geometries, which is why it appears to “work” so often: it returns something valid by deleting the problem. Since GEOS 3.10, make_valid implements the structured repair that keeps the parts, and the difference between the two is exactly the data you would otherwise lose without a log line.

How do we validate code lists that are governed externally? Pin the version, cache the members, and validate against the cache — then check the register for a new version on a schedule rather than on every run. Validating against a live remote register makes the pipeline’s verdict depend on network state and on someone else’s release timing, which breaks reproducibility. The pinned-cache pattern is worked through in validating code list values against INSPIRE registers.

What conformance rate is good enough to publish? That is a policy decision, and the useful engineering answer is that the number must be published alongside the data rather than chosen at publication time. Most programmes settle on 100% for structural rules — validity, mandatory completeness, coverage closure — and a stated percentage for measures where source quality is outside the publisher’s control. The failure mode to avoid is a threshold that moves whenever a run breaches it.

Related Jump to heading#

• Geometry Validity & Repair Rules — Gate 2: the invalidity taxonomy, the repair ladder, and the area-delta budget
• Topology Rule Enforcement — Gate 3: gaps, overlaps, coincident boundaries, and indexed evaluation at scale
• Attribute Domain & Code List Validation — Gate 4: enumerations, ranges, and externally governed registers
• Duplicate Detection & Feature Deduplication — Gate 5: match rules, survivor precedence, and cross-vintage identity
• Geospatial Compliance Reporting & Audit Trails — where the quality report becomes published, retained evidence
• Automated Attribute Transformation & ETL Workflows — the casting and flattening stages whose output these gates inspect