Geospatial Schema Standardization & Mapping Automation

Geometry Validity & Repair Rules

An invalid geometry is not a formatting problem. It is a statement about space that has no consistent interpretation: a polygon whose boundary crosses itself does not have a well-defined interior, so every question you can ask of it — its area, whether a point lies inside it, whether it touches its neighbour — has an answer that depends on which library you asked. Repairing such a feature is therefore not cosmetic tidying but a decision about what the data was supposed to say, and it belongs in a stage that is explicit about the decision it made and how much the feature moved as a result. This stage is Gate 2 of Spatial Data Quality Validation & Geometry Integrity, and it runs on one feature at a time.

The scope boundary matters because two neighbouring stages look superficially similar. This page owns intra-feature correctness: is this single geometry a legal OGC simple feature, and can it be made one. It does not own relationships between features — a parcel that overlaps its neighbour can be perfectly valid on its own, and that case belongs to topology rule enforcement. It also does not own coordinate correctness: a polygon in the wrong projection is valid, and putting it right is the job of CRS normalization and sync. What arrives here has already been reprojected and cast; what leaves here is either a valid geometry with a recorded provenance of any repair, or a quarantined feature with a coded reason.

The Invalidity Taxonomy GEOS Actually Reports

Before writing a repair rule it is worth being precise about the defect classes that occur in practice, because the repair that is correct for one is destructive for another. GEOS — the engine behind Shapely, PostGIS and GDAL — reports invalidity through a small, stable vocabulary, and shapely.validation.explain_validity returns it as a string with the offending coordinate attached.

Four Polygon Invalidity Classes, Drawn Panel one shows a bowtie: a quadrilateral whose boundary crosses itself at a central node, so the interior is split into two lobes of ambiguous orientation. Panel two shows a hole ring drawn entirely outside its shell ring, which makes the subtraction undefined. Panel three shows two shell rings of one multipolygon where the smaller sits inside the larger, so the same area is claimed twice. Panel four shows a ring where two consecutive vertices sit at the same coordinate, producing a zero-length segment. Each panel is captioned with the string GEOS reports for it. Self-intersection boundary crosses at one node interior undefined on both lobes "Self-intersection[x y]" Hole outside shell hole ring sits beyond the shell subtraction has nothing to remove "Hole lies outside shell" Nested shells two shells of one multipolygon the same area is claimed twice "Nested shells" Repeated vertex two positions at one coordinate zero-length segment in the ring valid, but breaks many consumers

The fourth panel is the trap. A repeated vertex does not make a polygon invalid under the OGC rules, so a validity-only gate passes it — and then a downstream simplification, a buffer of zero width, or an export to a format that de-duplicates positions produces a shape that is invalid, in a stage that has no idea why. Defects that are legal but hostile deserve their own rule tier, which is why the manifest below separates reject, repair and normalize rather than treating validity as a boolean.

Declarative Configuration Manifest

The rule set is a version-controlled document, not a chain of if statements. Each entry names a defect class, the action to take, and the budget within which the action is allowed to change the feature.

yaml
# validity_rules.yaml — shapely >=2.0, pyyaml >=6.0
ruleset_version: "2.1.0"          # MANDATORY: bumped on any behavioural change
target_crs_units: "metre"         # MANDATORY: budgets below are expressed in these units
rules:
  - defect: self_intersection     # MANDATORY: GEOS defect class
    action: repair                # MANDATORY: repair | normalize | reject
    method: make_valid            # MANDATORY when action=repair
    area_delta_budget: 0.001      # OPTIONAL: fraction; default 0.001
    allow_type_change: false      # OPTIONAL: default false — see notes
  - defect: hole_outside_shell
    action: repair
    method: make_valid
    area_delta_budget: 0.005      # a mis-placed hole legitimately changes area
    allow_type_change: false
  - defect: nested_shells
    action: reject                # ambiguous ownership; a human must decide
    quarantine_code: GEOM_NESTED_SHELLS
  - defect: ring_self_intersection
    action: repair
    method: make_valid
    area_delta_budget: 0.001
  - defect: repeated_vertex
    action: normalize             # legal but hostile: collapse, then log
    method: remove_repeated_points
    tolerance: 0.001              # metres; 1 mm
  - defect: empty_geometry
    action: reject
    quarantine_code: GEOM_EMPTY
  - defect: non_finite_ordinate
    action: reject
    quarantine_code: GEOM_NON_FINITE
precision:
  grid_size: 0.0001               # OPTIONAL: snap ordinates to 0.1 mm before testing
  enforce_2d: true                # OPTIONAL: drop Z/M before validity testing
Field Required Meaning
ruleset_version Mandatory Recorded on every repaired feature so a result can be reproduced later
target_crs_units Mandatory Guards against applying metre budgets to a geographic CRS
defect Mandatory One of the GEOS defect classes; unknown values fail the manifest load
action Mandatory repair attempts a fix, normalize cleans a legal-but-hostile shape, reject quarantines
method Conditional Required for repair and normalize; the named Shapely operation
area_delta_budget Optional Fraction of original area the repair may change; default 0.001
allow_type_change Optional Whether a polygon may become a multipolygon; default false
quarantine_code Conditional Required for reject; the code written to the quality report
precision.grid_size Optional Coordinate snapping applied before testing, in CRS units
precision.enforce_2d Optional Strips Z and M so validity is judged in the plane, as OGC defines it

Two of these deserve argument rather than defaults. allow_type_change exists because there genuinely are datasets — dissolved land-cover, for instance — where a repaired bowtie should become a multipolygon, and forcing rejection there quarantines thousands of legitimate features. The rule is that it may be enabled per dataset, never globally, and that enabling it is a reviewed change because it breaks the one-feature-one-geometry assumption that joins depend on. precision.grid_size matters because the most common source of “invalid at 1e-12” geometries is arithmetic noise from a reprojection, and snapping to a declared grid turns an unbounded class of near-degenerate cases into an exactly reproducible one.

Preprocessing Requirements

Three things must be true before a geometry reaches the validity test, and each of them is a defect in its own right if it is not.

Dimensionality is settled. OGC validity is a planar concept. A polygon with Z ordinates is tested by projecting to the plane, and if the pipeline later needs the heights, they must be preserved elsewhere — as an attribute, or in a separate stage — because a repair operation is not obliged to keep them. Set enforce_2d and record that the third dimension was dropped.

The CRS is projected and the units are known. Every budget in the manifest is an area fraction or a linear tolerance. A 1 mm tolerance applied to a geometry in EPSG:4326 is a tolerance of roughly 100 km at the equator. The stage must refuse to run on a geographic CRS unless the manifest explicitly declares degree units, and it should fail loudly rather than convert silently — the conversion is a decision, and it belongs to the projection normalization workflow.

Coordinates are finite. A NaN ordinate propagates through GEOS as an exception or, worse, as a silently degenerate result. Screen for non-finite values first, because the check is trivial and the failure mode without it is confusing: make_valid on a geometry with a NaN can return an empty geometry, which then looks like an unrepairable feature rather than a corrupt input.

Execution Engine & Precision Guards

The engine below evaluates one feature against the manifest. It is written to be boring: no retries, no heuristics, no fallback to a different method if the first one fails. Every branch ends in a recorded outcome.

python
# quality/geometry_gate.py — shapely >=2.0, pyarrow >=14 — Python 3.10+
import logging
import math
from dataclasses import dataclass

from shapely import make_valid, remove_repeated_points, set_precision
from shapely.geometry.base import BaseGeometry
from shapely.validation import explain_validity

logger = logging.getLogger("quality.geometry")

# GEOS phrases mapped onto the manifest's defect vocabulary. Anything GEOS
# reports that is not in this map is an unknown defect and is rejected, never
# guessed at — a new GEOS message must be a code change, not a silent pass.
DEFECT_PHRASES: dict[str, str] = {
    "self-intersection": "self_intersection",
    "ring self-intersection": "ring_self_intersection",
    "hole lies outside shell": "hole_outside_shell",
    "holes are nested": "hole_outside_shell",
    "nested shells": "nested_shells",
    "interior is disconnected": "self_intersection",
    "too few points in geometry component": "empty_geometry",
}


@dataclass(frozen=True)
class GateResult:
    verdict: str            # clean | normalized | repaired | quarantined
    geometry: BaseGeometry | None
    defect: str
    code: str
    area_delta: float


def classify(geom: BaseGeometry) -> str:
    """Map the GEOS explanation onto a manifest defect class."""
    message = explain_validity(geom).lower()
    for phrase, defect in DEFECT_PHRASES.items():
        if phrase in message:
            return defect
    logger.error("unmapped GEOS validity message: %s", message)
    return "unknown"


def has_non_finite(geom: BaseGeometry) -> bool:
    return any(not math.isfinite(v) for v in geom.bounds)


def run_gate(geom: BaseGeometry, rules: dict, precision: dict) -> GateResult:
    if geom is None or geom.is_empty:
        return GateResult("quarantined", None, "empty_geometry", "GEOM_EMPTY", 0.0)
    if has_non_finite(geom):
        return GateResult("quarantined", None, "non_finite_ordinate", "GEOM_NON_FINITE", 0.0)

    grid = precision.get("grid_size")
    if grid:
        geom = set_precision(geom, grid)            # deterministic ordinate snapping

    cleaned = remove_repeated_points(geom, tolerance=rules["repeated_vertex"]["tolerance"])
    normalized = cleaned.equals_exact(geom, tolerance=0.0) is False

    if cleaned.is_valid:
        verdict = "normalized" if normalized else "clean"
        return GateResult(verdict, cleaned, "repeated_vertex" if normalized else "", "", 0.0)

    defect = classify(cleaned)
    rule = rules.get(defect)
    if rule is None or rule["action"] == "reject":
        code = (rule or {}).get("quarantine_code", "GEOM_INVALID")
        return GateResult("quarantined", None, defect, code, 0.0)

    repaired = make_valid(cleaned)
    if repaired.is_empty or not repaired.is_valid:
        return GateResult("quarantined", None, defect, "GEOM_UNREPAIRABLE", 0.0)

    if repaired.geom_type != cleaned.geom_type and not rule.get("allow_type_change", False):
        return GateResult("quarantined", None, defect, "GEOM_TYPE_DRIFT", 0.0)

    before = abs(cleaned.buffer(0).area) or 1.0
    delta = abs(repaired.area - before) / before
    if delta > rule.get("area_delta_budget", 0.001):
        return GateResult("quarantined", None, defect, "GEOM_REPAIR_DRIFT", delta)

    logger.info("repaired %s defect=%s area_delta=%.6f", cleaned.geom_type, defect, delta)
    return GateResult("repaired", repaired, defect, "", delta)

The precision guards are the parts most often left out, and each of them prevents a specific class of non-reproducible behaviour. set_precision makes the result independent of the floating-point history of the input, so a geometry that was reprojected on one machine and one that arrived pre-projected produce identical verdicts. Comparing with equals_exact(tolerance=0.0) rather than == distinguishes “the normalizer changed something” from “the normalizer returned an equal but differently-constructed object”. And the unknown classification is deliberately fatal: a GEOS upgrade that introduces a new message must break the build rather than quietly widen what the gate accepts.

The Repair Ladder and Its Two Exits Four ordered rungs run down the left. Rung one snaps ordinates to the precision grid. Rung two removes repeated points. Rung three applies the structured make_valid repair. Rung four measures the area delta and the geometry type against the budget. Each rung passes downward to the next; rungs three and four also branch right to quarantine when the repair fails, changes type, or exceeds the budget. The accepted exit at the bottom carries the defect class, the method used, and the measured area delta with the feature. 1 · snap to precision grid set_precision(grid_size) 2 · collapse repeated vertices legal but hostile — normalize and log 3 · structured repair make_valid — never buffer(0) 4 · measure what changed area delta and geometry type vs budget Quarantine GEOM_UNREPAIRABLE GEOM_TYPE_DRIFT GEOM_REPAIR_DRIFT original geometry retained Accepted verdict: clean · normalized · repaired defect class and method recorded area delta carried to the manifest ruleset_version stamped repair failed over budget already valid

Failure Modes & Fallback Routing

Failure Typical cause Deterministic action
GEOM_EMPTY Upstream filter produced an empty intersection; a reader returned a null geometry for a present row Quarantine; never substitute an empty polygon, which silently zeroes areas in aggregates
GEOM_NON_FINITE A reprojection outside the CRS area of use; a division by zero in an upstream calculation Quarantine and raise a run-level alert — a single NaN usually means many
GEOM_INVALID (unmapped defect) A GEOS version reporting a message the classifier does not know Quarantine and fail the build; a new message must be reviewed, not absorbed
GEOM_UNREPAIRABLE Degenerate input: a “polygon” of three collinear points, a ring of two positions Quarantine with the original geometry retained as evidence
GEOM_TYPE_DRIFT A bowtie repaired into a multipolygon where the dataset assumes one part per feature Quarantine unless allow_type_change is set for this dataset
GEOM_REPAIR_DRIFT A repair that moved area beyond the budget — usually a hole that was never a hole Quarantine; the size of the delta is recorded so the budget can be argued about with evidence
GEOM_NESTED_SHELLS Two shells of one multipolygon overlapping, often from a bad dissolve upstream Quarantine for human resolution; the correct answer depends on which shell is authoritative

The rule underneath the table is that no failure produces a modified feature that continues down the pipeline. Either the repair succeeded within budget and the feature continues with a repair record, or the feature stops. There is no middle state in which a partially-fixed geometry proceeds with a warning, because a warning is a log line nobody reads and a quarantine row is a number in the quality report.

Compliance Reporting Output

Every feature that passes through this gate contributes one row to the quality report, whether it was clean or not. Reporting only failures makes conformance rates impossible to compute and makes a stage that silently stopped running indistinguishable from one that found nothing wrong.

python
# quality/geometry_report.py — pyarrow >=14 — Python 3.10+
import pyarrow as pa

GEOMETRY_GATE_SCHEMA = pa.schema([
    ("feature_id",      pa.string()),      # stable identity, survives quarantine
    ("run_id",          pa.string()),
    ("ruleset_version", pa.string()),      # which manifest produced this verdict
    ("verdict",         pa.string()),      # clean | normalized | repaired | quarantined
    ("defect",          pa.string()),      # manifest defect class, "" when clean
    ("code",            pa.string()),      # quarantine code, "" when accepted
    ("area_delta",      pa.float64()),     # fraction; 0.0 when nothing moved
    ("geos_message",    pa.string()),      # verbatim explain_validity output
    ("geom_type_in",    pa.string()),
    ("geom_type_out",   pa.string()),
])

The verbatim GEOS message is worth the storage. When a defect class turns out to be systematically mis-repaired six months later, the only way to find every affected feature is to search on the message that GEOS produced at the time — the classifier’s mapping may have changed since. This is the same evidence discipline the lineage manifest applies to transformations, and the two tables are joined on run_id and feature_id when an auditor asks what happened to a specific parcel.

CI Integration

Gate this stage with a fixture corpus rather than with production data, so the test tells you about the rules and not about today’s input.

python
# tests/test_geometry_gate.py — pytest >=7, shapely >=2.0
import pytest
from shapely.geometry import Polygon

from quality.geometry_gate import run_gate
from quality.manifest import load_rules

RULES, PRECISION = load_rules("validity_rules.yaml")

BOWTIE = Polygon([(0, 0), (10, 0), (0, 10), (10, 10), (0, 0)])
COLLINEAR = Polygon([(0, 0), (5, 0), (10, 0), (0, 0)])
CLEAN = Polygon([(0, 0), (10, 0), (10, 10), (0, 10), (0, 0)])


def test_clean_geometry_is_untouched():
    result = run_gate(CLEAN, RULES, PRECISION)
    assert result.verdict == "clean"
    assert result.area_delta == 0.0


def test_bowtie_is_quarantined_for_type_drift():
    # make_valid turns a bowtie into a MultiPolygon; the default manifest
    # forbids type change, so this must NOT come back repaired.
    result = run_gate(BOWTIE, RULES, PRECISION)
    assert result.verdict == "quarantined"
    assert result.code == "GEOM_TYPE_DRIFT"


def test_degenerate_polygon_is_unrepairable():
    result = run_gate(COLLINEAR, RULES, PRECISION)
    assert result.verdict == "quarantined"


@pytest.mark.parametrize("ruleset", ["validity_rules.yaml"])
def test_manifest_rejects_unknown_defect_names(ruleset):
    # The negative control: the loader must refuse a defect class it cannot map.
    with pytest.raises(ValueError):
        load_rules(ruleset, extra_rules=[{"defect": "banana", "action": "repair"}])

The last test is the negative control the whole section insists on: it feeds the machinery input it must refuse. Without it, a typo in a defect name would silently disable a rule and every subsequent run would report a cleaner dataset than it has. Wire the suite into the same workflow that runs the schema drift gate, so a change to the rules and a change to the schema are reviewed under one board.

Deeper Implementation Walkthroughs

Two cases are common enough and fiddly enough to have their own runnable procedures. Repairing self-intersecting polygons with make_valid works through the structured-repair path end to end, including how to decide when a bowtie legitimately becomes two parts. Enforcing ring orientation and winding order in GeoJSON covers the defect that is not a validity failure at all but breaks map clients in the field — RFC 7946’s right-hand rule, and why a shapefile round-trip so often violates it.

Frequently Asked Questions

Does make_valid ever lose data? Not in the way buffer(0) does. buffer(0) resolves a self-intersection by keeping whichever lobe survives the offsetting arithmetic, discarding the rest with no signal. make_valid performs a structured node-and-rebuild that preserves every part, which is why it can return a MultiPolygon where the input was a Polygon. The data is not lost; the type contract is what changes, and that is why the gate rejects type drift unless a dataset opts in.

Why snap coordinates before testing validity rather than after repairing? Because snapping can itself create or resolve a self-intersection, and doing it after the test would mean the geometry that was judged is not the geometry that ships. Snapping first makes the pipeline idempotent: running the gate twice over its own output produces an identical result, which is the property that makes re-runs safe after an incident.

Should the repaired geometry replace the original in storage? Keep both, at least for the retention window the compliance policy defines. Publish the repaired geometry, retain the original in the quarantine partition addressed by feature_id and run_id. The cost is small — repairs are rare in a healthy dataset — and it is the only way to answer whether a boundary dispute originated in the source or in your pipeline.

How is this different from the validity check PostGIS already does on insert? ST_IsValid answers the same OGC question, and a CHECK constraint on insert is a good last line of defence. What it cannot do is classify the defect, apply a budgeted repair, record the area delta, or route the feature to a quarantine store that the quality report can address. A constraint tells you the load failed; this gate tells you which rule failed, by how much, and what happened to the feature.