Geospatial Schema Standardization & Mapping Automation

Matching Features Across Vintages with Stable IDs

Deduplication within a delivery asks whether two records describe the same thing right now. Matching across vintages asks something harder: whether the parcel in this year’s delivery is the parcel from last year’s, after a boundary correction, a split, a merge, or a renumbering by a source system that considers its identifiers internal. Get it wrong in one direction and every feature looks new, so every downstream join breaks and the change report says the municipality was rebuilt. Get it wrong in the other and a genuine subdivision is recorded as a boundary edit, quietly erasing the event that mattered. This procedure implements the cross-delivery half of duplicate detection and feature deduplication, inside Spatial Data Quality Validation & Geometry Integrity.

The steps map to configure (Step 1, identifier-first matching), execute (Step 2, geometric fallback), validate (Steps 3–4, classification and identifier policy), and log (Step 5, the crosswalk). The organizing principle is that identity is a decision the publisher makes and records, never something inferred fresh on each run.

Prerequisites checklist

Step 1: Match on the persistent identifier before touching geometry

python
# geopandas >= 0.14 — Python 3.10+
import geopandas as gpd

CRS = "EPSG:25832"
previous = gpd.read_file("vintages/2025/parcels.gpkg").to_crs(CRS).set_index("stable_id")
current = gpd.read_file("vintages/2026/parcels.gpkg").to_crs(CRS).set_index("stable_id")

carried = previous.index.intersection(current.index)
dropped = previous.index.difference(current.index)      # candidates for retired / split / merged
appeared = current.index.difference(previous.index)     # candidates for created / split / merged

Doing this first is not just an optimization, though on a large layer it removes 95% of the work. It is a statement about authority: where a stable identifier exists on both sides, it decides the match, and no geometric evidence overrides it. A parcel whose boundary was corrected by four metres is still the same parcel if the identifier says so, and a pipeline that lets geometry outvote the identifier will re-invent identities every time a survey is refreshed.

Step 2: Compute overlap ratios for everything left over

python
# shapely >= 2.0 — Python 3.10+
from shapely import STRtree


def overlap_ratios(old_geoms, new_geoms):
    """Yield (old_index, new_index, ratio_of_old, ratio_of_new) for intersecting pairs."""
    tree = STRtree(new_geoms)
    for i, old in enumerate(old_geoms):
        for j in tree.query(old, predicate="intersects").tolist():
            shared = old.intersection(new_geoms[j]).area
            if shared <= 0.0:
                continue
            yield i, j, shared / old.area, shared / new_geoms[j].area

Two ratios, not one. The fraction of the old feature covered by the new one and the fraction of the new feature covered by the old one answer different questions, and their combination is what distinguishes the lifecycle events. A split produces one old feature with several new ones, each covering a small fraction of the old but nearly all of themselves; a merge is the mirror image. Reporting a single Jaccard-style similarity collapses both into “somewhat similar” and loses the event.

Lifecycle Events Read From Two Overlap Ratios Four panels each show a dashed old outline and a solid new outline. In retained, the two coincide and both ratios are above ninety-eight per cent. In changed, the new outline is offset slightly and both ratios sit between eighty and ninety-eight per cent. In split, one dashed old outline contains two solid new outlines, so each new feature covers only part of the old but almost all of itself. In merged, two dashed old outlines are covered by one solid new outline, so each old feature is almost entirely covered while the new one is only partly covered by each. The caption notes that a single similarity score cannot separate split from merged. Retained old→new 0.99 · new→old 0.99 same identifier, no event Changed old→new 0.88 · new→old 0.88 same identifier, boundary edited Split old→new 0.47 each · new→old 0.99 old retired, two new identifiers Merged old→new 0.99 each · new→old 0.49 both old retired, one new identifier a single similarity score cannot separate the third panel from the fourth

Step 3: Classify the lifecycle event from the ratio pair

python
# Python 3.10+
RETAINED = 0.98        # both ratios at or above → geometry unchanged
CHANGED = 0.80         # both ratios at or above → same feature, edited boundary
PART_OF = 0.90         # one side almost fully covered → a split or merge component


def classify(old_to_new: float, new_to_old: float, old_partners: int, new_partners: int) -> str:
    if old_to_new >= RETAINED and new_to_old >= RETAINED:
        return "retained"
    if old_to_new >= CHANGED and new_to_old >= CHANGED and old_partners == 1 == new_partners:
        return "changed"
    if new_to_old >= PART_OF and new_partners == 1 and old_partners > 1:
        return "split"          # this new feature is one piece of one old feature
    if old_to_new >= PART_OF and old_partners == 1 and new_partners > 1:
        return "merged"         # this old feature was absorbed into one new feature
    return "ambiguous"

ambiguous is a real outcome and must stay in the vocabulary. A parcel that was simultaneously split and had its boundary corrected produces ratios that fit no clean pattern, and forcing it into “changed” writes a false history. Ambiguous relationships go to review with both geometries attached, and their count over successive deliveries is a direct measure of how disruptive the source’s editing practices are.

Step 4: Assign identifiers by the lifecycle rule, and never reuse one

Event Identifier outcome Rationale
retained Keep the existing stable identifier Nothing happened worth recording beyond the vintage
changed Keep the identifier; record the boundary edit and its area delta The feature persists; its shape was corrected
split Retire the parent; mint one new identifier per child, each citing the parent Two things now exist where one did; neither is “the” original
merged Retire every parent; mint one new identifier citing all parents The new feature is not either parent
created Mint a new identifier with no predecessor Genuinely new ground, e.g. reclaimed land or a first registration
retired Retire the identifier with no successor Deregistered; the identifier is never reissued
ambiguous Hold both vintages; assign nothing until reviewed Guessing here writes a false history that is hard to unwind

The one rule with no exceptions is that a retired identifier is never reused. Reissuing 14-023-9 to a different piece of ground five years later silently corrupts every archived document, report and correspondence that cited it, and the corruption is undetectable from the data alone.

Whether split should keep the parent identifier for the largest child is the argument this table settles deliberately. Keeping it is tempting — most downstream joins survive — but it asserts that the 60% child is the parent, which is untrue and produces area histories that show a parcel shrinking by 40% with no event. Minting two children and citing the parent keeps the history honest, and the crosswalk in Step 5 is what keeps the joins working.

Step 5: Publish the crosswalk

python
# pyarrow >= 14 — Python 3.10+
import pyarrow as pa

CROSSWALK_SCHEMA = pa.schema([
    ("from_vintage",   pa.string()),      # "2025"
    ("to_vintage",     pa.string()),      # "2026"
    ("event",          pa.string()),      # retained | changed | split | merged | created | retired
    ("predecessor_id", pa.string()),      # "" for created
    ("successor_id",   pa.string()),      # "" for retired
    ("old_to_new",     pa.float64()),
    ("new_to_old",     pa.float64()),
    ("area_delta",     pa.float64()),     # signed, in CRS units
    ("decided_by",     pa.string()),      # stable_id | overlap | manual_review
])

decided_by is the field that makes the table auditable rather than merely useful: it distinguishes relationships the identifier settled from those geometry inferred and from those a person adjudicated. When a downstream system disputes a match, that column is the first thing anyone looks at. The crosswalk joins the lineage manifest by vintage and belongs in the published dataset metadata as a distribution of its own — consumers need it more than they need most of what portals publish.

Verification

python
# pytest >= 7
def test_boundary_correction_keeps_the_identifier(vintage_pair):
    crosswalk = match_vintages(*vintage_pair)
    row = crosswalk.loc["PARCEL-14-023-9"]
    assert row.event == "changed" and row.successor_id == "PARCEL-14-023-9"

def test_subdivision_retires_parent_and_mints_two_children(vintage_pair):
    rows = crosswalk[crosswalk.predecessor_id == "PARCEL-14-050-0"]
    assert set(rows.event) == {"split"} and len(rows) == 2
    assert "PARCEL-14-050-0" not in set(crosswalk.successor_id)

def test_no_identifier_is_ever_reused(all_crosswalks):
    retired = {r.predecessor_id for r in all_crosswalks if r.event in {"split", "merged", "retired"}}
    minted = {r.successor_id for r in all_crosswalks if r.event in {"split", "merged", "created"}}
    assert retired.isdisjoint(minted)

The third test runs over the whole crosswalk history rather than one pair of vintages, and it is the assertion most worth keeping: identifier reuse is introduced by a well-meaning change years after the code was written, and nothing else detects it.

A healthy annual delivery logs a distribution that should be boring:

text
INFO quality.vintage retained=212,004 changed=1,918 split=214 merged=97
INFO quality.vintage created=381 retired=142 ambiguous=6 decided_by_stable_id=213,922

Troubleshooting

Symptom Likely cause Fix
Every feature classified as created and retired The source reissued its internal identifiers and they were used as the stable identifier Mint publisher-owned identifiers on first sight and store the source identifier as an ordinary attribute
Splits reported as changed PART_OF threshold too low, so one child looks like the whole parent Raise the threshold and require old_partners > 1 explicitly, as in the classifier above
Large ambiguous population Two vintages in different datums, so everything is offset Check the transformation history before the match — see datum transformation fallback chains
Overlap computation is slow Ratios computed for every pair rather than index candidates Query the index per old feature, as in Step 2; never cross-join the vintages
Downstream joins break after a subdivision The crosswalk is computed but not published Publish it as a distribution alongside the dataset, not as an internal table
A retired identifier reappears The mint function derives identifiers from source attributes that were recycled Mint from a counter or a UUID and record the derivation; never derive identity from mutable attributes