Best Practices for Spatial Data Dictionary Versioning Jump to heading#

Spatial data dictionaries degrade rapidly when version control is treated as an afterthought. Government tech teams and Python ETL engineers routinely hit schema drift, broken coordinate reference system mappings, and compliance failures when spatial attributes evolve without strict versioning — a zoning_code enum quietly gains a value, a geometry column flips from Polygon to MultiPolygon, or an SRID changes under a consumer that never asked for it. This procedure is one of the Cross-Platform Schema Translation implementations inside Geospatial Schema Architecture & Standards Mapping: it makes dictionary evolution deterministic through a versioned registry, automated diffing, and threshold-gated ingestion, so a schema change can never silently break a downstream reader.

The walkthrough maps 1:1 to the ETL phases this site standardizes on — configure (Step 1, the registry; Step 2, the drift detector), validate (Step 3, threshold gates), and log (Step 4, CI gating; Step 5, fallback routing and audit) — so each step drops straight into an existing stage rather than living as a standalone script. It assumes you have already decided which fields belong in the dictionary; the surrounding translation workflow owns that mapping, while this page owns the contract that governs how those fields are allowed to change over time.

Prerequisites checklist Jump to heading#

Confirm the environment before standing up the registry. The most common silent failure is a validator built against a different PROJ build than the one that wrote the data, so pin and verify the spatial stack explicitly.

• jsonschema >= 4.21 installed for baseline type and pattern constraints (python -c "import jsonschema; print(jsonschema.__version__)").
• pyproj >= 3.6 and shapely >= 2.0 against the same PROJ build (python -c "import pyproj; print(pyproj.proj_version_str)" reports >= 9.3); shapely.make_valid is the topology-repair API in 2.x.
• PyYAML >= 6.0 available so version manifests can be authored in YAML and loaded as the JSON Schema instance.
• A version-controlled repository for schema manifests with immutable history (tag or content-hash each release; never force-push a published version).
• EPSG registry reachable offline — pyproj.CRS.from_epsg(4326) resolves from the bundled proj.db without network access so SRID checks are reproducible in CI.
• A writable path for the quarantine/ directory and the transformation-lineage log the fallback layer appends to.

Step 1: Establish a deterministic schema registry Jump to heading#

Every spatial dataset must be anchored to a machine-readable schema definition that tracks structural changes independently of the underlying geometry. Store dictionary definitions in version-controlled YAML or JSON Schema files, explicitly tagging major, minor, and patch increments, and align field naming with your Cross-Platform Schema Translation manifests so a version bump means the same thing across ArcGIS, PostGIS, and GeoPackage targets.

Major versions indicate breaking changes such as CRS transformations, geometry-type alterations, or mandatory-field removals. Minor versions introduce additive attributes or extended enumerations. Patch versions correct typographical errors or update precision tolerances without altering downstream ETL logic. Maintain the registry as immutable snapshots: each manifest carries a version string, effective_date, and a deprecation_policy block so lifecycle management is fully declarative.

# schema_v2.1.0.yaml  (PyYAML >= 6.0; loaded as the JSON Schema instance)
version: "2.1.0"
effective_date: "2024-06-01"
deprecation_policy:
  sunset_date: "2025-06-01"
  fallback_version: "2.0.0"
  migration_script: "scripts/migrate_v2.0_to_v2.1.py"
geometry:
  type: "MultiPolygon"
  srid: 4326
  precision_meters: 0.01
  validation: "must_be_valid_topology"
attributes:
  parcel_id:    { type: "string", nullable: false, pattern: "^[A-Z]{2}-\\d{6}$" }
  zoning_code:  { type: "string", nullable: true,  enum: ["R1", "R2", "C1", "I1"] }
  last_updated: { type: "string", nullable: false, format: "date-time" }
  area_sqm:     { type: "number", nullable: true,  minimum: 0 }

The manifest separates mandatory directives — which every consumer relies on — from optional ones that a minor bump may add:

The increment a schema change earns is not cosmetic: it determines whether downstream consumers can keep reading without code changes, must adapt, or must be routed through a compatibility shim. The diagram below shows how the three increment classes act on a 1.4.2 baseline and how a sunset major version falls back to its declared fallback_version.

Step 2: Implement automated drift detection Jump to heading#

Manual schema reviews fail at scale. Deploy a Python validation step that computes structural diffs between the active dictionary and incoming data payloads. Use jsonschema for baseline type constraints, then layer a spatial-specific drift detector that flags coordinate-precision degradation, missing mandatory fields, and unauthorized CRS shifts. Treat a CRS shift as drift, not as something to silently reproject — that reprojection belongs to CRS Normalization & Sync, and the dictionary’s job is only to assert that the SRID still matches the contract.

The validator below handles missing fields gracefully, checks SRID alignment, and auto-repairs invalid topology before a record reaches production storage.

# jsonschema >= 4.21, pyproj >= 3.6, shapely >= 2.0
import json
import sys
from typing import Any

import pyproj
import shapely
from jsonschema import validate, ValidationError
from shapely.errors import ShapelyError
from shapely.geometry import shape


class SpatialSchemaValidator:
    def __init__(self, schema_path: str) -> None:
        with open(schema_path, "r") as f:
            self.schema = json.load(f)
        self.expected_srid = self.schema.get("geometry", {}).get("srid")
        self.precision_m = self.schema.get("geometry", {}).get("precision_meters")
        # Confirm the declared SRID resolves offline; a bad code is itself drift.
        self.target_crs = pyproj.CRS.from_epsg(self.expected_srid)

    def validate_record(self, record: dict[str, Any]) -> dict[str, Any]:
        """Validate a single GeoJSON-like feature against the active dictionary."""
        errors: list[str] = []
        warnings: list[str] = []

        # 1. Missing mandatory fields -> hard fail (drift against the contract).
        props = record.get("properties", {})
        for attr, rules in self.schema.get("attributes", {}).items():
            if not rules.get("nullable", True) and attr not in props:
                errors.append(f"Missing mandatory field: {attr}")

        # 2. Baseline type / pattern / enum constraints via JSON Schema.
        try:
            validate(instance=record, schema=self.schema)
        except ValidationError as ve:
            errors.append(f"Schema violation: {ve.message}")

        # 3. Spatial validation + CRS-mismatch detection.
        try:
            geom = shape(record.get("geometry"))
            if not geom.is_valid:
                geom = shapely.make_valid(geom)  # shapely 2.x repair API
                warnings.append("Invalid topology auto-repaired")

            input_crs = props.get("crs_srid")
            if input_crs and int(input_crs) != self.expected_srid:
                errors.append(
                    f"CRS mismatch: expected EPSG:{self.expected_srid}, "
                    f"got EPSG:{input_crs}"
                )
        except (ShapelyError, TypeError, KeyError) as exc:
            errors.append(f"Geometry processing failed: {exc}")

        return {
            "status": "PASS" if not errors else "FAIL",
            "errors": errors,
            "warnings": warnings,
            "record_id": props.get("parcel_id", "unknown"),
        }


if __name__ == "__main__":
    validator = SpatialSchemaValidator("schema_v2.1.0.json")
    test_feature = {
        "type": "Feature",
        "properties": {
            "parcel_id": "AB-123456",
            "zoning_code": "R1",
            "last_updated": "2024-05-15T10:00:00Z",
        },
        "geometry": {
            "type": "Polygon",
            "coordinates": [[[-122.4, 37.7], [-122.4, 37.8],
                             [-122.3, 37.8], [-122.3, 37.7], [-122.4, 37.7]]],
        },
    }
    result = validator.validate_record(test_feature)
    print(json.dumps(result, indent=2))
    sys.exit(0 if result["status"] == "PASS" else 1)

Step 3: Enforce thresholds and routing rules Jump to heading#

Silent data corruption occurs when validation lacks hard boundaries. Configure explicit tolerance thresholds so degraded datasets cannot propagate downstream. Apply the following rules during ingestion, in order, short-circuiting on the first hard failure:

• Mandatory field coverage: reject payloads where non-nullable attribute coverage falls below 98%.
• Geometry validity: quarantine batches where valid topology drops under 99.5%; every auto-repair attempt is logged.
• CRS mismatch tolerance: zero tolerance — any deviation from the declared SRID routes the record straight to staging.
• Precision degradation: reject coordinate sets exceeding the schema-defined precision_meters threshold by more than 2x.
• Fallback routing: route failed records to a quarantine/ directory with a .failed.json manifest and fire an automated alert to the data-stewardship team.

These thresholds compose into a single deterministic gate: every incoming record is tested against each rule in sequence, and the first hard failure short-circuits the record into quarantine rather than letting it reach production storage. The diagram below traces that decision path, showing which checks merely warn (auto-repair, then continue) and which checks reject outright. The numeric deviation matrices behind the CRS and precision gates are standardized in Unit Conversion & Tolerance Thresholds; this step only enforces them.

Step 4: Gate the registry in CI and handle pipeline failures Jump to heading#

Automated drift detection must run on every pull request and nightly ingestion cycle. Configure CI to execute schema validation before merging or deploying ETL jobs; reserve retry logic only for transient failures such as fetching a remote manifest, using the Error Handling & Retry Logic patterns rather than retrying a deterministic drift failure. The following GitHub Actions workflow blocks merges on critical drift and routes failures to a dedicated artifact store.

# .github/workflows/spatial-schema-validation.yml
name: Spatial Schema Validation
on:
  pull_request:
    paths:
      - 'schemas/*.yaml'
      - 'data/**/*.geojson'
  schedule:
    - cron: '0 2 * * *'   # nightly drift check

jobs:
  validate-spatial-dict:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: pip install "jsonschema>=4.21" "pyproj>=3.6" "shapely>=2.0" "PyYAML>=6.0"
      - name: Run schema drift check
        id: validation
        continue-on-error: true
        run: |
          python -m spatial_validator \
            --schema schemas/schema_v2.1.0.yaml \
            --input data/latest.geojson
      - name: Route CI failure to quarantine
        if: steps.validation.outcome == 'failure'
        run: |
          mkdir -p quarantine
          cp data/latest.geojson quarantine/latest_failed.geojson
          echo "::warning::Spatial drift detected. Records quarantined for manual review."
      - name: Upload quarantine artifacts
        if: steps.validation.outcome == 'failure'
        uses: actions/upload-artifact@v4
        with:
          name: failed-spatial-records
          path: quarantine/

Step 5: Route incompatible consumers to a fallback version Jump to heading#

Government and enterprise spatial pipelines must adhere to strict metadata standards, and a major version bump cannot strand a consumer that has not migrated. When a legacy system cannot read a newer dictionary version, the deprecation policy fires and the record is served from the declared fallback_version through an explicit compatibility layer that strips non-essential attributes, downgrades geometry precision, and remaps deprecated fields to their historical equivalents. Cross-platform target mapping — INSPIRE, FGDC, and ISO 19115 alignment — is owned by Cross-Platform Schema Translation; this layer only handles the version downgrade, and every transformation it applies is written to an audit log so a compliance reviewer can reconstruct exactly what each consumer received.

# PyYAML >= 6.0
import logging

logger = logging.getLogger("dict_fallback")


def downgrade_to_fallback(record: dict, policy: dict) -> dict:
    """Remap a record to the declared fallback version and log the transformation."""
    fallback = policy["fallback_version"]
    field_map = policy.get("field_renames", {})   # new_name -> historical_name
    props = dict(record.get("properties", {}))

    for new_name, old_name in field_map.items():
        if new_name in props:
            props[old_name] = props.pop(new_name)

    # Drop attributes that did not exist in the fallback contract.
    for added in policy.get("added_attributes", []):
        props.pop(added, None)

    record["properties"] = props
    record.setdefault("properties", {})["schema_version"] = fallback
    logger.info(
        "downgraded record_id=%s to fallback=%s renames=%d dropped=%d",
        props.get("parcel_id", "unknown"), fallback,
        len(field_map), len(policy.get("added_attributes", [])),
    )
    return record

For the authoritative spatial constraints these contracts build on, reference the GeoJSON specification (IETF RFC 7946) for geometry and property structure.

Verification Jump to heading#

Confirm the registry and gate actually behave — do not trust an exit code of 0 alone. Assert that a conforming record passes, that a CRS-mismatched record is rejected, and that the fallback downgrade is reversible:

v = SpatialSchemaValidator("schema_v2.1.0.json")

# 1. A conforming record passes with no errors.
good = v.validate_record(test_feature)
assert good["status"] == "PASS", good["errors"]

# 2. A wrong SRID is rejected with zero tolerance.
bad = dict(test_feature)
bad["properties"] = {**test_feature["properties"], "crs_srid": 3857}
assert v.validate_record(bad)["status"] == "FAIL"
assert any("CRS mismatch" in e for e in v.validate_record(bad)["errors"])

# 3. The fallback downgrade stamps the historical version.
policy = {"fallback_version": "2.0.0", "added_attributes": ["area_sqm"]}
out = downgrade_to_fallback(dict(test_feature, properties=dict(test_feature["properties"])), policy)
assert out["properties"]["schema_version"] == "2.0.0"
assert "area_sqm" not in out["properties"]

A healthy CI run prints no Schema violation or CRS mismatch lines for conforming data, leaves the quarantine/ directory empty, and writes one lineage entry per accepted record. On the command line, python -m spatial_validator --schema schemas/schema_v2.1.0.yaml --input data/latest.geojson should exit 0 and report status: PASS.

Troubleshooting Jump to heading#

Related Jump to heading#

• Cross-Platform Schema Translation — the parent workflow that maps each dictionary version onto Esri, PostGIS, and GeoPackage targets
• Geospatial Schema Architecture & Standards Mapping — the standards framework these versioned dictionaries plug into
• CRS Normalization & Sync — owns the reprojection a SRID-drift detection only flags
• Unit Conversion & Tolerance Thresholds — the deviation matrices behind the precision and CRS gates in Step 3
• Error Handling & Retry Logic — backoff wrappers for transient manifest-fetch failures in the CI gate