Geospatial Schema Standardization & Mapping Automation

Implementing Local Government Data Dictionaries in Automated ETL Pipelines

Municipal GIS operations generate heterogeneous datasets across planning, public works, and emergency management, and each office names, types, and projects its attributes differently. When those layers flow into a shared catalog without a contract, a missing parcel_id, an unrecognized zoning code, or a State Plane survey extract silently corrupts spatial joins, public portals, and inter-agency exchanges. A data dictionary enforcement stage is the discrete pipeline step that turns a published Local Government Data Dictionary into runnable validation — checking every attribute against declared constraints, normalizing cross-agency naming, synchronizing coordinate reference systems, and routing non-conforming records through configurable channels. This stage operates at the core of Geospatial Schema Architecture & Standards Mapping, the discipline responsible for guaranteeing that every dataset entering the system conforms to an agreed structural and semantic contract.

This page is scoped to the dictionary-enforcement contract: declaring the field manifest, normalizing agency aliases, validating attributes and routing failures, and emitting an audit trail. It deliberately stops at the municipal-catalog boundary. Producing the formal metadata records that European and North American mandates require is the job of INSPIRE Directive Schema Compliance and FGDC Metadata Mapping; moving a validated municipal layer into a different platform’s native schema belongs to Cross-Platform Schema Translation. Here we focus on replacing manual reconciliation spreadsheets with a config-as-code stage whose pass/fail behaviour is deterministic and auditable.

Local Government Data Dictionary Enforcement Heterogeneous agency layers from planning, public works, and emergency management are checked against a version-controlled data dictionary registry that enforces domain and cardinality constraints; conforming records merge into the standardized catalog while violations route to a remediation channel. Planning Layer Public Works Emergency Mgmt Dictionary domain · card. Standardized Catalog interoperable Remediation channel pass fail

Declarative Dictionary Manifest

The foundation of automated enforcement is a version-controlled YAML manifest that defines expected fields, data types, cardinality, domain constraints, fallback routing, and tolerance thresholds. Rather than hardcoding validation logic, pipeline engineers externalize dictionary rules into a machine-readable registry. This decouples the published contract from the code that enforces it and lets non-developer GIS staff amend field rules through reviewable pull requests.

Each field entry explicitly declares mandatory or optional status, acceptable value domains, and fallback behaviour. The following manifest demonstrates a production-ready dictionary for parcel exports:

yaml
# schema_registry.yml  (validated against the field table below)
schema_version: "1.2.0"
target_crs: "EPSG:4326"           # canonical CRS every conforming record is reprojected to
fields:
  PARCEL_ID:
    type: string
    mandatory: true
    pattern: "^[A-Z]{2}-\\d{6}$"   # two-letter county prefix + six digits
  ZONING_CODE:
    type: string
    mandatory: true
    domain: ["R-1", "R-2", "C-1", "I-1", "AG"]   # closed value set
  LAST_MODIFIED:
    type: datetime
    mandatory: true
    format: "ISO8601"             # coerced from legacy formats before this gate
  OWNER_NAME:
    type: string
    mandatory: false
    fallback: "UNKNOWN"           # injected when absent; never quarantines
  ASSESSMENT_VALUE:
    type: float
    mandatory: false
    fallback: 0.0
tolerance:
  numeric_precision: 0.001        # absolute epsilon for float comparisons
  crs_buffer_meters: 0.5          # max post-reproject residual before reject

Every directive in the manifest maps to a column in the field contract below. Reviewers read this table, not the validator source, to understand what the stage enforces:

Directive Required? Applies to Behaviour when violated
type mandatory every field cast failure routes the record to reject
mandatory mandatory every field true + null → quarantine; false + null → inject fallback
domain optional string fields value outside the set → quarantine
pattern optional string fields regex mismatch → quarantine
format optional datetime fields unparseable after coercion → quarantine
fallback optional optional fields only substituted value when the field is absent
tolerance.numeric_precision optional float fields deviation under epsilon passes; over → reject
tolerance.crs_buffer_meters optional geometry residual over buffer after reproject → reject

Preprocessing: Alias Normalization & Temporal Coercion

The validation engine assumes one canonical schema, but agencies rarely deliver one. Before any field rule executes, two preprocessing passes reshape incoming layers into the dictionary’s expected form.

First, a deterministic alias table maps agency-specific column names to canonical dictionary keys. Planning may export APN, the assessor PID, and a legacy system PARCEL_NO; all three resolve to PARCEL_ID before the loop runs. Keeping aliases in the same version-controlled config as the dictionary means a new source is onboarded by adding rows, not by branching the validator:

yaml
# alias_table.yml
aliases:
  PARCEL_ID:   ["APN", "PID", "PARCEL_NO", "PARCELID"]
  ZONING_CODE: ["ZONE", "ZONING", "ZONE_CLS"]
  LAST_MODIFIED: ["EDIT_DATE", "MODIFIED", "LAST_EDIT"]

Second, temporal attributes require strict field renaming and type coercion rules before the ISO 8601 gate can run. Multi-decade municipal layers carry YYYYMMDD integers, Unix epoch seconds, and locale-specific strings. Apply explicit coercion that converts each known legacy format to a datetime object and records the original value and the rule applied in an audit column, so the transformation stays reproducible and reversible. Records whose dates cannot be coerced by any known rule are quarantined rather than guessed.

Validation Engine & Precision Guards

The validation stage executes a deterministic, ordered sequence per record: mandatory-null evaluation, type casting, domain or pattern checking, datetime parsing, optional-fallback injection, and finally CRS synchronization. Using geopandas and standard libraries, the engine reads the YAML dictionary and applies it to incoming GeoDataFrame or GeoPackage streams.

Deterministic Per-Record Validation Gate Sequence An incoming record flows left to right through four ordered gates read from the schema dictionary: a mandatory-null check, a type cast, a pattern or domain check, and a datetime parse. Failing any gate routes the record to the quarantine output with a logged reason. A record that clears every gate has optional-field fallbacks injected and is reprojected to the target CRS before being written to the compliant catalog. Incoming record mandatory null? type cast pattern / domain datetime parse Compliant catalog + fallback inject + to_crs(target) all pass Quarantine + compliance log fail → quarantine

Mandatory field checks must distinguish between structural nulls and acceptable omissions. When a required attribute is absent, the engine routes the record to quarantine rather than halting the batch; when an optional attribute is absent, it injects the declared fallback and continues. The deeper strategies for handling missing mandatory fields in municipal GIS exports show how to set soft-fail thresholds and hold non-conforming records for manual review. The following implementation enforces the YAML contract with explicit per-gate error capture:

python
# requires: geopandas >=0.14, pyproj >=3.6, pyyaml >=6.0, pandas >=2.0  (Python 3.10+)
import re
import logging
import yaml
import pandas as pd
import geopandas as gpd
from datetime import datetime
from pyproj.exceptions import CRSError

logger = logging.getLogger("data_dictionary")


def load_schema(config_path: str) -> dict:
    with open(config_path, "r", encoding="utf-8") as f:
        return yaml.safe_load(f)


def validate_and_route(
    gdf: gpd.GeoDataFrame, schema: dict
) -> tuple[gpd.GeoDataFrame, gpd.GeoDataFrame, list[dict]]:
    compliant: list[pd.Series] = []
    quarantine: list[pd.Series] = []
    audit: list[dict] = []

    for idx, row in gdf.iterrows():
        record_valid = True
        for field_name, rules in schema["fields"].items():
            value = row.get(field_name)
            is_mandatory = rules.get("mandatory", False)

            # 1. Mandatory-null gate
            if pd.isna(value) and is_mandatory:
                record_valid = False
                audit.append({"row": idx, "field": field_name, "rule": "mandatory", "outcome": "quarantine"})
                break

            # 2. Optional-fallback injection (never quarantines)
            if pd.isna(value) and not is_mandatory:
                row[field_name] = rules.get("fallback")
                continue

            # 3. Type cast — irrecoverable failures reject rather than quarantine
            try:
                if rules["type"] == "float":
                    value = float(value)
                    row[field_name] = value
            except (TypeError, ValueError):
                record_valid = False
                audit.append({"row": idx, "field": field_name, "rule": "type_cast", "outcome": "reject"})
                break

            # 4. Domain / pattern gate (string fields)
            if rules["type"] == "string" and rules.get("domain"):
                if value not in rules["domain"]:
                    record_valid = False
                    audit.append({"row": idx, "field": field_name, "rule": "domain", "outcome": "quarantine"})
                    break
            elif rules["type"] == "string" and rules.get("pattern"):
                if not re.match(rules["pattern"], str(value)):
                    record_valid = False
                    audit.append({"row": idx, "field": field_name, "rule": "pattern", "outcome": "quarantine"})
                    break

            # 5. Datetime gate (legacy formats already coerced upstream)
            elif rules["type"] == "datetime":
                try:
                    datetime.fromisoformat(str(value).replace("Z", "+00:00"))
                except ValueError:
                    record_valid = False
                    audit.append({"row": idx, "field": field_name, "rule": "datetime", "outcome": "quarantine"})
                    break

        (compliant if record_valid else quarantine).append(row)

    valid_gdf = gpd.GeoDataFrame(compliant, crs=gdf.crs)
    quarantine_gdf = gpd.GeoDataFrame(quarantine, crs=gdf.crs)

    # 6. Deterministic CRS synchronization with explicit error capture
    target = schema["target_crs"]
    if not valid_gdf.empty and valid_gdf.crs is not None and valid_gdf.crs.to_string() != target:
        try:
            valid_gdf = valid_gdf.to_crs(target)
        except CRSError as exc:
            logger.error("CRS sync to %s failed: %s", target, exc)
            raise

    return valid_gdf, quarantine_gdf, audit

Type coercion respects the manifest’s tolerance windows: numeric deviations under the configured numeric_precision epsilon pass, while a post-reproject residual exceeding crs_buffer_meters triggers a reject rather than silently shifting geometry. CRS synchronization itself is a narrow slice of CRS Normalization & Sync; pipelines that ingest many authority projections should delegate the heavy lifting to a dedicated projection normalization workflow and treat the dictionary’s to_crs step as a final assertion rather than the primary reprojection engine.

Failure Modes & Fallback Routing

No record exits the stage silently. Every outcome is one of three terminal states — compliant, quarantine, or reject — and each failure type maps to a deterministic recovery action so operators never have to guess why a layer was held back.

Failure type Likely cause Outcome Deterministic recovery action
Missing mandatory field vendor transform dropped a column; alias not mapped quarantine add the missing alias or backfill from source, then re-ingest
Out-of-domain value new zoning code not yet in the dictionary quarantine extend the domain list via PR, or correct the source value
Pattern mismatch malformed PARCEL_ID (wrong prefix or digit count) quarantine fix the source key; verify the county prefix table
Type-cast failure non-numeric bytes in a float field reject route to reject log; flag the source extract as corrupt
Unparseable datetime legacy format with no coercion rule quarantine add a coercion rule for the format and reprocess
CRS sync error undefined or unsupported source CRS raise / halt batch assign the correct source CRS upstream and rerun

Quarantined records are recoverable by a clerk or a corrected rule and stay in the remediation queue; rejected records failed an irrecoverable operation and go straight to the reject log. The CRS sync error is the one case that halts the batch deliberately, because an unknown spatial reference makes every downstream geometry untrustworthy. When transient infrastructure faults (a grid download timing out, a locked GeoPackage) cause that halt, wrap the run in the error handling and retry logic patterns so genuine transients retry with backoff instead of quarantining valid data.

Compliance Reporting Output

The stage’s audit trail is what makes enforcement defensible to oversight bodies. Alongside the compliant and quarantine GeoPackages, the engine writes a machine-readable compliance summary that records, per run: the dictionary schema_version applied, total records in and out, counts by outcome, and a per-record reject log keyed by row index, field, and violated rule. The audit list returned above serializes directly to JSON or Parquet:

json
{
  "schema_version": "1.2.0",
  "run_id": "2026-06-25T08:14:02Z",
  "source_layer": "planning_parcels_q2",
  "records_in": 18422,
  "compliant": 18097,
  "quarantined": 311,
  "rejected": 14,
  "violations": [
    {"row": 402, "field": "ZONING_CODE", "rule": "domain", "outcome": "quarantine"},
    {"row": 1187, "field": "PARCEL_ID", "rule": "pattern", "outcome": "quarantine"}
  ]
}

Pinning the schema_version in every report means a dataset rejected last quarter can be re-evaluated against the exact dictionary revision that judged it. For large batches, write the audit trail as Parquet via pyarrow (>=14) so compliance history stays queryable without rehydrating geometry, and feed those counts into the same lineage manifests that the standards-mapping clusters consume for formal metadata generation.

CI Integration

Dictionary enforcement belongs in continuous integration, not just nightly ETL. The following GitHub Actions workflow runs validation on pull requests that touch ingest data, the dictionary, or the validator, and blocks the merge when mandatory compliance thresholds are breached by exiting non-zero:

yaml
# .github/workflows/schema-validation.yml
name: Schema Compliance Validation
on:
  pull_request:
    paths:
      - 'data/**/*.gpkg'
      - 'config/schema_registry.yml'
      - 'config/alias_table.yml'
      - 'scripts/validate.py'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: pip install "geopandas>=0.14" "pyproj>=3.6" "pyarrow>=14" "pyyaml>=6.0" pandas
      - name: Run dictionary validation
        run: python scripts/validate.py --config config/schema_registry.yml --input data/ingest/
      - name: Upload compliance report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: compliance-report
          path: reports/

A companion pytest fixture loads schema_registry.yml, feeds known-bad fixtures through validate_and_route, and asserts that each crafted violation lands in the expected outcome bucket — so a change to the dictionary that accidentally loosens a domain or pattern fails the build rather than reaching production. For authoritative references, validate spatial containers against the OGC GeoPackage Standard and align metadata output with the FGDC Content Standard for Digital Geospatial Metadata.

By externalizing Local Government Data Dictionaries into version-controlled manifests and enforcing them through a deterministic engine with explicit failure routing, agencies eliminate manual reconciliation overhead and guarantee that downstream analytics, public portals, and inter-agency exchanges consume structurally sound, temporally consistent, and spatially aligned geospatial assets.

Frequently Asked Questions

Why externalize the data dictionary into YAML instead of hardcoding validation in Python? A version-controlled manifest lets GIS analysts who do not write Python review and amend field rules through pull requests, and it produces a diffable record of every schema change. Hardcoded validation couples the contract to code releases and hides the rule set from the people who own the data.

Should alias normalization run before or after validation? Before. Agencies export the same concept under names like APN, PID, and PARCEL_NO; mapping every variant to the canonical dictionary key first means the engine evaluates one stable schema instead of duplicating every rule per agency.

What is the difference between quarantining a record and rejecting it? Quarantine holds records a human or corrected rule can recover — a missing mandatory field or an unknown zoning code — for remediation and re-ingest. Reject marks records that failed an irrecoverable operation such as a type cast on garbage bytes and routes them straight to the reject log.

How do legacy date formats fit into ISO 8601 validation? A coercion step converts each known legacy format — YYYYMMDD integers, Unix epoch seconds, locale strings — to a datetime object before the ISO 8601 gate runs, recording the original value and applied rule in an audit column so the transformation is reproducible.