Geospatial Schema Standardization & Mapping Automation

Attribute Domain & Code List Validation

Type coercion answers whether a value can be stored in a column. Domain validation answers whether it is allowed to be. Those are different questions, and the gap between them is where most published spatial data goes wrong: a land-use column typed as text will happily accept residental, a building-height column typed as float will happily accept -3.2, and a status column will happily accept an empty string on a field the target schema declares mandatory. None of these fails a cast. All of them fail a conformance test, usually at the regulator rather than in the pipeline. This is Gate 4 of Spatial Data Quality Validation & Geometry Integrity, and it runs after geometry because a feature with an undefined geometry has no business being argued about attribute by attribute.

The scope boundary is worth stating precisely because the neighbouring stage looks similar. Field renaming and type coercion owns getting values into the right columns with the right physical types; it is a transformation. This page owns whether the resulting values are members of their declared domain; it is a judgement, and it never rewrites a value. A code that is not in the register is not silently mapped to the nearest one — it is quarantined, because guessing at meaning is exactly the failure that domain validation exists to prevent.

Three Kinds of Domain, Three Different Risks

The word “domain” covers three quite different constructs, and conflating them is why so many validators are simultaneously too strict and too weak.

Local Enumeration, Numeric Range, Governed Register Three panels compare the domain kinds. A local enumeration lists values your own schema owns, changes only when you change it, and can be validated from the repository. A numeric range is defined by minimum, maximum and unit, and its main risk is unit drift rather than membership. A governed register is owned by an external authority, has a URI and a version, changes on someone else's schedule, and must be pinned and cached so a pipeline verdict never depends on network state. Local enumeration owned by your target schema changes only in a reviewed commit validated from the repository RISK drifting from the source vocabulary the list stops matching reality Numeric range minimum, maximum, and a unit inclusive bounds stated explicitly null policy separate from bounds RISK unit drift, not membership feet arriving in a metre column Governed register owned by an external authority identified by URI and version amended on someone else's schedule RISK a verdict that depends on the network pin the version, cache the members

The third panel is the one that breaks pipelines quietly. An INSPIRE code list is a resource in a registry with its own identifier and version history; a value that was legal in one version can be deprecated in the next, and a new value can appear that your enumeration has never heard of. A validator that fetches the register at run time makes every verdict a function of network state and of someone else’s release timing. A validator that hard-codes the members passes for a year and then rejects a whole delivery for reasons nobody can explain. The pinned-cache pattern, worked through in validating code list values against INSPIRE registers, is the answer to both.

Declarative Configuration Manifest

yaml
# attribute_domains.yaml — pandera >=0.18, pyyaml >=6.0
manifest_version: "3.0.0"          # MANDATORY
dataset: "municipal_parcels"       # MANDATORY
null_policy: "explicit"            # MANDATORY: explicit | permissive
                                   #   explicit = empty string is NOT null; it is a value
fields:
  - name: parcel_id                # MANDATORY
    kind: identifier               # MANDATORY: identifier | enumeration | range | register | free_text
    required: true                 # MANDATORY
    pattern: "^[0-9]{2}-[0-9]{3}-[0-9]{1}$"   # OPTIONAL: anchored regex
    unique: true                   # OPTIONAL: default false
  - name: land_use
    kind: register                 # externally governed
    required: true
    register_uri: "http://inspire.ec.europa.eu/codelist/HILUCSValue"   # MANDATORY for register
    register_version: "2023-04-11"                                     # MANDATORY: pinned
    cache_path: "registers/hilucs-2023-04-11.json"                     # MANDATORY: offline copy
    allow_deprecated: false        # OPTIONAL: default false
  - name: building_height_m
    kind: range
    required: false
    minimum: 0.0                   # OPTIONAL, inclusive
    maximum: 300.0                 # OPTIONAL, inclusive
    unit: "metre"                  # MANDATORY for range: guards unit drift
    null_action: pass              # OPTIONAL: pass | quarantine; default pass when not required
  - name: ownership_status
    kind: enumeration
    required: true
    values: ["public", "private", "mixed", "unknown"]   # MANDATORY for enumeration
    case_sensitive: true           # OPTIONAL: default true — see notes
  - name: last_survey_date
    kind: range
    required: true
    minimum: "1900-01-01"
    maximum: "today"               # OPTIONAL keyword: evaluated once per run, then frozen
    unit: "date"
conditional_rules:
  - when: {field: ownership_status, equals: "private"}
    then_required: ["owner_reference"]     # conditional mandatory field
Field Required Meaning
manifest_version Mandatory Stamped on every rejection so a verdict is reproducible
null_policy Mandatory Whether an empty string counts as null; the single most common source of false conformance
kind Mandatory Closed vocabulary; determines which other fields are required
required Mandatory Whether a null is a rejection regardless of the domain test
register_uri + register_version Conditional Both mandatory for kind: register; the version is pinned, never “latest”
cache_path Conditional The offline member list the run actually validates against
unit Conditional Mandatory for numeric and date ranges; a range without a unit is not a rule
case_sensitive Optional Defaults to true; case-insensitive matching hides upstream vocabulary drift
conditional_rules Optional Cross-field obligations, evaluated after per-field rules pass

Two defaults are deliberately strict. case_sensitive: true means Private fails against private, which feels pedantic until you notice that a source system that starts capitalizing its values has changed something, and a case-insensitive validator will never tell you. allow_deprecated: false means a value that the register has retired is a rejection rather than a warning, because a deprecated code in a published dataset is a conformance finding waiting to happen.

Preprocessing Requirements

Casting is complete and typed. Domain rules operate on typed values, not on strings that look like numbers. building_height_m must already be a float when it arrives, which is the responsibility of the type coercion stage. A range check against a string column silently compares lexicographically and reports that "9.5" exceeds "300.0".

Nulls are canonical. Every reader has its own idea of missing: shapefiles use empty strings, some CSV exports use the literal NULL, ArcGIS exports sometimes use <Null>, and a few sources use -9999. Normalize these to a single null representation before this gate, and record which sentinel each source used, because “how many nulls were sentinels” is a data-quality measure in its own right.

Registers are fetched at build time, not run time. The cache file named in the manifest is produced by a separate, scheduled job that reads the register, records its version and content hash, and commits the result. The validation run reads only the committed file. This is the same reproducibility argument that governs PROJ grid pinning: a pipeline whose verdict depends on a network fetch is not deterministic.

Execution Engine & Precision Guards

python
# quality/domains.py — pandera >=0.18, pandas >=2.1, pyarrow >=14 — Python 3.10+
import json
import logging
from dataclasses import dataclass
from pathlib import Path

import pandas as pd

logger = logging.getLogger("quality.domains")

SENTINEL_NULLS = {"", "NULL", "<Null>", "N/A", "-9999"}


@dataclass(frozen=True)
class Rejection:
    feature_id: str
    field: str
    code: str          # ATTR_UNDEFINED_CODE, ATTR_MANDATORY_NULL, ATTR_OUT_OF_RANGE, ...
    observed: str      # the value as received, truncated for the log
    expected: str      # a compact description of the domain


def load_register(spec: dict) -> tuple[frozenset[str], frozenset[str]]:
    """Return (active_members, deprecated_members) from the pinned cache file."""
    path = Path(spec["cache_path"])
    if not path.is_file():
        raise FileNotFoundError(
            f"register cache missing for {spec['name']}: {path}. "
            "Run the register sync job; never fetch at validation time."
        )
    payload = json.loads(path.read_text(encoding="utf-8"))
    if payload["version"] != spec["register_version"]:
        raise ValueError(
            f"{spec['name']}: cache is version {payload['version']}, "
            f"manifest pins {spec['register_version']} — refusing to validate."
        )
    active = frozenset(m["id"] for m in payload["members"] if m.get("status") == "valid")
    deprecated = frozenset(m["id"] for m in payload["members"] if m.get("status") != "valid")
    return active, deprecated


def check_register_field(df: pd.DataFrame, spec: dict) -> list[Rejection]:
    active, deprecated = load_register(spec)
    field = spec["name"]
    rejections: list[Rejection] = []

    for feature_id, value in zip(df["feature_id"], df[field]):
        if value is None or (isinstance(value, str) and value in SENTINEL_NULLS):
            if spec["required"]:
                rejections.append(Rejection(feature_id, field, "ATTR_MANDATORY_NULL",
                                            repr(value), "non-null"))
            continue
        if value in active:
            continue
        if value in deprecated:
            code = "ATTR_DEPRECATED_CODE" if not spec.get("allow_deprecated") else ""
            if code:
                rejections.append(Rejection(feature_id, field, code, str(value),
                                            f"active member of {spec['register_uri']}"))
            continue
        rejections.append(Rejection(feature_id, field, "ATTR_UNDEFINED_CODE", str(value),
                                    f"member of {spec['register_uri']}@{spec['register_version']}"))

    logger.info("register field %s: %d rejection(s) over %d row(s)",
                field, len(rejections), len(df))
    return rejections

The version assertion in load_register is the guard that matters most. Without it, a register sync job that failed last night leaves yesterday’s cache in place and the run validates against a stale vocabulary while reporting success. With it, the mismatch is a hard failure with both versions named, which turns a silent conformance regression into a five-second diagnosis.

The other guard is that rejections carry the observed value. A quality report that says “37 features failed domain validation on land_use” is a work order with no information; one that says “37 features carried residental” identifies a single upstream typo and makes the fix a one-line change at the source rather than an investigation.

Failure Modes & Fallback Routing

Failure Typical cause Deterministic action
ATTR_MANDATORY_NULL A source that exports empty strings for unpopulated mandatory fields Quarantine the feature; never substitute a default, which fabricates a fact
ATTR_UNDEFINED_CODE Upstream typo, a local extension of a governed vocabulary, or a register amended since the pin Quarantine with the observed value recorded; a cluster of identical values indicates the third cause
ATTR_DEPRECATED_CODE The register retired a member and the source has not migrated Quarantine unless allow_deprecated is set for a stated migration window
ATTR_OUT_OF_RANGE Unit drift — feet in a metre column — or a sentinel that escaped null normalization Quarantine; check the distribution before widening the range, since a 3.28 ratio is a unit, not an outlier
ATTR_PATTERN_MISMATCH An identifier format change at the source, or a leading zero lost to a spreadsheet Quarantine; identifier formats must never be “repaired” because the repair guesses
ATTR_NOT_UNIQUE A join that fanned out upstream, or genuine duplicate records Quarantine all members of the duplicated key and hand them to duplicate detection
ATTR_CONDITIONAL_MISSING A conditional obligation the source does not model Quarantine; the condition is part of the target schema’s contract
REGISTER_CACHE_STALE The register sync job failed and left an old pin in place Fail the run before validating anything, naming both versions

There is one deliberate omission from that table: no row produces a corrected value. Attribute domain validation is the gate that must never repair. A land-use code mapped from residental to residential is a guess that will be right most of the time and catastrophically wrong when the source actually meant something else, and once it is applied there is no evidence left that a guess was made. Where a source genuinely uses a local vocabulary that maps onto a governed one, that mapping is a declared crosswalk in the schema mapping stage, reviewed and version-controlled, not an inference made in a validator.

Compliance Reporting Output

python
# quality/domain_report.py — pyarrow >=14 — Python 3.10+
import pyarrow as pa

DOMAIN_REPORT_SCHEMA = pa.schema([
    ("dataset_id",        pa.string()),
    ("run_id",            pa.string()),
    ("manifest_version",  pa.string()),
    ("field",             pa.string()),
    ("kind",              pa.string()),      # enumeration | range | register | identifier
    ("register_uri",      pa.string()),      # "" for local domains
    ("register_version",  pa.string()),      # the pin actually validated against
    ("population",        pa.int64()),       # non-null values evaluated
    ("nulls",             pa.int64()),       # separated from failures on purpose
    ("rejections",        pa.int64()),
    ("distinct_bad",      pa.int64()),       # how many distinct offending values
    ("top_bad_value",     pa.string()),      # the most frequent one, for triage
    ("result",            pa.string()),      # pass | fail | not_evaluated
])

distinct_bad and top_bad_value exist because they change what a steward does next. Ten thousand rejections across one distinct value is a source-system configuration change and takes minutes to fix; ten thousand rejections across four thousand distinct values is a vocabulary mismatch and takes a crosswalk. Reporting only the count makes those two indistinguishable, and teams reliably guess wrong. The rows feed the CI conformance scorecard and, through the lineage manifest, the ISO 19157 domain-consistency element of the published dataset metadata.

CI Integration

The framework-level enforcement is best expressed with pandera, whose schema objects can be generated from the same manifest the runtime engine reads — one source of truth, two consumers.

python
# tests/test_domains.py — pytest >=7, pandera >=0.18, pandas >=2.1
import pandas as pd
import pytest

from quality.domains import check_register_field, load_register
from quality.manifest import load_domain_manifest

MANIFEST = load_domain_manifest("attribute_domains.yaml")
LAND_USE = next(f for f in MANIFEST["fields"] if f["name"] == "land_use")


def test_known_member_passes():
    df = pd.DataFrame({"feature_id": ["a"], "land_use": ["1_1_1"]})
    assert check_register_field(df, LAND_USE) == []


def test_unknown_member_is_rejected_with_the_observed_value():
    df = pd.DataFrame({"feature_id": ["b"], "land_use": ["residental"]})
    (rejection,) = check_register_field(df, LAND_USE)
    assert rejection.code == "ATTR_UNDEFINED_CODE"
    assert rejection.observed == "residental"     # the typo must reach the report


def test_stale_cache_fails_the_run_not_the_feature():
    # Negative control: a cache whose version does not match the pin must refuse
    # to validate at all rather than quietly validating against the wrong list.
    stale = dict(LAND_USE, register_version="2019-01-01")
    with pytest.raises(ValueError, match="refusing to validate"):
        load_register(stale)

Run this suite in the same workflow as the pull-request schema-drift gate, and add a scheduled job that re-runs only test_stale_cache_fails_the_run_not_the_feature against the live register: it is the cheapest possible detector for “the authority published a new version and nobody noticed”.

Deeper Implementation Walkthroughs

Validating code list values against INSPIRE registers implements the register sync job, the cache format, and the version-pin assertion end to end, including how to handle a register that publishes hierarchical values. Enforcing attribute domains with pandera schemas generates a typed schema from the manifest so the same rules run inside the ETL job, inside the test suite, and as a pre-commit hook on fixture data.

Frequently Asked Questions

Why is an empty string not treated as null? Because the two mean different things and only one of them is a defect the source can fix. A null says “this was not recorded”; an empty string usually says “a system wrote a value of zero length”, which is almost always a bug in an export. Collapsing them makes a mandatory-field rule pass on data that carries no information, and it is the single most common way a conformance report overstates completeness. The null_policy setting makes the choice explicit so at least it is a decision.

What if a source legitimately extends a governed code list? Then the extension is part of your target schema and belongs in the manifest as a documented local enumeration layered over the register, with its own identifier space so that a consumer can tell governed values from local ones. What must not happen is quietly adding the local value to the register’s member set in the cache file — that turns your copy of a public vocabulary into a fork nobody knows exists.

Should range checks reject or clamp? Reject. Clamping a 984-metre building height to the 300-metre maximum produces a value that is both wrong and plausible, and it destroys the evidence that would have identified the real cause, which is nearly always feet arriving in a metre column. The correct handling of unit drift is upstream, in the unit conversion and tolerance stage, and a cluster of out-of-range values with a consistent ratio is the strongest signal that stage is misconfigured.

How often should pinned registers be refreshed? On a schedule that matches the authority’s amendment cadence, reviewed like a dependency bump: the sync job opens a pull request with the new version and the diff of members added, deprecated and removed. That diff is the useful artefact — it tells you before you upgrade how many of your existing features would start failing, which is precisely the information a “latest” fetch destroys.