Datum Transformation Fallback Chains Jump to heading

Geospatial ETL pipelines routinely encounter legacy coordinate definitions, missing transformation grids, or deprecated EPSG codes during ingestion. When a direct transformation to a target datum fails, the pipeline must not terminate — it must route deterministically to the next viable path. That routing mechanism is a datum transformation fallback chain. This pipeline stage operates at the core of CRS Normalization & Sync, the discipline concerned with guaranteeing that every geometry in the system shares a coherent spatial reference before downstream schema operations begin.

This page covers what fallback chains are responsible for and where they stop. Selecting and validating EPSG codes for mixed-datum inputs is handled by Projection Normalization Workflows. Applying numeric thresholds once a path is selected belongs to Unit Conversion & Tolerance Thresholds. Combining outputs from multiple authority CRS sources into a single harmonized dataset is covered by Multi-CRS Dataset Harmonization. This page is narrowly scoped to the routing contract itself: how to declare, execute, and audit which transformation path ran and why.

Declarative Configuration Manifest Jump to heading

Resilient transformation routing begins with config-as-code. Every fallback chain is declared in a version-controlled YAML manifest that specifies transformation priorities, grid dependencies, and tolerance thresholds. Decoupling the routing policy from the execution code means that schema mapping automation can reference transformation policies directly without modifying Python source.

# transformation_chains.yaml  — pyproj >=3.6, pyyaml >=6.0
chains:
  - chain_id: "legacy_nad27_to_wgs84"       # MANDATORY: unique pipeline identifier
    source_crs: "EPSG:4267"                  # MANDATORY: EPSG, URI, or WKT2 string
    target_crs: "EPSG:4326"                  # MANDATORY: EPSG, URI, or WKT2 string
    tolerance_meters: 1.5                    # MANDATORY: max allowable positional deviation (float > 0)
    strict_mode: true                        # MANDATORY: if true, reject when accuracy unknown (-1)
    fallback_order:                          # MANDATORY: ordered list, min 1 entry
      - strategy: "grid_based_ntv2"
        priority: 1
      - strategy: "helmert_approximation"
        priority: 2
      - strategy: "geocentric_translation"
        priority: 3
    # --- OPTIONAL FIELDS ---
    description: "Fallback for legacy municipal survey data lacking NADCON grids."
    grid_override_path: "/opt/grids/custom_nad27_wgs84.gtx"
    max_attempts: 3                          # default: len(fallback_order)
    audit_tag: "municipal_survey_2023"       # written verbatim to lineage record

  - chain_id: "state_plane_to_wgs84"
    source_crs: "EPSG:2263"                  # NY State Plane, feet
    target_crs: "EPSG:4326"
    tolerance_meters: 0.05
    strict_mode: true
    fallback_order:
      - strategy: "grid_based_ntv2"
        priority: 1
      - strategy: "helmert_approximation"
        priority: 2
    description: "High-accuracy transform for NYC parcel boundary data."

Mandatory vs optional field reference:

Field	Required	Type	Notes
`chain_id`	Yes	string	Unique across the manifest; used as the audit lineage key
`source_crs`	Yes	string	Validated against PROJ database at startup
`target_crs`	Yes	string	Validated against PROJ database at startup
`tolerance_meters`	Yes	float > 0	Applies to horizontal deviation; add `tolerance_vertical_meters` for LiDAR/elevation
`strict_mode`	Yes	bool	When `true`, reject any path where `accuracy == -1` (unknown)
`fallback_order`	Yes	list, min 1	Ordered by priority integer ascending; first match wins
`description`	No	string	Written to audit trail; aids incident triage
`grid_override_path`	No	string	Absolute path to `.tif` / `.gtx` grid; checked for existence at startup
`max_attempts`	No	int	Caps routing retries; defaults to `len(fallback_order)`
`audit_tag`	No	string	Freeform tag written to every lineage record produced by this chain

Reject manifests at parse time that reference non-existent grid files, omit mandatory fields, define circular chain references, or declare tolerance_meters <= 0. Enforce this with Pydantic validation before any geometry touches the routing engine.

The chain_id is the binding key between this manifest and every audit record the stage later emits, so renaming a chain is a breaking change — treat the manifest as part of the schema contract, not as tunable configuration.

Preprocessing Requirements Jump to heading

Before the fallback routing engine executes, input data must satisfy three preconditions:

CRS definition is resolvable. Every source dataset must carry a readable CRS — a .prj sidecar for shapefiles, a crs attribute for GeoDataFrames, or an embedded crs field in GeoJSON. Records with None or empty CRS must be quarantined to the rejection queue before routing begins. The Projection Normalization Workflows stage, which runs upstream of this one, is responsible for resolving ambiguous or malformed CRS strings to canonical EPSG codes.
Geometry is valid. Routing a self-intersecting or empty geometry through a datum shift propagates corruption silently. Run shapely.validation.make_valid() on every feature before it enters the chain. Flag geometries that remain invalid after repair and route them to the rejection queue with a GEOMETRY_INVALID status code.
PROJ data grids are present. Grid-based transformations (NTv2, NADCON, VERTCON) require the proj-data package or equivalent grid files at the path PROJ expects ($PROJ_DATA). Verify this at pipeline startup — not per-feature — to avoid expensive per-feature failures:

import subprocess, sys

def assert_proj_grids_available(required_grids: list[str]) -> None:
    """Fail fast at startup if required PROJ grid files are missing."""
    result = subprocess.run(
        ["projinfo", "--searchpaths"], capture_output=True, text=True
    )
    search_paths = result.stdout.strip().split("\n")
    for grid in required_grids:
        found = any(
            (Path(p) / grid).exists() for p in search_paths if p
        )
        if not found:
            sys.exit(
                f"FATAL: Required PROJ grid '{grid}' not found in search paths: "
                f"{search_paths}. Install proj-data or set PROJ_DATA."
            )

Execution Engine and Precision Guards Jump to heading

pyproj.TransformerGroup (pyproj >=3.6) enumerates every available transformation path between a source and target CRS, ordered by PROJ’s internal preference — exact/grid-based first, parameterized/approximate last. The routing engine iterates this list, evaluates each path against the chain’s tolerance_meters, and returns the first path that passes. The accuracy attribute reports the estimated horizontal error in metres; -1 means PROJ could not determine accuracy for that path.

# pipeline/datum_routing.py  — pyproj >=3.6
from __future__ import annotations
import logging
from dataclasses import dataclass
from pyproj import CRS, TransformerGroup, Transformer
from pyproj.exceptions import CRSError, ProjError

logger = logging.getLogger(__name__)


@dataclass
class RoutingResult:
    transformer: Transformer
    strategy_index: int       # 0-based position in TransformerGroup.transformers
    is_exact: bool
    accuracy_m: float | None  # None when PROJ reports -1
    chain_id: str


def resolve_transformation(
    chain_id: str,
    src_crs: str,
    dst_crs: str,
    tolerance_m: float,
    strict_mode: bool = False,
) -> RoutingResult:
    """
    Enumerate transformation paths via TransformerGroup and return the first
    path within tolerance.  Raises ProjError if no acceptable path exists.

    Args:
        chain_id:     Manifest chain identifier, written to audit records.
        src_crs:      Source CRS as EPSG string, URI, or WKT2.
        dst_crs:      Target CRS as EPSG string, URI, or WKT2.
        tolerance_m:  Maximum allowable horizontal error in metres.
        strict_mode:  If True, reject paths where accuracy is unknown (-1).
    """
    try:
        src = CRS(src_crs)
        dst = CRS(dst_crs)
    except CRSError as exc:
        logger.critical("[%s] Invalid CRS definition — %s", chain_id, exc)
        raise

    try:
        group = TransformerGroup(src, dst, always_xy=True)
    except ProjError as exc:
        logger.critical("[%s] TransformerGroup construction failed — %s", chain_id, exc)
        raise

    if not group.transformers:
        raise ProjError(
            f"[{chain_id}] No transformation path found for {src_crs} → {dst_crs}. "
            "Verify that PROJ data grids are installed (proj-data package)."
        )

    for idx, t in enumerate(group.transformers):
        raw_accuracy: float = getattr(t, "accuracy", -1.0)
        accuracy_m: float | None = None if raw_accuracy < 0 else raw_accuracy

        if t.is_exact:
            logger.info("[%s] Exact path selected (index %d).", chain_id, idx)
            return RoutingResult(t, idx, True, accuracy_m, chain_id)

        if accuracy_m is None:
            if strict_mode:
                logger.warning(
                    "[%s] Skipping path (index %d): accuracy unknown and strict_mode=True.",
                    chain_id, idx,
                )
                continue
            # Non-strict: accept unknown accuracy with a warning
            logger.warning(
                "[%s] Accepting path (index %d) with unknown accuracy (strict_mode=False).",
                chain_id, idx,
            )
            return RoutingResult(t, idx, False, None, chain_id)

        if accuracy_m <= tolerance_m:
            logger.info(
                "[%s] Approximate path selected (index %d, accuracy=%.4f m).",
                chain_id, idx, accuracy_m,
            )
            return RoutingResult(t, idx, False, accuracy_m, chain_id)

        logger.debug(
            "[%s] Path (index %d) rejected: accuracy %.4f m exceeds tolerance %.4f m.",
            chain_id, idx, accuracy_m, tolerance_m,
        )

    raise ProjError(
        f"[{chain_id}] No transformation path within {tolerance_m} m tolerance for "
        f"{src_crs} → {dst_crs}. Check PROJ grid availability or raise tolerance_meters."
    )

Important axis-order note: always_xy=True is mandatory when constructing TransformerGroup objects for pipeline use. Omit it only when you explicitly require authority-mandated axis order (latitude-first for EPSG:4326 in strict ISO 19111 compliance contexts). Silent axis swaps are a leading source of coordinate corruption in production.

Failure Modes and Fallback Routing Jump to heading

No silent failures are acceptable. Every path through the routing engine must produce an explicit audit record, whether or not a transformation succeeded.

Failure type	Likely cause	Deterministic recovery action
`CRSError` on group construction	Malformed or unrecognised EPSG/WKT2 string	Quarantine feature to `INVALID_CRS` rejection queue; log `src_crs` and `dst_crs` verbatim
`group.transformers` is empty	PROJ database lacks any registered path	Quarantine to `NO_PATH` rejection queue; alert if count exceeds 1% of batch
All paths exceed `tolerance_meters`	Missing NTv2/NADCON grids; only coarse Helmert paths available	Quarantine to `TOLERANCE_EXCEEDED` rejection queue; log best available accuracy
`accuracy == -1` and `strict_mode=True`	Path exists but PROJ cannot estimate its error	Skip path and continue; quarantine if no other path qualifies
Grid file path invalid at startup	`grid_override_path` in manifest points to missing file	`SystemExit(1)` — do not start the pipeline
Geometry invalid before transform	Self-intersection, null coordinate, or empty geometry	Run `make_valid()`; if still invalid, quarantine to `GEOMETRY_INVALID`
Vertical datum shift fails	VERTCON/GEOID grids absent; elevation LiDAR workflow	Quarantine to `VERTICAL_TRANSFORM_FAILED`; treat horizontal and vertical passes as separate gates

For large batch jobs, track the per-chain fallback activation rate. When the TOLERANCE_EXCEEDED queue grows beyond 5% of total features for a given chain, trigger an automated alert — this pattern signals that the PROJ grid package installed on the worker does not match the one used during chain configuration.

If you are running Batch Schema Processing Pipelines over tens of thousands of shapefiles, instantiate a single TransformerGroup per unique source-target CRS pair rather than per feature. Group features by CRS before the routing loop to avoid repeated PROJ database lookups.

Compliance Reporting Output Jump to heading

Every feature that passes through the routing engine — whether it succeeds or is quarantined — must produce a structured lineage record. Write this record to your audit trail before yielding the transformed geometry downstream.

# pipeline/audit.py  — pyarrow >=14
import json
import datetime
from pathlib import Path


LINEAGE_SCHEMA = {
    "chain_id": str,
    "feature_id": str,
    "src_crs": str,
    "dst_crs": str,
    "strategy_index": int,
    "is_exact": bool,
    "accuracy_m": "float | null",
    "tolerance_m": float,
    "outcome": str,           # PASS | INVALID_CRS | NO_PATH | TOLERANCE_EXCEEDED | ...
    "timestamp_utc": str,     # ISO 8601
    "audit_tag": "str | null",
}


def write_lineage_record(
    audit_path: Path,
    chain_id: str,
    feature_id: str,
    src_crs: str,
    dst_crs: str,
    result: "RoutingResult | None",
    outcome: str,
    tolerance_m: float,
    audit_tag: str | None = None,
) -> None:
    """Append one JSON lineage record to the audit NDJSON file."""
    record = {
        "chain_id": chain_id,
        "feature_id": feature_id,
        "src_crs": src_crs,
        "dst_crs": dst_crs,
        "strategy_index": result.strategy_index if result else -1,
        "is_exact": result.is_exact if result else False,
        "accuracy_m": result.accuracy_m if result else None,
        "tolerance_m": tolerance_m,
        "outcome": outcome,
        "timestamp_utc": datetime.datetime.utcnow().isoformat() + "Z",
        "audit_tag": audit_tag,
    }
    with audit_path.open("a") as fh:
        fh.write(json.dumps(record) + "\n")

The audit file is newline-delimited JSON (NDJSON) so that downstream tools — including jq, duckdb, and pyarrow.json — can stream it without loading the entire file into memory. Field names map directly to the lineage schema required by Field Renaming & Type Coercion Rules when the CRS stage feeds into attribute transformation. Set retention to a minimum of 90 days for datasets subject to FGDC or INSPIRE audit requirements.

Rejection log format: quarantined features are written to a companion NDJSON file with the same schema plus a rejection_reason field containing the full exception message. This separation keeps the primary audit trail free of error noise while retaining the full diagnostics for incident review.

CI Integration Jump to heading

Automated testing must confirm that every chain in the manifest resolves without raising ProjError under simulated grid-missing conditions. Use pytest fixtures that temporarily suppress PROJ grid search paths to exercise all fallback tiers.

# tests/test_fallback_chains.py  — pytest >=7, pyproj >=3.6
import os
import pytest
from pyproj.exceptions import ProjError
from pipeline.datum_routing import resolve_transformation


CHAINS = [
    ("legacy_nad27_to_wgs84", "EPSG:4267", "EPSG:4326", 1.5, True),
    ("state_plane_to_wgs84",  "EPSG:2263", "EPSG:4326", 0.05, True),
]


@pytest.mark.parametrize("chain_id,src,dst,tol,strict", CHAINS)
def test_chain_resolves_with_grids(chain_id, src, dst, tol, strict):
    """Full grid environment: must return a RoutingResult without raising."""
    result = resolve_transformation(chain_id, src, dst, tol, strict)
    assert result.transformer is not None
    assert result.chain_id == chain_id


@pytest.mark.parametrize("chain_id,src,dst,tol,strict", CHAINS)
def test_chain_fallback_without_exact_grids(chain_id, src, dst, tol, strict, tmp_path, monkeypatch):
    """
    Simulate missing NTv2/NADCON grids by pointing PROJ_DATA at an empty dir.
    The chain must either fall back gracefully or raise ProjError — never silently
    return a corrupt transformer.
    """
    monkeypatch.setenv("PROJ_DATA", str(tmp_path))
    try:
        result = resolve_transformation(chain_id, src, dst, tol, strict)
        # If it resolves, it must not be an exact path (grids are gone)
        assert not result.is_exact, (
            f"Expected non-exact fallback when grids are absent, got is_exact=True "
            f"for chain '{chain_id}'."
        )
    except ProjError:
        pass  # Acceptable: no fallback within tolerance


def test_invalid_crs_raises_immediately():
    from pyproj.exceptions import CRSError
    with pytest.raises(CRSError):
        resolve_transformation("bad_chain", "EPSG:9999999", "EPSG:4326", 1.0, False)

Gate this suite in CI before any merge that touches PROJ grid configuration, the chain manifest, or the routing engine. The GitHub Actions step below runs in under 30 seconds on a standard ubuntu-latest runner:

# .github/workflows/validate-chains.yml
name: Validate Transformation Chains
on: [push, pull_request]

jobs:
  test-chains:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install "pyproj>=3.6" "pydantic>=2.0" "pyyaml>=6.0" "pytest>=7" proj-data
      - name: Validate chain manifest schema
        run: python -m pipeline.validate_manifest transformation_chains.yaml
      - name: Run routing tests
        run: pytest tests/test_fallback_chains.py -v --tb=short

Alert on fallback activation rates using a post-run metrics step: if outcome != "PASS" rows in the audit NDJSON exceed 5% of total records, fail the CI run and page the on-call GIS engineer. The Error Handling & Retry Logic patterns — including exponential backoff for transient grid-download failures — integrate here as a wrapper around the routing engine rather than inside it.

Frequently Asked Questions Jump to heading

When should a chain prefer a Helmert approximation over a grid-based path? Never by configuration — prefer grids whenever they resolve within tolerance_meters. A Helmert (7-parameter) approximation only becomes the selected path when the grid-based NTv2/NADCON tier is unavailable or returns an accuracy worse than the chain tolerance. Because the routing engine iterates TransformerGroup.transformers in PROJ’s preference order (exact paths first), Helmert is reached only as a deterministic fallback, and the lineage record’s strategy_index proves which tier actually ran.

What is the difference between strict_mode and just lowering tolerance_meters? tolerance_meters gates paths whose accuracy PROJ can estimate. strict_mode gates paths whose accuracy is reported as -1 (unknown). A path with unknown accuracy can never be compared against a numeric tolerance, so the two controls are orthogonal: set a tight tolerance_meters for known-accuracy gating and strict_mode: true to forbid the pipeline from silently accepting paths PROJ cannot characterise.

Why instantiate one TransformerGroup per CRS pair instead of per feature? TransformerGroup construction triggers PROJ database lookups and grid-availability probing, which dominate runtime when repeated per feature. Grouping features by their resolved source-target CRS pair before the routing loop — the pattern used in Batch Schema Processing Pipelines — turns thousands of lookups into one per distinct pair and keeps memory flat across large shapefile batches.

How do vertical datum shifts fit into a horizontal fallback chain? They do not share a tolerance. A horizontal NTv2 path and a vertical VERTCON/GEOID path are independent gates; a feature can pass the horizontal transform and still be quarantined to VERTICAL_TRANSFORM_FAILED if the GEOID grid is absent. Declare a separate tolerance_vertical_meters and evaluate the two passes as distinct audit outcomes rather than collapsing them into one PASS/FAIL.

Deeper implementation walkthroughs Jump to heading

Step-by-step builds for the hardest parts of this stage live in the focused implementation guides under this topic. For resolving the EPSG inputs that feed a chain, work through step-by-step EPSG code normalization for mixed datasets. For the numeric gate that a selected path must clear, see setting tolerance thresholds for automated coordinate snapping. To wrap the routing engine in transient-failure recovery for unreliable grid downloads, follow implementing exponential backoff in schema mapping jobs.

Projection Normalization Workflows — resolving ambiguous EPSG codes before fallback chains receive them
Unit Conversion & Tolerance Thresholds — standardized deviation matrices for horizontal and vertical accuracy gates
Multi-CRS Dataset Harmonization — combining outputs from multiple CRS authority sources into a single dataset
Error Handling & Retry Logic — exponential backoff and circuit-breaker patterns for transient ETL failures
Batch Schema Processing Pipelines — processing large shapefile batches where per-chain CRS grouping matters

Datum Transformation Fallback Chains Jump to heading#

Declarative Configuration Manifest Jump to heading#

Preprocessing Requirements Jump to heading#

Execution Engine and Precision Guards Jump to heading#

Failure Modes and Fallback Routing Jump to heading#

Compliance Reporting Output Jump to heading#

CI Integration Jump to heading#

Frequently Asked Questions Jump to heading#

Deeper implementation walkthroughs Jump to heading#

Related Jump to heading#