Geospatial Schema Standardization & Mapping Automation

Handling Missing NTv2 Grid Files in Production

A datum transformation that ran flawlessly in CI can silently degrade the moment it deploys to a worker whose PROJ data package differs from the developer’s. NTv2 grid shift files — us_noaa_nadcon5*.tif, ca_nrc_ntv2_0.tif, and their national equivalents — are not bundled with the pyproj wheel; when they are absent, PROJ does not raise, it substitutes a coarser path and returns coordinates that are 1–2 metres off. This procedure hardens the missing-grid case as a detectable, alerted, non-silent event. It implements the recovery contract defined by Datum Transformation Fallback Chains, the parent stage inside CRS Normalization & Sync that owns which path runs when a direct, grid-based shift is unavailable.

The steps map to the phases this site standardizes on — configure (Step 1, startup probe), execute (Steps 2-3, path inspection and controlled fallback), validate (Step 4, grid pinning), and log (Step 5, provenance). The goal is not to avoid the coarse path — sometimes it is the only path — but to guarantee that using it is a logged, alerted decision rather than an invisible accident.

Prerequisites checklist

Step 1: Probe grid availability at startup

Missing grids must fail fast and loud at process start, never per-feature deep inside a batch. Enumerate every grid your manifest depends on and confirm it resolves in a PROJ search path before any geometry is touched. Run this once during bootstrap.

python
# pyproj >= 3.6 — Python 3.10+
import logging
from pathlib import Path

import pyproj
from pyproj.datadir import get_data_dir

logger = logging.getLogger("crs.grids")

# Grids the transformation manifest depends on. Keep this list in version control.
REQUIRED_GRIDS: tuple[str, ...] = (
    "us_noaa_nadcon5_nad27_nad83_1986_conus.tif",  # NAD27 -> NAD83 CONUS
    "ca_nrc_ntv2_0.tif",                            # NAD27 -> NAD83 Canada
)


def probe_grids(required: tuple[str, ...] = REQUIRED_GRIDS) -> dict[str, bool]:
    """Return {grid_name: is_present}. Never transform before this is clean."""
    search_dirs = [Path(get_data_dir())]
    availability: dict[str, bool] = {}
    for grid in required:
        found = any((d / grid).is_file() for d in search_dirs)
        availability[grid] = found
        level = logging.INFO if found else logging.ERROR
        logger.log(level, "grid %s present=%s (PROJ_DATA=%s)", grid, found, get_data_dir())
    return availability

Startup rules:

  • Treat a missing grid as a deployment-blocking condition in strict environments; downgrade to a warning only where the coarse path is explicitly signed off.
  • Probe against pyproj.datadir.get_data_dir(), not a hardcoded path — it reflects PROJ_DATA and the wheel default in the true resolution order.
  • Never call pyproj sync at runtime to self-heal; on-demand fetching makes the result depend on network state and breaks reproducibility.

Step 2: Inspect unavailable_operations

TransformerGroup is the authoritative source of truth for which path is missing and why. Its available_operations list holds paths PROJ can execute now; unavailable_operations holds paths that exist in the database but need a grid that is not installed, each exposing grid_name and a url to fetch it. This is the same inspection surface the parent Datum Transformation Fallback Chains routing engine consumes.

python
# pyproj >= 3.6 — Python 3.10+
from dataclasses import dataclass

from pyproj import CRS
from pyproj.transformer import TransformerGroup


@dataclass(frozen=True)
class PathReport:
    have_grid_path: bool          # a grid-based operation is installed and usable
    best_available_accuracy_m: float | None
    missing_grids: tuple[str, ...]  # grids that would unlock a more accurate path


def report_paths(src_epsg: int, dst_epsg: int) -> PathReport:
    group = TransformerGroup(CRS.from_epsg(src_epsg), CRS.from_epsg(dst_epsg), always_xy=True)

    missing: list[str] = []
    for op in group.unavailable_operations:
        # Each grid dependency PROJ knows about but cannot resolve locally.
        missing.extend(g.short_name for g in op.grids if not g.available)

    best = group.transformers[0] if group.transformers else None
    acc = getattr(best, "accuracy", -1.0) if best else -1.0
    # A grid-based available op is exact; if the best available op is coarse, we degraded.
    have_grid = bool(group.transformers) and group.transformers[0].is_exact

    return PathReport(
        have_grid_path=have_grid,
        best_available_accuracy_m=None if acc < 0 else acc,
        missing_grids=tuple(dict.fromkeys(missing)),  # dedupe, keep order
    )

Inspection rules:

  • A non-empty unavailable_operations list is the definitive signal that a more accurate path exists but is grid-deficient — not that no path exists.
  • Read op.grids[*].available per operation; a single operation can depend on several tiled grids where only some are present.
  • Cache one PathReport per distinct source/target pair. Constructing TransformerGroup per feature dominates runtime, exactly as covered in Multi-CRS Dataset Harmonization.

Step 3: Degrade to a controlled fallback with alerting

When the grid path is missing, the pipeline must make a decision, not drift. Either reject the batch (accuracy-critical) or accept the coarse path with a degraded-accuracy flag and an alert. The numeric threshold that separates “acceptable coarse” from “reject” is owned by Unit Conversion & Tolerance Thresholds; this step only routes on that decision.

python
# pyproj >= 3.6, geopandas >= 0.14 — Python 3.10+
import geopandas as gpd
from pyproj import CRS
from pyproj.transformer import TransformerGroup


class GridMissingError(RuntimeError):
    """Raised when an accuracy-critical transform lacks its NTv2 grid."""


def transform_with_fallback(
    gdf: gpd.GeoDataFrame, dst_epsg: int, tolerance_m: float, strict: bool
) -> gpd.GeoDataFrame:
    src = CRS.from_user_input(gdf.crs)
    group = TransformerGroup(src, CRS.from_epsg(dst_epsg), always_xy=True)
    if not group.transformers:
        raise GridMissingError(f"no path {gdf.crs} -> EPSG:{dst_epsg}")

    best = group.transformers[0]
    accuracy = getattr(best, "accuracy", -1.0)
    grid_deficient = bool(group.unavailable_operations)

    if grid_deficient and (accuracy < 0 or accuracy > tolerance_m):
        msg = f"grid missing: best accuracy={accuracy} m exceeds tolerance {tolerance_m} m"
        if strict:
            raise GridMissingError(msg)
        # Non-strict: emit a pageable alert, then proceed on the coarse path.
        logger.error("GRID_MISSING alert: %s (src=%s dst=EPSG:%d)", msg, gdf.crs, dst_epsg)

    out = gdf.to_crs(epsg=dst_epsg)
    out.attrs["degraded_accuracy"] = grid_deficient
    out.attrs["operation_name"] = best.description
    return out

Fallback rules:

  • In strict mode, a missing accuracy-critical grid raises and routes the asset to the reconciliation queue — never silently ships a 2 m error.
  • Emit GRID_MISSING as a structured, pageable alert, not a debug line; a grid regression on one worker is an incident, not noise.
  • Stamp degraded_accuracy and the selected operation_name onto the output so downstream consumers can judge fitness for use.

Step 4: Pin and install grids with projsync

The permanent fix is to stop depending on ambient PROJ data. Fetch the exact grids at build time into a directory you bake into the image, and pin them by name so a proj-data release cannot shift results underneath you.

bash
# Build-time only — pyproj >= 3.6
export PROJ_DATA=/opt/proj-grids
mkdir -p "$PROJ_DATA"

# Fetch only the grids the manifest declares, into the pinned dir.
pyproj sync --target-dir "$PROJ_DATA" --source-id us_noaa --list-files
pyproj sync --target-dir "$PROJ_DATA" --source-id us_noaa
pyproj sync --target-dir "$PROJ_DATA" --source-id ca_nrc

# Freeze an inventory so drift is detectable in CI.
( cd "$PROJ_DATA" && sha256sum *.tif ) > grids.lock

Bake PROJ_DATA=/opt/proj-grids and PROJ_NETWORK=OFF into the runtime environment so the container never reaches the CDN in production. Commit grids.lock and diff it in CI; a changed hash is a deliberate, reviewed grid update, not a surprise.

Step 5: Log grid provenance to the audit trail

Every transform records which grid backed it. This row is what an auditor reads to confirm a NAD27→NAD83 shift used the published NADCON5 grid rather than a Helmert approximation.

python
# pyarrow >= 14 — Python 3.10+
import datetime as dt

import pyarrow as pa
import pyarrow.parquet as pq


def append_grid_lineage(path: str, src: str, dst: str, report: "PathReport") -> None:
    row = {
        "src_crs": src,
        "dst_crs": dst,
        "grid_path_used": report.have_grid_path,
        "best_accuracy_m": report.best_available_accuracy_m,
        "missing_grids": ",".join(report.missing_grids) or None,
        "timestamp_utc": dt.datetime.now(dt.UTC).isoformat(),
    }
    table = pa.Table.from_pylist([row])
    pq.write_to_dataset(table, root_path=path)  # append-only lineage partition

Persist these rows to the append-only lineage store consumed by Geospatial Compliance Reporting & Audit Trails; grid_path_used=false rows are the exact set a data-quality review must inspect.

Verification

Confirm the guard actually catches an absent grid rather than trusting a green build:

python
# Point PROJ at an empty dir to simulate a stripped production worker.
import os, tempfile, pyproj

os.environ["PROJ_DATA"] = tempfile.mkdtemp()
pyproj.datadir.set_data_dir(os.environ["PROJ_DATA"])

report = report_paths(4267, 4326)  # NAD27 -> WGS84
assert report.have_grid_path is False, "expected grid-deficient with empty PROJ_DATA"
assert report.missing_grids, "unavailable_operations should name the missing NADCON5 grid"

On the CLI, projinfo -s EPSG:4267 -t EPSG:4326 --grid-check known_available prints Grid ... available: false for a stripped worker; after Step 4 the same command reports available: true, and pyproj sync --list-files --source-id us_noaa echoes the pinned filenames.

Troubleshooting

Symptom Likely cause Fix
Output shifted 1–2 m vs a reference layer NADCON5/NTv2 grid absent; PROJ substituted a coarse geocentric path Run the Step 1 probe; if have_grid_path is false, pyproj sync the grid (Step 4) and re-run.
unavailable_operations non-empty but transform still “works” A more accurate grid path exists yet is grid-deficient; PROJ used the next path Treat as degraded — assert on unavailable_operations in Step 2 and alert per Step 3.
Results differ between CI and production Divergent proj-data package versions across images Pin PROJ_DATA and commit grids.lock; diff the hashes in CI (Step 4).
Grid fetch hangs or 404s at runtime PROJ_NETWORK=ON reaching the CDN mid-batch Set PROJ_NETWORK=OFF in production; fetch only at build time.
accuracy == -1 on the selected path PROJ cannot estimate error for the fallback operation In strict mode, reject and reconcile; otherwise flag degraded_accuracy=true and page.