Geospatial Schema Standardization & Mapping Automation

Handling Missing NTv2 Grid Files in Production

A datum transformation that ran flawlessly in CI can silently degrade the moment it deploys to a worker whose PROJ data package differs from the developer’s. NTv2 grid shift files — us_noaa_nadcon5*.tif, ca_nrc_ntv2_0.tif, and their national equivalents — are not bundled with the pyproj wheel; when they are absent, PROJ does not raise, it substitutes a coarser path and returns coordinates that are 1–2 metres off. This procedure hardens the missing-grid case as a detectable, alerted, non-silent event. It implements the recovery contract defined by Datum Transformation Fallback Chains, the parent stage inside CRS Normalization & Sync that owns which path runs when a direct, grid-based shift is unavailable.

The steps map to the phases this site standardizes on — configure (Step 1, startup probe), execute (Steps 2-3, path inspection and controlled fallback), validate (Step 4, grid pinning), and log (Step 5, provenance). The goal is not to avoid the coarse path — sometimes it is the only path — but to guarantee that using it is a logged, alerted decision rather than an invisible accident.

Prerequisites checklist

Step 1: Probe grid availability at startup

Missing grids must fail fast and loud at process start, never per-feature deep inside a batch. Enumerate every grid your manifest depends on and confirm it resolves in a PROJ search path before any geometry is touched. Run this once during bootstrap.

python
# pyproj >= 3.6 — Python 3.10+
import logging
from pathlib import Path

import pyproj
from pyproj.datadir import get_data_dir

logger = logging.getLogger("crs.grids")

# Grids the transformation manifest depends on. Keep this list in version control.
REQUIRED_GRIDS: tuple[str, ...] = (
    "us_noaa_nadcon5_nad27_nad83_1986_conus.tif",  # NAD27 -> NAD83 CONUS
    "ca_nrc_ntv2_0.tif",                            # NAD27 -> NAD83 Canada
)


def probe_grids(required: tuple[str, ...] = REQUIRED_GRIDS) -> dict[str, bool]:
    """Return {grid_name: is_present}. Never transform before this is clean."""
    search_dirs = [Path(get_data_dir())]
    availability: dict[str, bool] = {}
    for grid in required:
        found = any((d / grid).is_file() for d in search_dirs)
        availability[grid] = found
        level = logging.INFO if found else logging.ERROR
        logger.log(level, "grid %s present=%s (PROJ_DATA=%s)", grid, found, get_data_dir())
    return availability

Startup rules:

  • Treat a missing grid as a deployment-blocking condition in strict environments; downgrade to a warning only where the coarse path is explicitly signed off.
  • Probe against pyproj.datadir.get_data_dir(), not a hardcoded path — it reflects PROJ_DATA and the wheel default in the true resolution order.
  • Never call pyproj sync at runtime to self-heal; on-demand fetching makes the result depend on network state and breaks reproducibility.

Step 2: Inspect unavailable_operations

TransformerGroup is the authoritative source of truth for which path is missing and why. Its available_operations list holds paths PROJ can execute now; unavailable_operations holds paths that exist in the database but need a grid that is not installed, each exposing grid_name and a url to fetch it. This is the same inspection surface the parent Datum Transformation Fallback Chains routing engine consumes.

python
# pyproj >= 3.6 — Python 3.10+
from dataclasses import dataclass

from pyproj import CRS
from pyproj.transformer import TransformerGroup


@dataclass(frozen=True)
class PathReport:
    have_grid_path: bool          # a grid-based operation is installed and usable
    best_available_accuracy_m: float | None
    missing_grids: tuple[str, ...]  # grids that would unlock a more accurate path


def report_paths(src_epsg: int, dst_epsg: int) -> PathReport:
    group = TransformerGroup(CRS.from_epsg(src_epsg), CRS.from_epsg(dst_epsg), always_xy=True)

    missing: list[str] = []
    for op in group.unavailable_operations:
        # Each grid dependency PROJ knows about but cannot resolve locally.
        missing.extend(g.short_name for g in op.grids if not g.available)

    best = group.transformers[0] if group.transformers else None
    acc = getattr(best, "accuracy", -1.0) if best else -1.0
    # A grid-based available op is exact; if the best available op is coarse, we degraded.
    have_grid = bool(group.transformers) and group.transformers[0].is_exact

    return PathReport(
        have_grid_path=have_grid,
        best_available_accuracy_m=None if acc < 0 else acc,
        missing_grids=tuple(dict.fromkeys(missing)),  # dedupe, keep order
    )

Inspection rules:

  • A non-empty unavailable_operations list is the definitive signal that a more accurate path exists but is grid-deficient — not that no path exists.
  • Read op.grids[*].available per operation; a single operation can depend on several tiled grids where only some are present.
  • Cache one PathReport per distinct source/target pair. Constructing TransformerGroup per feature dominates runtime, exactly as covered in Multi-CRS Dataset Harmonization.
What TransformerGroup Reports When a Grid Is Absent A transformer group for one datum pair is drawn as two lists. The available operations list holds a coarse three-parameter path with an accuracy of about one metre. The unavailable operations list holds the grid-based path with an accuracy of about five centimetres, blocked because its NTv2 grid file is not installed, together with the grid name and its download URL. The router selects the best available path, tags every transformed record with a degraded-accuracy flag, and emits a GRID_MISSING alert. A note records that without this inspection PROJ would silently take the coarse path and report nothing. available_operations NAD27 to NAD83 (3-parameter) accuracy ≈ 1.0 m · no grid required unavailable_operations NADCON5 grid path · accuracy ≈ 0.05 m blocked: grid file not installed router decision take the best available path tag records accuracy_degraded=true emit GRID_MISSING alert selected reported

Step 3: Degrade to a controlled fallback with alerting

When the grid path is missing, the pipeline must make a decision, not drift. Either reject the batch (accuracy-critical) or accept the coarse path with a degraded-accuracy flag and an alert. The numeric threshold that separates “acceptable coarse” from “reject” is owned by Unit Conversion & Tolerance Thresholds; this step only routes on that decision.

python
# pyproj >= 3.6, geopandas >= 0.14 — Python 3.10+
import geopandas as gpd
from pyproj import CRS
from pyproj.transformer import TransformerGroup


class GridMissingError(RuntimeError):
    """Raised when an accuracy-critical transform lacks its NTv2 grid."""


def transform_with_fallback(
    gdf: gpd.GeoDataFrame, dst_epsg: int, tolerance_m: float, strict: bool
) -> gpd.GeoDataFrame:
    src = CRS.from_user_input(gdf.crs)
    group = TransformerGroup(src, CRS.from_epsg(dst_epsg), always_xy=True)
    if not group.transformers:
        raise GridMissingError(f"no path {gdf.crs} -> EPSG:{dst_epsg}")

    best = group.transformers[0]
    accuracy = getattr(best, "accuracy", -1.0)
    grid_deficient = bool(group.unavailable_operations)

    if grid_deficient and (accuracy < 0 or accuracy > tolerance_m):
        msg = f"grid missing: best accuracy={accuracy} m exceeds tolerance {tolerance_m} m"
        if strict:
            raise GridMissingError(msg)
        # Non-strict: emit a pageable alert, then proceed on the coarse path.
        logger.error("GRID_MISSING alert: %s (src=%s dst=EPSG:%d)", msg, gdf.crs, dst_epsg)

    out = gdf.to_crs(epsg=dst_epsg)
    out.attrs["degraded_accuracy"] = grid_deficient
    out.attrs["operation_name"] = best.description
    return out

Fallback rules:

  • In strict mode, a missing accuracy-critical grid raises and routes the asset to the reconciliation queue — never silently ships a 2 m error.
  • Emit GRID_MISSING as a structured, pageable alert, not a debug line; a grid regression on one worker is an incident, not noise.
  • Stamp degraded_accuracy and the selected operation_name onto the output so downstream consumers can judge fitness for use.

Step 4: Pin and install grids with projsync

The permanent fix is to stop depending on ambient PROJ data. Fetch the exact grids at build time into a directory you bake into the image, and pin them by name so a proj-data release cannot shift results underneath you.

bash
# Build-time only — pyproj >= 3.6
export PROJ_DATA=/opt/proj-grids
mkdir -p "$PROJ_DATA"

# Fetch only the grids the manifest declares, into the pinned dir.
pyproj sync --target-dir "$PROJ_DATA" --source-id us_noaa --list-files
pyproj sync --target-dir "$PROJ_DATA" --source-id us_noaa
pyproj sync --target-dir "$PROJ_DATA" --source-id ca_nrc

# Freeze an inventory so drift is detectable in CI.
( cd "$PROJ_DATA" && sha256sum *.tif ) > grids.lock

Bake PROJ_DATA=/opt/proj-grids and PROJ_NETWORK=OFF into the runtime environment so the container never reaches the CDN in production. Commit grids.lock and diff it in CI; a changed hash is a deliberate, reviewed grid update, not a surprise.

Step 5: Log grid provenance to the audit trail

Every transform records which grid backed it. This row is what an auditor reads to confirm a NAD27→NAD83 shift used the published NADCON5 grid rather than a Helmert approximation.

python
# pyarrow >= 14 — Python 3.10+
import datetime as dt

import pyarrow as pa
import pyarrow.parquet as pq


def append_grid_lineage(path: str, src: str, dst: str, report: "PathReport") -> None:
    row = {
        "src_crs": src,
        "dst_crs": dst,
        "grid_path_used": report.have_grid_path,
        "best_accuracy_m": report.best_available_accuracy_m,
        "missing_grids": ",".join(report.missing_grids) or None,
        "timestamp_utc": dt.datetime.now(dt.UTC).isoformat(),
    }
    table = pa.Table.from_pylist([row])
    pq.write_to_dataset(table, root_path=path)  # append-only lineage partition

Persist these rows to the append-only lineage store consumed by Geospatial Compliance Reporting & Audit Trails; grid_path_used=false rows are the exact set a data-quality review must inspect.

Verification

Confirm the guard actually catches an absent grid rather than trusting a green build:

python
# Point PROJ at an empty dir to simulate a stripped production worker.
import os, tempfile, pyproj

os.environ["PROJ_DATA"] = tempfile.mkdtemp()
pyproj.datadir.set_data_dir(os.environ["PROJ_DATA"])

report = report_paths(4267, 4326)  # NAD27 -> WGS84
assert report.have_grid_path is False, "expected grid-deficient with empty PROJ_DATA"
assert report.missing_grids, "unavailable_operations should name the missing NADCON5 grid"

On the CLI, projinfo -s EPSG:4267 -t EPSG:4326 --grid-check known_available prints Grid ... available: false for a stripped worker; after Step 4 the same command reports available: true, and pyproj sync --list-files --source-id us_noaa echoes the pinned filenames.

Troubleshooting

Symptom Likely cause Fix
Output shifted 1–2 m vs a reference layer NADCON5/NTv2 grid absent; PROJ substituted a coarse geocentric path Run the Step 1 probe; if have_grid_path is false, pyproj sync the grid (Step 4) and re-run.
unavailable_operations non-empty but transform still “works” A more accurate grid path exists yet is grid-deficient; PROJ used the next path Treat as degraded — assert on unavailable_operations in Step 2 and alert per Step 3.
Results differ between CI and production Divergent proj-data package versions across images Pin PROJ_DATA and commit grids.lock; diff the hashes in CI (Step 4).
Grid fetch hangs or 404s at runtime PROJ_NETWORK=ON reaching the CDN mid-batch Set PROJ_NETWORK=OFF in production; fetch only at build time.
accuracy == -1 on the selected path PROJ cannot estimate error for the fallback operation In strict mode, reject and reconcile; otherwise flag degraded_accuracy=true and page.