Multi-CRS Dataset Harmonization Jump to heading#

Federated GIS programs almost never receive a single, clean projection. A county parcel layer arrives in a State Plane foot system, a state hydrography extract in geographic degrees, and a federal imagery footprint in UTM metres — and all three must coexist in one queryable dataset before any join, overlay, or schema mapping can run. Reprojecting each source ad hoc, with hardcoded EPSG codes scattered through transform scripts, produces topology breaks at shared boundaries, silent precision drift in spatial indexes, and lineage gaps that fail a compliance audit. Multi-CRS dataset harmonization is the ingestion-stage operation that resolves every incoming layer to one authoritative target CRS through a single deterministic, config-driven routine.

This pipeline stage operates at the core of CRS Normalization & Sync — its parent discipline — which guarantees that every geometry in the system shares a coherent spatial reference before downstream operations begin. This page is scoped to the combination problem — merging many source CRS into one output — and where it hands off to its neighbours. Choosing which transformation path runs when a direct one is unavailable belongs to Datum Transformation Fallback Chains. Resolving ambiguous or deprecated EPSG codes on a single layer is the job of Projection Normalization Workflows. Applying the numeric deviation gates once a path is selected is owned by Unit Conversion & Tolerance Thresholds. Harmonization composes those primitives into one routine that ingests a heterogeneous batch and emits a uniform, audited result.

Harmonization manifest Jump to heading#

Pipeline behaviour is governed by a declarative YAML manifest, not by code. Hardcoded EPSG codes and inline transformation calls cannot be diffed, reviewed, or attached to a lineage record, which makes them unreproducible in production. The manifest pins the authoritative output CRS, the deviation budget, per-source routing, and the validation gates that run after every transform.

# harmonization.yaml — consumed by geopandas >=0.14 / pyproj >=3.6 pipeline
harmonization:
  target_crs: "EPSG:6348"          # MANDATORY: authoritative output CRS (NAD83(2011) UTM 18N)
  tolerance_meters: 0.05           # MANDATORY: max allowable positional deviation
  precision_decimals: 3            # MANDATORY: coordinate rounding precision (mm at metre units)
  sources:                         # OPTIONAL: per-source overrides; auto-detected if omitted
    - match_crs: "EPSG:2263"       #   NY State Plane Long Island (ftUS)
      fallback: "datum_chain_nad83"
    - match_crs: "EPSG:4326"       #   WGS84 geographic
      fallback: "direct"
  fallback_chains:                 # OPTIONAL: named datum-shift sequences for indirect paths
    datum_chain_nad83:
      - method: "grid"             #     prefer NTv2 grid shift
        grid_file: "us_noaa_nadcon5"
      - method: "helmert"          #     fall back to a 7-parameter approximation
        parameters: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  validation:                      # OPTIONAL: post-transform compliance gates
    topology_check: true
    max_area_distortion: 0.001     #   reject if reprojected area drifts >0.1%
    quarantine_on_failure: true    #   route failures to a sidecar, never drop silently

Every transformation run logs the SHA-256 hash of this manifest so any downstream record can be tied back to the exact configuration that produced it, satisfying federal data lineage requirements.

Preprocessing requirements Jump to heading#

Harmonization assumes a clean, per-source CRS declaration on entry. Three normalization steps must complete before the transform engine runs, or the routine will either reject the batch or — worse — reproject from a wrong assumption:

1. Resolve a CRS for every layer. Read .prj WKT, the GeoPackage gpkg_spatial_ref_sys table, or the GeoJSON crs member. A layer with no resolvable CRS is quarantined, never assigned a default.
2. Reject ambiguous or deprecated definitions. Deprecated EPSG codes and bare +proj strings without a datum are sent to single-layer cleanup in Projection Normalization Workflows before they reach this stage; the canonical procedure is the step-by-step EPSG code normalization for mixed datasets walkthrough.
3. Group by source CRS. Transforming row-by-row rebuilds a pyproj.Transformer per geometry and is an order of magnitude slower. Partition the batch into one frame per distinct source CRS so a transformer is constructed once and reused across the whole group.

# preprocess.py — group a heterogeneous batch by source CRS (geopandas >=0.14)
from __future__ import annotations
import geopandas as gpd

def group_by_source_crs(gdf: gpd.GeoDataFrame) -> dict[str, gpd.GeoDataFrame]:
    """Partition a mixed-CRS frame into one sub-frame per authority code.

    Layers with an unresolved CRS are excluded and handled by quarantine upstream.
    """
    if gdf.crs is not None:
        # Already single-CRS: a degenerate group of one.
        return {gdf.crs.to_authority()[1]: gdf}

    groups: dict[str, gpd.GeoDataFrame] = {}
    for code, frame in gdf.groupby("_source_epsg", dropna=True):
        groups[str(code)] = frame.set_crs(f"EPSG:{code}", allow_override=True)
    return groups

Execution engine & precision guards Jump to heading#

Once grouped, each source frame is reprojected as a unit. The engine catches CRSError and ProjError explicitly so a bad transform never propagates as a generic exception, enforces the configured precision, and uses shapely.ops.transform to reproject whole geometries — including multi-part and ring coordinates — rather than transforming bare coordinate pairs.

# engine.py — per-source reprojection with explicit precision and error guards
from __future__ import annotations
import logging
import geopandas as gpd
import pyproj
from pyproj.exceptions import CRSError, ProjError
from shapely.ops import transform as shapely_transform

log = logging.getLogger("harmonization")

def harmonize_group(
    frame: gpd.GeoDataFrame, src_crs: str, config: dict
) -> gpd.GeoDataFrame:
    """Reproject one source-CRS group to the manifest target CRS.

    pyproj >=3.6, shapely >=2.0. Raises on CRS construction or transform failure
    so the caller can route the whole group to quarantine deterministically.
    """
    tgt_crs = config["target_crs"]
    precision = config["precision_decimals"]
    try:
        transformer = pyproj.Transformer.from_crs(
            src_crs, tgt_crs, always_xy=True
        )
    except CRSError as exc:
        raise RuntimeError(f"cannot build transformer {src_crs}->{tgt_crs}: {exc}") from exc

    def _project(x, y, z=None):
        try:
            xt, yt = transformer.transform(x, y)
        except ProjError as exc:
            raise RuntimeError(f"transform failed {src_crs}->{tgt_crs}: {exc}") from exc
        return (round(xt, precision), round(yt, precision))

    out = frame.copy()
    out["geometry"] = out.geometry.apply(lambda g: shapely_transform(_project, g))
    out = out.set_crs(tgt_crs, allow_override=True)
    log.info("harmonized %d features %s->%s", len(out), src_crs, tgt_crs)
    return out

The deviation budget itself is not re-derived here; the engine defers the numeric gate to the standardized matrices in Unit Conversion & Tolerance Thresholds, and the snapping behaviour for near-coincident boundaries follows the tolerance thresholds for automated coordinate snapping procedure. When harmonizing mixed municipal zones, apply the exact scaling factors and false-origin offsets from your organization’s State Plane or UTM mapping tables so edge geometries are not truncated.

Failure modes & fallback routing Jump to heading#

No record is silently dropped or silently reprojected. Each failure type maps to a deterministic recovery action, and every quarantine event writes a row to the rejection log. The diagram below traces a single source group through the engine: a resolvable CRS with a transformation path that passes the gates flows to the harmonized output, while every other outcome is routed — never dropped — to either re-routing or quarantine.

Transient I/O and grid-download failures are not a harmonization concern — they are routed to the retry machinery in Error Handling & Retry Logic so a flaky network fetch never poisons a deterministic spatial result.

Compliance reporting output Jump to heading#

The stage emits machine-readable JSON lineage alongside a human-readable summary. Each harmonized source contributes one lineage record; each rejected feature contributes one rejection record. Both reference the manifest hash so an auditor can reconstruct exactly which configuration produced which geometry.

{
  "manifest_sha256": "9f2c…a17b",
  "target_crs": "EPSG:6348",
  "sources": [
    {
      "source_crs": "EPSG:2263",
      "feature_count": 14820,
      "transformation_path": "EPSG:2263->EPSG:6348 (NTv2: us_noaa_nadcon5)",
      "max_deviation_m": 0.021,
      "max_area_distortion": 0.0004,
      "status": "pass"
    }
  ],
  "rejections": [
    {
      "source_crs": "EPSG:4267",
      "feature_id": "parcel-88213",
      "error_code": "E_TOLERANCE",
      "measured_deviation_m": 0.084,
      "status": "quarantined"
    }
  ]
}

The lineage fields — source_crs, transformation_path, max_deviation_m, and manifest_sha256 — are the minimum set required to defend a harmonized layer in a federal or INSPIRE data-quality review. Quarantine records carry an explicit error_code so remediation can be triaged without re-running the batch.

CI integration Jump to heading#

The harmonization routine is gated in CI against a synthetic fixture covering every source CRS the program ingests, so a PROJ database update or a dependency bump cannot silently shift results. A pytest fixture asserts both success geometry and quarantine routing.

# tests/test_harmonization.py — pytest fixture gating the stage
import geopandas as gpd
import pytest
from shapely.geometry import Point
from harmonize import harmonize_group

CONFIG = {"target_crs": "EPSG:6348", "precision_decimals": 3}

@pytest.fixture
def mixed_batch() -> gpd.GeoDataFrame:
    return gpd.GeoDataFrame(
        {"_source_epsg": [2263]},
        geometry=[Point(985_000, 200_000)],  # NY State Plane LI, ftUS
        crs="EPSG:2263",
    )

def test_harmonizes_to_target_crs(mixed_batch):
    out = harmonize_group(mixed_batch, "EPSG:2263", CONFIG)
    assert out.crs.to_authority() == ("EPSG", "6348")
    # Coordinates clamped to configured precision (no float drift).
    assert out.geometry.iloc[0].x == round(out.geometry.iloc[0].x, 3)

# .github/workflows/harmonization.yml
name: CRS Harmonization Validation
on:
  push:
    paths: ['config/harmonization.yaml', 'src/harmonize/**', 'tests/**']
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: pip install "geopandas>=0.14" "pyproj>=3.6" "shapely>=2.0" pyyaml pytest
      - name: Validate manifest schema
        run: python scripts/validate_config.py config/harmonization.yaml
      - name: Run harmonization tests
        run: pytest tests/test_harmonization.py --junitxml=reports/harmonization.xml

Pin the PROJ data version in the CI image and re-run the fixture whenever it changes; a shifted grid is the single most common cause of a silently regressed harmonization batch. All transformation paths should be cross-checked against the PROJ coordinate transformation library and the pyproj documentation before a manifest change is merged.

Production readiness checklist Jump to heading#

• target_crs matches the organizational spatial data infrastructure (SDI) mandate
• tolerance_meters aligns with the project’s survey accuracy requirement
• Per-source fallback chains explicitly name grid files and Helmert parameters
• Quarantine routing is enabled for tolerance, distortion, and topology violations
• CI blocks merges on harmonization test failures against the synthetic fixture
• Lineage logs retain the manifest hash and transformation path per source

Frequently asked questions Jump to heading#

Which CRS should be the authoritative target for a harmonized dataset? Pick the projected CRS mandated by the consuming spatial data infrastructure, not the CRS of the largest input. For analysis confined to a single UTM zone, a NAD83(2011) UTM projection such as EPSG:6348 keeps distances and areas in metres; for continental extents use the relevant national grid. Set target_crs once in the manifest so every source resolves to the same authority code, and let Projection Normalization Workflows canonicalize any inputs whose declared code is deprecated.

Why group features by source CRS before reprojecting instead of transforming row by row? A pyproj.Transformer is expensive to construct. Building one per row runs roughly an order of magnitude slower than partitioning the batch into one frame per distinct source CRS and reusing a single transformer across the whole group. Grouping also lets the tolerance and topology gates evaluate one transformation path at a time, which keeps the Unit Conversion & Tolerance Thresholds deviation matrices applicable per source rather than per feature.

What happens to a layer that arrives with no resolvable CRS? It is quarantined and never assigned a default. Harmonization writes an E_CRS_MISSING rejection record and routes the layer to the audit trail. Assigning an assumed CRS would silently reproject from a wrong origin — the single most damaging harmonization error — so detection halts the layer rather than guessing.

Does falling back to a Helmert approximation invalidate the harmonized output? No, but it must be flagged. When the preferred NTv2 or NADCON grid is missing, the named Datum Transformation Fallback Chains sequence degrades to a 7-parameter Helmert transform and emits a W_GRID_MISSING degraded-accuracy warning. The output stays usable for many purposes provided residual deviation remains inside tolerance_meters, and the lineage record names the approximate path so an auditor can judge fitness for use.

Related Jump to heading#

• Datum Transformation Fallback Chains — the routing contract that selects which transformation path a harmonized source uses
• Projection Normalization Workflows — canonicalizing ambiguous EPSG codes before harmonization receives them
• Unit Conversion & Tolerance Thresholds — the deviation matrices the tolerance gate enforces
• Batch Schema Processing Pipelines — running harmonization across large shapefile batches where per-CRS grouping matters
• Error Handling & Retry Logic — backoff and circuit-breaker patterns for the transient I/O failures harmonization defers