Implementing Unit Conversion & Tolerance Thresholds in Geospatial ETL Pipelines Jump to heading#

Geospatial data standardization requires deterministic handling of measurement units and spatial precision. In automated schema mapping pipelines, inconsistent input formats — ranging from imperial survey feet to metric decimal degrees — introduce cumulative drift if not normalized early. This stage operates at the core of CRS Normalization & Sync: the discipline concerned with guaranteeing that every geometry shares a coherent spatial reference and a single measurement system before downstream schema operations begin.

This page is scoped to the numeric contract — how raw values are scaled into canonical units and how the resulting residuals are gated. Selecting and validating EPSG codes for mixed inputs belongs to Projection Normalization Workflows. Routing around missing transformation grids is handled by Datum Transformation Fallback Chains, and merging outputs from several authority sources is covered by Multi-CRS Dataset Harmonization. Here the focus is config-as-code definitions, Python-based ETL execution, validation gatekeeping, and compliance reporting for government and enterprise GIS teams.

Configuration Manifest: Mandatory vs. Optional Fields Jump to heading#

All conversion factors, precision limits, and snapping tolerances must be externalized to YAML manifests. Hardcoded constants break reproducibility and violate audit requirements. The schema below enforces strict field classification:

# unit_conversion_config.yaml  (loaded with pyyaml >=6.0)
conversion_rules:
  linear_units:
    source: "survey_foot"          # MANDATORY: Input unit identifier (EPSG/OGC compliant)
    target: "meter"                # MANDATORY: Output unit identifier
    factor: 0.3048006096           # MANDATORY: Exact multiplier (no rounding)
    precision: 6                   # OPTIONAL: Decimal places for output (default: 6)
  angular_units:
    source: "dms"                  # OPTIONAL: Required only if angular attributes exist
    target: "decimal_degrees"
    precision: 9
tolerance_thresholds:
  coordinate_snapping: 0.001       # MANDATORY: Max snapping distance in target units
  attribute_rounding: 1e-6         # MANDATORY: Float precision floor for numeric fields
  topology_gap: 0.005              # OPTIONAL: Acceptable gap tolerance for polygon edges
  max_drift_percent: 0.05          # MANDATORY: Maximum allowable deviation from reference
attributes_to_convert:             # OPTIONAL: List of non-geometry columns requiring scaling
  - "elevation_ft"
  - "survey_distance"

Field compliance rules:

• Mandatory fields must be present for pipeline initialization. Missing values trigger a SchemaValidationError and halt execution — there is no silent default for a measurement contract.
• Optional fields inherit the safe defaults documented above. Omission does not block execution but must be logged for audit trails.
• The US Survey Foot factor (0.3048006096) is distinct from the International Foot (0.3048). Use the correct factor per the NIST US Survey Foot reference to prevent regulatory non-compliance. The US Survey Foot was officially retired on January 1, 2023, but legacy datasets encoded in this unit remain in widespread circulation.

Preprocessing Requirements Jump to heading#

Scaling is only deterministic when the input shape is known before any factor is applied. Run these checks at the boundary of the stage:

• Single declared source unit per dataset. A mixed batch — some features in feet, some in metres — must be split or tagged upstream; this stage assumes one source unit. Discovering and reconciling per-feature unit metadata is the job of Multi-CRS Dataset Harmonization, not this one.
• Known, valid target CRS. Linear scaling is meaningless on geographic (degree-based) coordinates. Confirm the geometry is in a projected CRS — and reproject first via Projection Normalization Workflows — before treating coordinates as scalable lengths.
• Numeric attribute dtypes. Columns named in attributes_to_convert must coerce cleanly to float64. Object-typed columns carrying stringified numbers ("1200.00") are cast explicitly; non-numeric tokens are routed to the rejection log rather than producing NaN silently.
• No pre-rounded geometry. If an upstream stage already snapped vertices to a coarser grid, re-snapping here compounds error. Treat geometry as raw until this stage owns the precision contract.

Execution Engine & Precision Guards Jump to heading#

The transformation stage executes in two phases: unit normalization and precision enforcement. Avoid implicit float multiplication; use explicit vectorized operations to maintain throughput and prevent silent precision loss.

# geopandas >=0.14, shapely >=2.0, numpy >=1.26  (Python 3.10+)
import geopandas as gpd
import numpy as np
from shapely.ops import transform
from typing import Any

class SchemaValidationError(ValueError):
    """Raised when a mandatory manifest field is missing or a value cannot be coerced."""


def apply_unit_scaling(gdf: gpd.GeoDataFrame, config: dict[str, Any]) -> gpd.GeoDataFrame:
    """Apply deterministic linear scaling to geometry and specified attributes."""
    try:
        linear = config["conversion_rules"]["linear_units"]
        factor = float(linear["factor"])
    except (KeyError, TypeError, ValueError) as exc:
        raise SchemaValidationError(f"linear_units.factor missing or invalid: {exc}") from exc
    precision = int(linear.get("precision", 6))

    # Coordinate scaling via Shapely transform (avoids row-level apply overhead)
    def scale_coords(x, y, z=None):
        if z is not None:
            return (x * factor, y * factor, z * factor)
        return (x * factor, y * factor)

    gdf = gdf.copy()
    gdf["geometry"] = gdf.geometry.apply(lambda geom: transform(scale_coords, geom))

    # Vectorized attribute conversion with explicit numeric coercion
    for col in config.get("attributes_to_convert", []):
        if col in gdf.columns:
            coerced = gpd.pd.to_numeric(gdf[col], errors="coerce")
            if coerced.isna().any():
                bad = gdf.loc[coerced.isna(), col].tolist()
                raise SchemaValidationError(f"Non-numeric tokens in '{col}': {bad[:5]}")
            gdf[col] = np.round(coerced.astype(float) * factor, precision)

    return gdf

Conversion factors that arrive from a CRS definition rather than the manifest should be read through pyproj so the exact authority value is used. Wrap that lookup so a malformed definition surfaces as a controlled error rather than a stack trace:

# pyproj >=3.6
from pyproj import CRS
from pyproj.exceptions import CRSError

def linear_factor_from_crs(crs_input: str) -> float:
    """Return the unit-to-metre factor declared by a projected CRS."""
    try:
        crs = CRS.from_user_input(crs_input)
    except CRSError as exc:
        raise SchemaValidationError(f"Unparseable CRS '{crs_input}': {exc}") from exc
    if crs.is_geographic:
        raise SchemaValidationError("Refusing to scale a geographic CRS as linear length.")
    return crs.axis_info[0].unit_conversion_factor

For teams managing heterogeneous municipal datasets, aligning these parameters with the projection stage ensures that linear distortions do not compound during unit translation. When scaling intersects with geodetic transformations, the datum fallback strategy preserves coordinate integrity across legacy survey networks.

Threshold Validation & Compliance Gatekeeping Jump to heading#

After transformation, the pipeline must enforce tolerance thresholds before committing to the target datastore. Validation occurs in three sequential checks:

1. Coordinate snapping: Vertices within coordinate_snapping distance of a reference grid are snapped to eliminate micro-slivers. The deterministic snapping algorithm itself is detailed in Setting tolerance thresholds for automated coordinate snapping.
2. Attribute rounding: Numeric outputs are clamped to attribute_rounding to prevent floating-point artifacts from propagating to downstream analytics.
3. Drift verification: The pipeline computes (max(observed - expected) / expected) * 100 against a control sample. If the result exceeds max_drift_percent, the batch is quarantined.

The three checks partition every residual along a single signed axis. Anything beneath attribute_rounding collapses to zero, anything between the snap floor and coordinate_snapping is realigned to the nearest reference vertex, the band up to max_drift_percent is accepted as-is, and anything beyond is quarantined. The diagram below maps those bands onto one residual axis so the boundary behaviour is unambiguous:

# numpy >=1.26  (Python 3.10+)
def validate_drift(
    gdf: gpd.GeoDataFrame,
    reference_gdf: gpd.GeoDataFrame,
    max_drift_pct: float,
) -> bool:
    """Verify coordinate drift against a trusted reference dataset.

    Returns True if drift is within acceptable bounds, False otherwise.
    Requires that gdf and reference_gdf share the same index alignment.
    """
    if reference_gdf.empty or len(gdf) == 0:
        return True  # Skip if no reference available

    sample_size = min(100, len(gdf), len(reference_gdf))
    rng = np.random.default_rng(seed=42)  # Deterministic sample for reproducibility
    idx = rng.choice(min(len(gdf), len(reference_gdf)), sample_size, replace=False)

    observed = gdf.iloc[idx].geometry.centroid
    expected = reference_gdf.iloc[idx].geometry.centroid

    distances = observed.distance(expected)
    max_dist = distances.max()

    # Normalize by the diagonal of the expected extent to get a dimensionless ratio
    bounds = expected.total_bounds  # [minx, miny, maxx, maxy]
    extent_diag = ((bounds[2] - bounds[0]) ** 2 + (bounds[3] - bounds[1]) ** 2) ** 0.5
    if extent_diag == 0:
        return True  # Degenerate case

    drift_pct = (max_dist / extent_diag) * 100
    return drift_pct <= max_drift_pct

Failure Modes & Fallback Routing Jump to heading#

No condition in this stage may fail silently. Each fault maps to a deterministic recovery action and a logged outcome:

Compliance Reporting Output Jump to heading#

Every run writes a lineage record so an auditor can reconstruct exactly what was applied. The stage emits one structured entry per batch to the audit trail, suitable for the ISO 19115 lineage element:

# Python 3.10+
import json, hashlib, datetime as dt

def write_lineage(config: dict, gdf_in, gdf_out, drift_ok: bool, rejected: int) -> dict:
    record = {
        "stage": "unit_conversion_tolerance",
        "timestamp_utc": dt.datetime.now(dt.timezone.utc).isoformat(),
        "source_unit": config["conversion_rules"]["linear_units"]["source"],
        "target_unit": config["conversion_rules"]["linear_units"]["target"],
        "factor_applied": config["conversion_rules"]["linear_units"]["factor"],
        "precision": config["conversion_rules"]["linear_units"].get("precision", 6),
        "thresholds": config["tolerance_thresholds"],
        "features_in": len(gdf_in),
        "features_out": len(gdf_out),
        "features_rejected": rejected,
        "drift_within_tolerance": drift_ok,
        "config_sha256": hashlib.sha256(
            json.dumps(config, sort_keys=True).encode()
        ).hexdigest(),
    }
    return record

The required lineage fields are: the applied conversion factor, output precision, every active tolerance threshold, in/out/rejected feature counts, the drift verdict, and a hash of the exact manifest used. The rejection log captures, per quarantined feature, its identifier, the failure category from the table above, and the residual or token that triggered it — never a bare count. These fields satisfy ISO 19115 lineage metadata and FGDC compliance audits.

CI Integration Jump to heading#

Embed validation into continuous integration so schema violations are caught before deployment, not in production. The following GitHub Actions workflow is version-agnostic and relies on standard Python packaging:

name: Validate Unit Conversion Pipeline
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.x'
      - name: Install dependencies
        run: pip install geopandas pyproj shapely pyyaml pytest
      - name: Run validation suite
        run: pytest tests/test_unit_conversion.py --junitxml=report.xml
      - name: Upload compliance report
        uses: actions/upload-artifact@v4
        with:
          name: etl-validation-report
          path: report.xml

A minimal pytest fixture pins the two failure modes most likely to regress on a PROJ database update — wrong-foot drift and geographic-CRS misuse:

# pytest >=8.0
import pytest

@pytest.fixture
def survey_foot_config():
    return {
        "conversion_rules": {"linear_units": {
            "source": "survey_foot", "target": "meter",
            "factor": 0.3048006096, "precision": 6}},
        "tolerance_thresholds": {
            "coordinate_snapping": 0.001, "attribute_rounding": 1e-6,
            "max_drift_percent": 0.05},
    }

def test_international_foot_factor_trips_drift(survey_foot_config, sample_gdf, reference_gdf):
    wrong = dict(survey_foot_config)
    wrong["conversion_rules"]["linear_units"]["factor"] = 0.3048  # international foot
    out = apply_unit_scaling(sample_gdf, wrong)
    assert not validate_drift(out, reference_gdf, max_drift_pct=0.05)

All transformations must log the applied factor, precision, and drift metrics to satisfy ISO 19115 metadata requirements and FGDC compliance audits.

Frequently Asked Questions Jump to heading#

What is the difference between the US Survey Foot and International Foot conversion factor? The US Survey Foot factor is 0.3048006096 metres; the International Foot factor is exactly 0.3048 metres. The two diverge by roughly 2 parts per million — about 3.2 mm per kilometre. Across a state-plane extent that is easily enough to exceed max_drift_percent, which is why the manifest must declare the exact factor for the dataset’s declared source unit rather than rounding to 0.3048. The wrong-foot case is the single most common cause of a quarantined batch in this stage.

Why must I reproject before applying linear unit scaling? Linear scaling multiplies coordinate values as though they are lengths. Geographic coordinates are angles in degrees, so multiplying them by a metre factor yields meaningless geometry. The execution engine refuses a geographic CRS with a SchemaValidationError; reproject to a projected CRS through Projection Normalization Workflows first, then treat the coordinates as scalable.

How are tolerance thresholds chosen for coordinate snapping and drift? Thresholds are derived from the dataset’s surveyed accuracy and storage precision, never picked arbitrarily. attribute_rounding sits at the float-noise floor (1e-6), coordinate_snapping is set just above the smallest legitimate vertex spacing so micro-slivers collapse while real detail survives, and max_drift_percent is tied to the accuracy class the data must certify against. The deterministic snapping mechanics are worked through in Setting tolerance thresholds for automated coordinate snapping.

What happens when no reference dataset is available to verify drift? Drift verification is skipped rather than silently passed. The stage writes a drift_unverified warning into the lineage record so an auditor can see that converted geometry was accepted without a golden-sample comparison, and the batch is flagged for later review against a trusted control.

Related Jump to heading#

• Setting tolerance thresholds for automated coordinate snapping — the deterministic vertex-snapping algorithm behind the first validation check.
• CRS Normalization & Sync — parent discipline and the stage that owns spatial reference coherence.
• Projection Normalization Workflows — reproject to a projected CRS before treating coordinates as scalable lengths.
• Datum Transformation Fallback Chains — route around missing transformation grids while preserving accuracy.
• Multi-CRS Dataset Harmonization — reconcile per-feature unit and CRS metadata across mixed sources.
• Field Renaming & Type Coercion Rules — downstream attribute casting that consumes these canonical numeric values.