Geospatial Schema Standardization & Mapping Automation

Capture Failed Feature Transforms in a Dead-Letter Queue

A retry policy answers “what do we do when a transform fails transiently?” It says nothing about the feature that fails permanently — the polygon with a corrupt ring, the record whose parcel_id is null under a NOT NULL contract, the geometry that no CRS repair will fix. Left unhandled, that one feature either aborts a ten-million-row batch or gets swallowed and never accounted for. A dead-letter queue (DLQ) is the durable, structured landing zone for exactly those features: everything that exhausts its retries lands there with enough context to diagnose and replay, and nothing disappears silently. This procedure operates inside Error Handling & Retry Logic, which owns the retry policy and backoff contract; the DLQ is the terminal sink that catches whatever the retries could not.

The design goal is auditability under load. Every dead-lettered feature must be attributable (which batch, which rule, which error), replayable (re-run it after a fix without re-running the whole job), and bounded (a “poison message” that fails every replay must not loop forever). Those three properties drive the schema, the storage layout, and the replay guard below.

Prerequisites checklist

Step 1: Define the DLQ record schema

The record is the contract. It must be self-describing enough that someone six months later can diagnose a failure without the original code, and versioned so the schema can evolve without invalidating old entries. Store the original feature verbatim (as GeoJSON, preserving geometry) plus structured error context. Never store only an error string — you cannot replay a string.

python
# pyarrow >=14 — Python 3.10+
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
import uuid

DLQ_SCHEMA_VERSION = "1.0"


@dataclass(slots=True)
class DeadLetter:
    dlq_id: str                    # unique per dead-letter event
    feature_id: str                # stable key of the source feature
    batch_id: str                  # which run produced it
    schema_version: str            # DLQ_SCHEMA_VERSION at write time
    failed_rule: str               # e.g. "coerce:built_year->int32"
    error_type: str                # exception class name
    error_message: str             # str(exc), truncated
    attempts: int                  # how many retries were spent
    feature_geojson: str           # verbatim source feature (geometry intact)
    source_crs: str | None         # e.g. "EPSG:26910" — needed to replay
    first_seen: str                # ISO-8601 UTC
    replay_count: int = 0          # incremented each replay attempt
    quarantined: bool = False      # poison flag

    @classmethod
    def create(cls, **kw) -> "DeadLetter":
        return cls(
            dlq_id=str(uuid.uuid4()),
            schema_version=DLQ_SCHEMA_VERSION,
            first_seen=datetime.now(timezone.utc).isoformat(),
            **kw,
        )

Keeping source_crs on the record is not optional for geospatial DLQs: a feature replayed without its original CRS reprojects from the wrong datum and “succeeds” with silently wrong coordinates. The full geometry-preservation rules are covered in preserving geometry during attribute flattening.

Step 2: Write failed features to durable storage

Append dead letters to a Parquet dataset partitioned by batch_id and ingest date. Parquet gives you a typed, compressed, queryable store the DLQ can grow into without a database, and it aligns with the append-only lineage manifests format so the two share tooling. Write atomically — a partial file is worse than a missing one because it corrupts every scan of the partition.

python
# pyarrow >=14 — Python 3.10+
from pathlib import Path
import pyarrow as pa
import pyarrow.parquet as pq


def append_dead_letters(records: list[DeadLetter], root: str) -> Path:
    """Atomically append a batch of dead letters to the partitioned DLQ."""
    if not records:
        raise ValueError("refusing to write an empty DLQ batch")

    table = pa.Table.from_pylist([asdict(r) for r in records])
    batch_id = records[0].batch_id
    part_dir = Path(root) / f"batch_id={batch_id}"
    part_dir.mkdir(parents=True, exist_ok=True)

    final = part_dir / f"{records[0].dlq_id}.parquet"
    tmp = final.with_suffix(".parquet.tmp")
    pq.write_table(table, tmp, compression="zstd")
    tmp.replace(final)                       # atomic rename on POSIX
    return final

Do not write one file per feature at scale — that produces the small-files problem and destroys read performance. Buffer dead letters per batch (or per N records) and flush in groups, which is why append_dead_letters takes a list.

Step 3: Quarantine poison messages and replay survivors

Replay re-runs a dead letter through the same transform after an operator has (presumably) fixed the underlying rule or data. The danger is the poison message: a feature that fails every single time and, if replayed unconditionally, loops forever consuming the queue. Guard with a hard replay cap. When a record crosses it, set quarantined=True and stop replaying it automatically — it now needs a human.

python
# geopandas >=0.14 — Python 3.10+
import json
import logging

log = logging.getLogger("dlq.replay")
MAX_REPLAYS = 3


def replay(record: DeadLetter, transform) -> DeadLetter | None:
    """Re-run one dead letter. Return None on success, updated record on failure."""
    if record.quarantined:
        log.info("skip quarantined %s", record.dlq_id)
        return record

    if record.replay_count >= MAX_REPLAYS:
        record.quarantined = True
        log.warning("poison message quarantined: %s (%s)",
                    record.feature_id, record.failed_rule)
        return record

    record.replay_count += 1
    feature = json.loads(record.feature_geojson)
    try:
        transform(feature, source_crs=record.source_crs)
        log.info("replay ok: %s after %d attempt(s)",
                 record.feature_id, record.replay_count)
        return None                          # cleared — drop from active DLQ
    except Exception as exc:  # noqa: BLE001
        record.error_type = type(exc).__name__
        record.error_message = str(exc)[:500]
        return record                        # still failing — keep in DLQ

The replay function is intentionally idempotent per feature: because it keys on feature_id and returns None only on genuine success, running replay twice cannot double-process a feature or lose one that is still broken.

Dead-letter queue lifecycle A transform whose retries are exhausted writes a dead letter into durable storage. Replay re-runs the feature: success clears it out of the active queue, a failure below the replay cap returns it to the DLQ, and exceeding the cap moves it to a quarantine hold for manual review. Retries exhausted DLQ (Parquet) durable · partitioned Replay < cap? and passes? Cleared drop from active DLQ Quarantine poison · manual review pass cap hit fail, under cap → back to DLQ

Step 4: Emit metrics and lineage

A DLQ you cannot see is a data-loss bug in waiting. Emit at least three signals: arrivals (features entering the DLQ), replay outcomes (cleared vs still-failing), and poison rate (fraction quarantined). A rising poison rate means a systemic problem — a broken rule or an upstream schema change — not scattered bad rows. Forward the same counters to the lineage manifest so failures are part of the audit trail, not a side channel.

python
# Python 3.10+
import json, logging
log = logging.getLogger("dlq.metrics")


def emit_dlq_metrics(batch_id: str, arrived: int, cleared: int,
                     quarantined: int) -> dict:
    active = arrived - cleared - quarantined
    poison_rate = round(quarantined / arrived, 4) if arrived else 0.0
    metrics = {
        "batch_id": batch_id,
        "dlq_arrived": arrived,
        "dlq_cleared": cleared,
        "dlq_quarantined": quarantined,
        "dlq_active": active,
        "poison_rate": poison_rate,
    }
    log.info("%s", json.dumps(metrics))
    if poison_rate > 0.05:                    # 5% quarantine = systemic alarm
        log.error("poison rate %.1f%% exceeds threshold for %s",
                  poison_rate * 100, batch_id)
    return metrics

Verification

Prove the DLQ round-trips: a failing feature lands durably, replays cleanly after a fix, and a genuine poison message quarantines instead of looping.

python
# 1. A failed feature is captured with replayable context.
dl = DeadLetter.create(feature_id="parcel-42", batch_id="run-2026-07-13",
                       failed_rule="coerce:built_year->int32",
                       error_type="ValueError", error_message="not an int",
                       attempts=3, feature_geojson=feature_json,
                       source_crs="EPSG:26910")
path = append_dead_letters([dl], root="/data/dlq")
assert path.exists() and path.stat().st_size > 0, "DLQ write did not persist"

# 2. Replay clears a fixed feature (transform now returns cleanly).
assert replay(dl, good_transform) is None, "fixed feature failed to clear"

# 3. A poison message quarantines at the cap, never loops.
poison = DeadLetter.create(feature_id="bad-ring", batch_id="run-2026-07-13",
                           failed_rule="repair:geometry", error_type="TopologyError",
                           error_message="self-intersection", attempts=3,
                           feature_geojson=bad_json, source_crs="EPSG:4326")
for _ in range(MAX_REPLAYS + 2):
    poison = replay(poison, always_fails)
assert poison.quarantined, "poison message never quarantined — infinite loop risk"

A healthy emit_dlq_metrics line shows dlq_active trending to zero as replays clear the backlog, with a poison_rate below your alarm threshold.

Troubleshooting

Symptom Likely cause Fix
Replay loops forever on the same feature No replay cap, or replay_count not persisted between runs Enforce MAX_REPLAYS; write the updated record back so the counter survives.
Replayed feature succeeds but coordinates are wrong source_crs dropped from the record; replay reprojected from the wrong datum Always persist source_crs; reproject from it, never from a default.
DLQ reads are slow / scans fail One tiny Parquet file per feature (small-files problem) or a partial .tmp left behind Batch writes per run; write to .tmp then atomic replace.
Poison rate spikes across a whole batch Upstream schema change broke a rule, not scattered bad data Alert on poison_rate; fix the rule, then bulk-replay the batch partition.
Features silently missing downstream Failures swallowed instead of dead-lettered Route every exhausted-retry feature to append_dead_letters; assert counts reconcile.