Geospatial Schema Standardization & Mapping Automation

FGDC Metadata Mapping: Implementation Patterns for Automated Schema Transformation

In production geospatial pipelines, FGDC Metadata Mapping operates as a deterministic transformation stage rather than a manual documentation exercise. This stage sits inside the Geospatial Schema Architecture & Standards Mapping discipline, where version-controlled configuration files replace ad-hoc translation scripts. Government data teams and Python ETL engineers require a config-as-code architecture that enforces strict schema alignment, applies configurable tolerance thresholds, and generates auditable compliance reports. This guide details the implementation of a metadata transformation stage, focusing on field-level mapping, validation rules, and fallback routing for non-conforming records.

This page covers the mapping of FGDC CSDGM source elements onto a target catalog schema — the parse, match, validate, and route loop for a single metadata standard. Where you need to emit a different standard from the mapped output, Converting FGDC CSDGM to ISO 19115 automatically handles cross-standard element translation; European conformance is owned by INSPIRE Directive Schema Compliance; and local synonym alignment lives in Local Government Data Dictionaries. This stage stops at producing a validated, mapped record and a compliance report — downstream publication and cross-platform emission are deliberately out of scope here.

FGDC Metadata Mapping Stage FGDC CSDGM XML elements are parsed into a directed transformation graph, mapped to target catalog attributes, and checked by a mandatory-element completeness gate; complete records produce a compliance report while incomplete records route to a fallback queue. CSDGM XML source metadata Parse to graph element nodes Map attributes target schema Complete? mandatory set Compliance report pass · audit trail Fallback queue quarantine · review yes missing field

Declarative Configuration Manifest

The foundation of a reliable transformation workflow is a declarative configuration layer. Hardcoded field translations introduce schema drift and break continuous integration pipelines. Instead, maintain a YAML mapping manifest that defines source FGDC CSDGM elements, target attributes, transformation functions, and compliance flags. When the pipeline initializes, a schema loader parses this manifest into a directed acyclic graph (DAG) of transformation nodes, validating that every mandatory rule resolves to a non-null XPath before a single record is processed.

yaml
# metadata_mapping.yaml — config-as-code mapping manifest
# Loaded once at pipeline init; PyYAML >=6.0 safe_load only.
mapping_rules:
  - source: "idinfo/citation/citeinfo/title"
    target: "dataset_title"
    mandatory: true
    strict_match: true
    fallback_value: null
    transform: "strip_whitespace"

  - source: "idinfo/descript/abstract"
    target: "summary"
    mandatory: false
    strict_match: false          # enables synonym-dictionary resolution
    fallback_value: "Abstract not provided."
    transform: "normalize_newlines"

  - source: "idinfo/citation/citeinfo/pubdate"
    target: "publication_date"
    mandatory: true
    strict_match: true
    fallback_value: null
    transform: "iso8601_parse"   # 8-digit YYYYMMDD → ISO 8601 date

  - source: "idinfo/spdom/bounding"
    target: "bbox"
    mandatory: true
    strict_match: true
    fallback_value: null
    transform: "fgdc_bounding_to_envelope"

Each directive controls one axis of the transform. The table below fixes the contract every rule must satisfy:

Directive Required Type Purpose
source mandatory string (XPath) CSDGM element path resolved against the parsed tree
target mandatory string Attribute name written to the output record
mandatory mandatory bool If true, a null result rejects the record at the validation gate
strict_match optional (default true) bool false enables synonym-dictionary (fuzzy) resolution
fallback_value optional (default null) any Substituted only for non-mandatory fields when the source is absent
transform optional (default identity) string Named coercion applied to the raw element text

Explicit mandatory and optional field definitions prevent silent data loss. The strict_match flag dictates whether exact XPath resolution is required; when set to false, the loader wires the rule to the fuzzy resolver described under preprocessing.

Preprocessing Requirements

FGDC CSDGM in the wild is rarely XSD-clean. Before any rule fires, the source tree must be normalized into a predictable shape, otherwise XPath resolution silently returns None and pushes records into the fallback queue for the wrong reason. The execution engine expects each input to satisfy these shape guarantees:

  • Encoding normalized to UTF-8. Legacy CSDGM exports are frequently Latin-1 or Windows-1252; decode and re-encode before parsing so lxml does not abort on byte 0x92 smart quotes.
  • Whitespace and newline collapse. Multi-line <abstract> blocks carry indentation from the original editor. Collapse runs of whitespace before matching so confidence scoring is not skewed by formatting.
  • Deprecated and vendor tags stripped. Esri and legacy NBII extensions inject non-CSDGM elements; strip unknown tags so the DAG only sees declared source paths.
  • Entity references resolved. Expand &amp;, &#xNN; and external entities up front; disable external entity loading to prevent XXE on untrusted government feeds.
  • Synonym dictionary loaded. For every rule with strict_match: false, load the alignment dictionary — optionally seeded from Local Government Data Dictionaries — so fuzzy resolution has a vocabulary to score against.

During mapping, apply a confidence scoring mechanism: exact string matches receive 1.0, semantic matches via synonym dictionaries receive 0.7–0.9, and unmapped fields below the 0.7 threshold trigger the fallback router.

Confidence-Score Routing of Mapped Fields Each FGDC source element is assigned a match confidence score. Exact XPath matches score 1.0 and are accepted; synonym-dictionary semantic matches score between 0.7 and 0.9 and are accepted with a logged note; unmapped fields score below the 0.7 threshold and route to the fallback router for review. Source element score the match score ≥ 0.7? threshold gate pass below threshold Exact match · accept confidence = 1.0 · XPath hit Synonym match · accept + log confidence 0.7–0.9 · dictionary Unmapped · fallback router confidence < 0.7 · review

Execution Engine & Precision Guards

The extraction stage must handle heterogeneous inputs without blocking downstream processes. Implement a Python parser using lxml for XML-based CSDGM records and fiona for metadata embedded in shapefiles, GeoPackages, and GeoTIFFs. The engine never lets a malformed source crash the batch: parse errors, missing mandatory fields, and transform failures are caught individually and converted into structured rejection records rather than uncaught exceptions.

python
# requires: python >=3.10, lxml >=5.1, PyYAML >=6.0
from lxml import etree
from dataclasses import dataclass, field
from typing import Any, Optional
import logging

log = logging.getLogger("fgdc.map")


@dataclass
class MappedRecord:
    values: dict[str, Any] = field(default_factory=dict)
    scores: dict[str, float] = field(default_factory=dict)
    rejected: bool = False
    reason: Optional[str] = None


class MappingEngine:
    SECURE_PARSER = etree.XMLParser(
        resolve_entities=False, no_network=True, huge_tree=False
    )

    def __init__(self, config: dict[str, Any], threshold: float = 0.7):
        self.rules = config["mapping_rules"]
        self.threshold = threshold

    def resolve(self, raw_xml: bytes) -> MappedRecord:
        rec = MappedRecord()
        try:
            tree = etree.fromstring(raw_xml, parser=self.SECURE_PARSER)
        except etree.XMLSyntaxError as exc:
            rec.rejected, rec.reason = True, f"parse_error: {exc}"
            log.warning("CSDGM parse failed: %s", exc)
            return rec

        for rule in self.rules:
            node = tree.find(rule["source"])
            value = node.text.strip() if node is not None and node.text else None
            score = 1.0 if value is not None else 0.0

            if value is None and rule["mandatory"]:
                rec.rejected = True
                rec.reason = f"missing_mandatory: {rule['target']}"
                log.info("reject %s: %s", rule["target"], rec.reason)
                return rec  # fail fast — do not emit a partial record

            if value is None and not rule["mandatory"]:
                value, score = rule.get("fallback_value"), 0.0

            rec.values[rule["target"]] = value
            rec.scores[rule["target"]] = score
        return rec

The SECURE_PARSER disables entity resolution and network access, closing the XXE vector that untrusted FGDC feeds can carry. Mandatory failures short-circuit the record so no partially mapped output reaches the validation gate. After mapping, a Pydantic model mirrors the target specification and applies type coercion before the record is accepted:

python
# requires: pydantic >=2.6
from pydantic import BaseModel, Field, ConfigDict, ValidationError
from typing import Optional


class TargetMetadata(BaseModel):
    model_config = ConfigDict(populate_by_name=True, str_strip_whitespace=True)

    dataset_title: str = Field(..., min_length=1)
    publication_date: str  # ISO 8601, already coerced by iso8601_parse
    summary: Optional[str] = None


def validate(rec: "MappedRecord") -> "MappedRecord":
    try:
        TargetMetadata(**rec.values)
    except ValidationError as exc:
        rec.rejected = True
        rec.reason = f"schema_violation: {exc.error_count()} error(s)"
    return rec

Mandatory fields use ... (Ellipsis) to enforce presence at runtime; optional fields default to None or fallback strings. This immediate validation gate prevents non-conforming records from propagating into spatial data catalogs.

Failure Modes & Fallback Routing

Non-conforming records are never discarded. Every failure type maps to one deterministic recovery action, so there are no silent drops and every rejection is reproducible from the audit trail.

Failure type Likely cause Deterministic recovery action
parse_error Malformed XML, bad encoding, truncated export Quarantine raw bytes, log XMLSyntaxError, flag for re-export
missing_mandatory Source CSDGM omits a required element Route to fallback queue with the missing target name attached
schema_violation Value fails Pydantic type/length coercion Quarantine with field-level error list; do not auto-substitute
low_confidence All fuzzy matches scored < 0.7 Hold for manual review; surface candidate synonyms in the log
xxe_blocked External entity reference in untrusted feed Reject record, raise security alert, do not retry automatically

Records that fail mandatory validation pass through a fallback router that quarantines the payload, attaches diagnostic logs, and triggers a manual review workflow. This pattern is critical when migrating legacy FGDC records toward modern standards, where historical data gaps are common — particularly when the eventual target is INSPIRE Directive Schema Compliance or an ISO 19115 conversion, both of which impose stricter cardinality than CSDGM ever required.

Compliance Reporting Output

The stage writes a machine-readable compliance report for every batch so audits never depend on re-running the pipeline. Each accepted and rejected record contributes a row carrying lineage and decision provenance:

json
{
  "record_id": "csdgm/parcels_2009.xml",
  "status": "rejected",
  "reason": "missing_mandatory: publication_date",
  "field_status": {
    "dataset_title":   { "value": "County Parcels", "confidence": 1.0 },
    "publication_date": { "value": null, "confidence": 0.0 }
  },
  "lineage": {
    "source_standard": "FGDC-STD-001-1998",
    "mapping_manifest": "metadata_mapping.yaml@a1b9f3c",
    "engine_version": "fgdc-map 2.4.0",
    "processed_at": "2026-06-25T14:02:11Z"
  }
}

The lineage block pins the exact manifest commit and engine version that produced each decision, which is what satisfies a federal compliance audit: any reviewer can reproduce the rejection from the recorded inputs. Confidence scores are retained per field so synonym-matched values (0.7–0.9) are traceable and can be promoted to strict rules once verified. The rejection log uses the same reason codes as the failure-mode table, so dashboards aggregate failures without parsing free-text messages.

CI Integration

Embed the transformation stage into continuous integration to enforce schema compliance before data publication. The job blocks merges when any mandatory field fails validation, ensuring only auditable, standards-compliant metadata reaches production catalogs.

yaml
# .github/workflows/fgdc-validate.yml
name: FGDC Metadata Validation
on:
  push:
    paths: ['data/metadata/**.xml', 'config/mapping.yaml']

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Install dependencies
        # lxml: CSDGM parsing · fiona: spatial-file metadata · pydantic: validation
        run: pip install "pydantic>=2.6" "lxml>=5.1" "fiona>=1.9" "PyYAML>=6.0"
      - name: Run schema validation
        run: python -m pipeline.validate data/metadata/ config/mapping.yaml
      - name: Upload compliance report
        if: always()                       # publish the report even on failure
        uses: actions/upload-artifact@v4
        with:
          name: metadata-audit-report
          path: reports/compliance_*.json

A matching pre-commit hook running the same pipeline.validate entry point catches missing mandatory elements before they ever reach the remote, keeping the audit trail clean. The same gate pattern is reused across INSPIRE Directive Schema Compliance and Cross-Platform Schema Translation, so a single CI convention covers every standard your catalog ingests.

Deeper Implementation Guides