Geospatial Schema Standardization & Mapping Automation

FGDC Metadata Mapping: Implementation Patterns for Automated Schema Transformation

In production geospatial pipelines, FGDC Metadata Mapping operates as a deterministic transformation stage rather than a manual documentation exercise. This stage sits inside the Geospatial Schema Architecture & Standards Mapping discipline, where version-controlled configuration files replace ad-hoc translation scripts. Government data teams and Python ETL engineers require a config-as-code architecture that enforces strict schema alignment, applies configurable tolerance thresholds, and generates auditable compliance reports. This guide details the implementation of a metadata transformation stage, focusing on field-level mapping, validation rules, and fallback routing for non-conforming records.

This page covers the mapping of FGDC CSDGM source elements onto a target catalog schema — the parse, match, validate, and route loop for a single metadata standard. Where you need to emit a different standard from the mapped output, Converting FGDC CSDGM to ISO 19115 automatically handles cross-standard element translation; European conformance is owned by INSPIRE Directive Schema Compliance; and local synonym alignment lives in Local Government Data Dictionaries. This stage stops at producing a validated, mapped record and a compliance report — downstream publication and cross-platform emission are deliberately out of scope here.

FGDC Metadata Mapping Stage FGDC CSDGM XML elements are parsed into a directed transformation graph, mapped to target catalog attributes, and checked by a mandatory-element completeness gate; complete records produce a compliance report while incomplete records route to a fallback queue. CSDGM XML source metadata Parse to graph element nodes Map attributes target schema Complete? mandatory set Compliance report pass · audit trail Fallback queue quarantine · review yes missing field

Declarative Configuration Manifest

The foundation of a reliable transformation workflow is a declarative configuration layer. Hardcoded field translations introduce schema drift and break continuous integration pipelines. Instead, maintain a YAML mapping manifest that defines source FGDC CSDGM elements, target attributes, transformation functions, and compliance flags. When the pipeline initializes, a schema loader parses this manifest into a directed acyclic graph (DAG) of transformation nodes, validating that every mandatory rule resolves to a non-null XPath before a single record is processed.

yaml
# metadata_mapping.yaml — config-as-code mapping manifest
# Loaded once at pipeline init; PyYAML >=6.0 safe_load only.
mapping_rules:
  - source: "idinfo/citation/citeinfo/title"
    target: "dataset_title"
    mandatory: true
    strict_match: true
    fallback_value: null
    transform: "strip_whitespace"

  - source: "idinfo/descript/abstract"
    target: "summary"
    mandatory: false
    strict_match: false          # enables synonym-dictionary resolution
    fallback_value: "Abstract not provided."
    transform: "normalize_newlines"

  - source: "idinfo/citation/citeinfo/pubdate"
    target: "publication_date"
    mandatory: true
    strict_match: true
    fallback_value: null
    transform: "iso8601_parse"   # 8-digit YYYYMMDD → ISO 8601 date

  - source: "idinfo/spdom/bounding"
    target: "bbox"
    mandatory: true
    strict_match: true
    fallback_value: null
    transform: "fgdc_bounding_to_envelope"

Each directive controls one axis of the transform. The table below fixes the contract every rule must satisfy:

Directive Required Type Purpose
source mandatory string (XPath) CSDGM element path resolved against the parsed tree
target mandatory string Attribute name written to the output record
mandatory mandatory bool If true, a null result rejects the record at the validation gate
strict_match optional (default true) bool false enables synonym-dictionary (fuzzy) resolution
fallback_value optional (default null) any Substituted only for non-mandatory fields when the source is absent
transform optional (default identity) string Named coercion applied to the raw element text

Explicit mandatory and optional field definitions prevent silent data loss. The strict_match flag dictates whether exact XPath resolution is required; when set to false, the loader wires the rule to the fuzzy resolver described under preprocessing.

Preprocessing Requirements

FGDC CSDGM in the wild is rarely XSD-clean. Before any rule fires, the source tree must be normalized into a predictable shape, otherwise XPath resolution silently returns None and pushes records into the fallback queue for the wrong reason. The execution engine expects each input to satisfy these shape guarantees:

  • Encoding normalized to UTF-8. Legacy CSDGM exports are frequently Latin-1 or Windows-1252; decode and re-encode before parsing so lxml does not abort on byte 0x92 smart quotes.
  • Whitespace and newline collapse. Multi-line <abstract> blocks carry indentation from the original editor. Collapse runs of whitespace before matching so confidence scoring is not skewed by formatting.
  • Deprecated and vendor tags stripped. Esri and legacy NBII extensions inject non-CSDGM elements; strip unknown tags so the DAG only sees declared source paths.
  • Entity references resolved. Expand &amp;, &#xNN; and external entities up front; disable external entity loading to prevent XXE on untrusted government feeds.
  • Synonym dictionary loaded. For every rule with strict_match: false, load the alignment dictionary — optionally seeded from Local Government Data Dictionaries — so fuzzy resolution has a vocabulary to score against.

During mapping, apply a confidence scoring mechanism: exact string matches receive 1.0, semantic matches via synonym dictionaries receive 0.7–0.9, and unmapped fields below the 0.7 threshold trigger the fallback router.

Confidence-Score Routing of Mapped Fields Each FGDC source element is assigned a match confidence score. Exact XPath matches score 1.0 and are accepted; synonym-dictionary semantic matches score between 0.7 and 0.9 and are accepted with a logged note; unmapped fields score below the 0.7 threshold and route to the fallback router for review. Source element score the match score ≥ 0.7? threshold gate pass below threshold Exact match · accept confidence = 1.0 · XPath hit Synonym match · accept + log confidence 0.7–0.9 · dictionary Unmapped · fallback router confidence < 0.7 · review

Execution Engine & Precision Guards

The extraction stage must handle heterogeneous inputs without blocking downstream processes. Implement a Python parser using lxml for XML-based CSDGM records and fiona for metadata embedded in shapefiles, GeoPackages, and GeoTIFFs. The engine never lets a malformed source crash the batch: parse errors, missing mandatory fields, and transform failures are caught individually and converted into structured rejection records rather than uncaught exceptions.

python
# requires: python >=3.10, lxml >=5.1, PyYAML >=6.0
from lxml import etree
from dataclasses import dataclass, field
from typing import Any, Optional
import logging

log = logging.getLogger("fgdc.map")


@dataclass
class MappedRecord:
    values: dict[str, Any] = field(default_factory=dict)
    scores: dict[str, float] = field(default_factory=dict)
    rejected: bool = False
    reason: Optional[str] = None


class MappingEngine:
    SECURE_PARSER = etree.XMLParser(
        resolve_entities=False, no_network=True, huge_tree=False
    )

    def __init__(self, config: dict[str, Any], threshold: float = 0.7):
        self.rules = config["mapping_rules"]
        self.threshold = threshold

    def resolve(self, raw_xml: bytes) -> MappedRecord:
        rec = MappedRecord()
        try:
            tree = etree.fromstring(raw_xml, parser=self.SECURE_PARSER)
        except etree.XMLSyntaxError as exc:
            rec.rejected, rec.reason = True, f"parse_error: {exc}"
            log.warning("CSDGM parse failed: %s", exc)
            return rec

        for rule in self.rules:
            node = tree.find(rule["source"])
            value = node.text.strip() if node is not None and node.text else None
            score = 1.0 if value is not None else 0.0

            if value is None and rule["mandatory"]:
                rec.rejected = True
                rec.reason = f"missing_mandatory: {rule['target']}"
                log.info("reject %s: %s", rule["target"], rec.reason)
                return rec  # fail fast — do not emit a partial record

            if value is None and not rule["mandatory"]:
                value, score = rule.get("fallback_value"), 0.0

            rec.values[rule["target"]] = value
            rec.scores[rule["target"]] = score
        return rec

The SECURE_PARSER disables entity resolution and network access, closing the XXE vector that untrusted FGDC feeds can carry. Mandatory failures short-circuit the record so no partially mapped output reaches the validation gate. After mapping, a Pydantic model mirrors the target specification and applies type coercion before the record is accepted:

python
# requires: pydantic >=2.6
from pydantic import BaseModel, Field, ConfigDict, ValidationError
from typing import Optional


class TargetMetadata(BaseModel):
    model_config = ConfigDict(populate_by_name=True, str_strip_whitespace=True)

    dataset_title: str = Field(..., min_length=1)
    publication_date: str  # ISO 8601, already coerced by iso8601_parse
    summary: Optional[str] = None


def validate(rec: "MappedRecord") -> "MappedRecord":
    try:
        TargetMetadata(**rec.values)
    except ValidationError as exc:
        rec.rejected = True
        rec.reason = f"schema_violation: {exc.error_count()} error(s)"
    return rec

Mandatory fields use ... (Ellipsis) to enforce presence at runtime; optional fields default to None or fallback strings. This immediate validation gate prevents non-conforming records from propagating into spatial data catalogs.

Failure Modes & Fallback Routing

Non-conforming records are never discarded. Every failure type maps to one deterministic recovery action, so there are no silent drops and every rejection is reproducible from the audit trail.

Failure type Likely cause Deterministic recovery action
parse_error Malformed XML, bad encoding, truncated export Quarantine raw bytes, log XMLSyntaxError, flag for re-export
missing_mandatory Source CSDGM omits a required element Route to fallback queue with the missing target name attached
schema_violation Value fails Pydantic type/length coercion Quarantine with field-level error list; do not auto-substitute
low_confidence All fuzzy matches scored < 0.7 Hold for manual review; surface candidate synonyms in the log
xxe_blocked External entity reference in untrusted feed Reject record, raise security alert, do not retry automatically

Records that fail mandatory validation pass through a fallback router that quarantines the payload, attaches diagnostic logs, and triggers a manual review workflow. This pattern is critical when migrating legacy FGDC records toward modern standards, where historical data gaps are common — particularly when the eventual target is INSPIRE Directive Schema Compliance or an ISO 19115 conversion, both of which impose stricter cardinality than CSDGM ever required.

Compliance Reporting Output

The stage writes a machine-readable compliance report for every batch so audits never depend on re-running the pipeline. Each accepted and rejected record contributes a row carrying lineage and decision provenance:

json
{
  "record_id": "csdgm/parcels_2009.xml",
  "status": "rejected",
  "reason": "missing_mandatory: publication_date",
  "field_status": {
    "dataset_title":   { "value": "County Parcels", "confidence": 1.0 },
    "publication_date": { "value": null, "confidence": 0.0 }
  },
  "lineage": {
    "source_standard": "FGDC-STD-001-1998",
    "mapping_manifest": "metadata_mapping.yaml@a1b9f3c",
    "engine_version": "fgdc-map 2.4.0",
    "processed_at": "2026-06-25T14:02:11Z"
  }
}

The lineage block pins the exact manifest commit and engine version that produced each decision, which is what satisfies a federal compliance audit: any reviewer can reproduce the rejection from the recorded inputs. Confidence scores are retained per field so synonym-matched values (0.7–0.9) are traceable and can be promoted to strict rules once verified. The rejection log uses the same reason codes as the failure-mode table, so dashboards aggregate failures without parsing free-text messages.

CI Integration

Embed the transformation stage into continuous integration to enforce schema compliance before data publication. The job blocks merges when any mandatory field fails validation, ensuring only auditable, standards-compliant metadata reaches production catalogs.

yaml
# .github/workflows/fgdc-validate.yml
name: FGDC Metadata Validation
on:
  push:
    paths: ['data/metadata/**.xml', 'config/mapping.yaml']

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.10'
      - name: Install dependencies
        # lxml: CSDGM parsing · fiona: spatial-file metadata · pydantic: validation
        run: pip install "pydantic>=2.6" "lxml>=5.1" "fiona>=1.9" "PyYAML>=6.0"
      - name: Run schema validation
        run: python -m pipeline.validate data/metadata/ config/mapping.yaml
      - name: Upload compliance report
        if: always()                       # publish the report even on failure
        uses: actions/upload-artifact@v4
        with:
          name: metadata-audit-report
          path: reports/compliance_*.json

A matching pre-commit hook running the same pipeline.validate entry point catches missing mandatory elements before they ever reach the remote, keeping the audit trail clean. The same gate pattern is reused across INSPIRE Directive Schema Compliance and Cross-Platform Schema Translation, so a single CI convention covers every standard your catalog ingests.

Where CSDGM and Modern Catalog Schemas Do Not Line Up

Most of the mapping is unremarkable: a title is a title, an abstract is an abstract. The work concentrates in a handful of places where CSDGM encodes something a modern catalog models differently, or does not model at all, and each of them needs a declared decision rather than a best effort at run time.

Dates that are not dates. CSDGM permits unknown, Unpublished material, and a range expressed as two elements, in a field whose modern counterpart expects a single ISO 8601 value. Coercing unknown to a null loses the distinction between “we do not know when this was published” and “this field was never filled in” — which matters, because only the second is a data-entry defect the source can fix. Map the CSDGM sentinel values to an explicit vocabulary in the target rather than to null, and let the date normalization rules handle only the values that genuinely are dates.

Free-text spatial reference. A large share of legacy CSDGM records describe their coordinate system in prose — “State Plane, Ohio South, feet” — in the Spatial_Reference_Information block, with no EPSG code anywhere. Resolving that text to an authority code is a lookup with a confidence score, not a parse, and it is one of the few places where a fuzzy match is defensible. The resolved code must then be validated against the actual coordinate values, because “feet” in an Ohio record means US survey feet often enough that assuming international feet introduces the error described in detecting US survey foot vs international foot errors.

Contacts as blocks, not references. CSDGM repeats the full contact block wherever a contact is needed, so one agency appears dozens of times with small variations in spelling, punctuation and job title. Modern catalogs reference a contact entity. Flattening the repeated blocks into references is a deduplication problem, and it should be treated as one — declared match rules, a survivor precedence, and a recorded merge — rather than as a string comparison inside the mapper. The discipline is the same as in duplicate detection and feature deduplication.

Entity and attribute descriptions. The Entity_and_Attribute_Information block is where CSDGM records carry their data dictionary — field names, definitions, and enumerated domains — in a structure most catalogs have no place for. It is also the most valuable content in an old record, because it is frequently the only surviving documentation of what a legacy column means. Extract it into the spatial data dictionary rather than discarding it because the target schema has no field for it.

Nested Metadata_Reference_Information. The elements describing the metadata about the metadata — when the record was last reviewed, by whom, against which standard — are routinely dropped in migration because they are not part of the dataset description. They are, however, exactly what an audit asks for, and they belong in the lineage record described above rather than in the catalog entry.

Frequently Asked Questions

Is CSDGM still worth mapping, or should legacy records be re-authored in ISO 19115? Mapping, in almost every case. Re-authoring means a person reading each old record and typing it into a new form, which is expensive and introduces transcription errors into records nobody has otherwise touched in fifteen years. A deterministic mapping preserves the original as evidence, produces the modern record automatically, and — importantly — can be re-run when the mapping improves. Keep the CSDGM source; treat the ISO record as derived output, exactly as converting CSDGM to ISO 19115 does.

How high should the fuzzy-match confidence threshold be? High enough that an accepted match needs no review, which in practice means about 0.9 for anything that affects meaning and 0.7 for cosmetic fields such as a contact’s job title. The important design point is not the number but the middle band: matches between the two thresholds go to a review queue with the candidate synonyms attached, and every promotion out of that queue becomes a hard-coded synonym in the manifest. Over a few batches the queue empties, because a vocabulary that repeats is a vocabulary you can enumerate.

What should happen when a mandatory element is missing from a source that will never be corrected? Record the gap rather than filling it. Agencies that no longer exist cannot supply a missing publication date, and inventing one — even a defensible one such as the file’s modification time — puts a fabricated fact into a catalog where nothing distinguishes it from a real one. Publish the record with the element absent and the reason recorded, or hold it in quarantine if the target schema forbids the absence; both are honest, and a fabricated date is not.

Does the mapping manifest need to track CSDGM’s own version? Yes, and it is a one-line field that saves a long argument later. FGDC-STD-001-1998 is the common baseline, but agency profiles extend it, and a record written against a biological or shoreline profile carries elements the base standard does not define. Recording the profile per source lets the mapper apply the right element set and, when an unmapped element appears, tells you immediately whether it is a profile extension or a defect.

Deeper Implementation Guides