Extract FGDC Metadata from Legacy Shapefiles Automatically Jump to heading#

Agency archives hold tens of thousands of shapefiles whose only surviving documentation lives in three sidecar files: an ArcGIS .shp.xml metadata blob, a .prj well-known-text projection string, and the .dbf attribute table itself. Re-keying that into a compliant record by hand does not scale, and the manual result is inconsistent across analysts. This procedure sits inside FGDC Metadata Mapping and owns the harvest step that feeds it: it reads whatever documentation each shapefile carries, derives the rest deterministically, and emits a structured FGDC Content Standard for Digital Geospatial Metadata (CSDGM) record plus a completeness score. Where the parent stage maps CSDGM elements onto a target catalog schema, this page produces the CSDGM record in the first place, from files that were never designed to be machine-read together.

The walkthrough maps 1:1 to the phases this site standardizes on — configure (Step 1, discover sources and load defaults), execute (Steps 2–3, harvest and gap-fill), validate (Step 4, assemble and score), and log (Step 5, batch report and CI gate). The governing rule is determinism: a value is either harvested from a real file or filled from an explicit, version-controlled default. The pipeline never invents an abstract, a date, or a bounding box — an unfillable mandatory element lowers the completeness score and is surfaced for review, exactly as the parent stage’s ISO 19115 conversion expects.

Prerequisites checklist Jump to heading#

Confirm the environment and the archive layout before the first batch. Legacy shapefile sets are inconsistent — some carry .shp.xml, some only .prj, some neither — so the pipeline must degrade gracefully rather than crash on the first bare file.

• fiona >= 1.9 and pyproj >= 3.6 installed against the same GDAL/PROJ build (python -c "import fiona; print(fiona.__gdal_version__)").
• lxml >= 5.1 for parsing the ArcGIS .shp.xml sidecar and emitting CSDGM XML.
• PyYAML >= 6.0 for the deterministic defaults manifest.
• Read access to the full shapefile set — .shp, .dbf, .prj, and any .shp.xml — with encodings noted (legacy DBF is often CP1252, not UTF-8).
• A version-controlled defaults manifest declaring fallback values for mandatory CSDGM elements that no sidecar can supply (contact org, access constraints, distribution).
• A writable path for the emitted CSDGM records and the batch completeness report.

Step 1: Discover sources and load defaults Jump to heading#

Enumerate shapefile sets by their .shp stem and probe for each optional sidecar so a missing file lowers a score rather than raising. Load the defaults manifest once; it supplies the mandatory CSDGM elements — such as metc (metadata contact) and accconst (access constraints) — that live in organizational policy, not in any shapefile.

# requires: python >=3.10, PyYAML >=6.0
from dataclasses import dataclass
from pathlib import Path
import yaml


@dataclass
class ShapefileSet:
    stem: str
    shp: Path
    dbf: Path
    prj: Path | None
    shp_xml: Path | None


def discover(root: Path) -> list[ShapefileSet]:
    sets = []
    for shp in root.rglob("*.shp"):
        sets.append(ShapefileSet(
            stem=shp.stem,
            shp=shp,
            dbf=shp.with_suffix(".dbf"),
            prj=shp.with_suffix(".prj") if shp.with_suffix(".prj").exists() else None,
            # ArcGIS writes metadata as "<name>.shp.xml", not "<name>.xml".
            shp_xml=Path(f"{shp}.xml") if Path(f"{shp}.xml").exists() else None,
        ))
    return sets


def load_defaults(path: Path) -> dict:
    with path.open(encoding="utf-8") as f:
        return yaml.safe_load(f)  # org contact, access constraints, distribution policy

Configuration rules:

• Key on the .shp stem; the ArcGIS sidecar is <name>.shp.xml, a common path bug.
• Treat every sidecar as optional so a bare shapefile still produces a partial, scored record.
• Keep the defaults manifest in version control; the report pins its commit for auditability.
• Never let a default masquerade as harvested data — tag its provenance in Step 3.

Step 2: Harvest sidecar, projection, and field metadata Jump to heading#

Harvest each source with the right tool: lxml for the .shp.xml (disabling external entities against XXE, the same guard the parent stage uses), pyproj to resolve the .prj WKT to an authoritative EPSG code, and fiona to read the DBF field names and types that populate CSDGM’s Entity and Attribute section. Resolving the .prj to an EPSG code — rather than copying raw WKT — is what makes the spatial reference block interoperable; that canonicalization is owned by CRS Normalization & Sync.

# requires: fiona >=1.9, pyproj >=3.6, lxml >=5.1  (Python 3.10+)
from pathlib import Path
import fiona
from lxml import etree
from pyproj import CRS
from pyproj.exceptions import CRSError

_PARSER = etree.XMLParser(resolve_entities=False, no_network=True, huge_tree=False)


def harvest_sidecar(shp_xml: Path | None) -> dict:
    if shp_xml is None:
        return {}
    tree = etree.parse(str(shp_xml), parser=_PARSER)
    def text(xpath: str) -> str | None:
        node = tree.find(xpath)
        return node.text.strip() if node is not None and node.text else None
    return {
        "title": text("idinfo/citation/citeinfo/title"),
        "abstract": text("idinfo/descript/abstract"),
        "pubdate": text("idinfo/citation/citeinfo/pubdate"),
    }


def harvest_crs(prj: Path | None) -> str | None:
    if prj is None:
        return None
    try:
        crs = CRS.from_wkt(prj.read_text(encoding="utf-8"))
        epsg = crs.to_epsg()
        return f"EPSG:{epsg}" if epsg else None  # unresolved WKT → gap, not a guess
    except CRSError:
        return None


def harvest_fields(dbf_shp: Path) -> list[dict]:
    with fiona.open(dbf_shp) as src:
        # schema["properties"] maps DBF field name -> "type:width" descriptor
        return [{"name": n, "type": t} for n, t in src.schema["properties"].items()]

Harvest rules:

• Parse the sidecar with entity resolution and network access disabled; government archives carry untrusted XML.
• Resolve .prj to an EPSG code; retain a None (a real gap) rather than embedding unresolved WKT.
• Read DBF field definitions with fiona, which surfaces the declared type and width per attribute.
• Decode DBF text with the archive’s actual encoding; a mojibake title is worse than an absent one.

Step 3: Fill gaps deterministically Jump to heading#

Merge the three harvest results, then backfill absent mandatory elements from the defaults manifest and from values the pipeline can compute — the bounding box from the geometry extent, the publication date from the file mtime only if policy permits. Every filled value carries a provenance tag so an auditor can tell a harvested title from a defaulted one.

# requires: fiona >=1.9  (Python 3.10+)
import fiona


def assemble_fields(harvested: dict, defaults: dict, dbf_shp) -> dict:
    record: dict[str, dict] = {}

    def put(key: str, value, source: str):
        record[key] = {"value": value, "source": source}

    for key in ("title", "abstract", "pubdate"):
        if harvested.get(key) is not None:
            put(key, harvested[key], "harvested")
        elif key in defaults:
            put(key, defaults[key], "default")   # explicit fallback, tagged
        else:
            put(key, None, "gap")                # unfillable: lowers the score

    # Bounding box is always computable from geometry — never a gap.
    with fiona.open(dbf_shp) as src:
        put("bbox", src.bounds, "computed")       # (minx, miny, maxx, maxy)
    return record

Gap-filling rules:

• Fill only from the defaults manifest or a computable value; never from a heuristic guess.
• Tag provenance (harvested, default, computed, gap) on every element for audit.
• Compute the bounding box from the geometry extent so the spatial-domain block is always present.
• Leave true gaps as None; they must depress the completeness score, not be masked.

Step 4: Assemble and score the CSDGM record Jump to heading#

Emit the CSDGM XML and compute a completeness score: the fraction of mandatory elements that resolved to a non-gap value. The score is the routing signal — a full record proceeds, a low one goes to review. This is the same completeness gate the parent stage applies, so a record scored here drops straight into FGDC Metadata Mapping without re-derivation.

# requires: lxml >=5.1  (Python 3.10+)
from lxml import etree

MANDATORY = ("title", "abstract", "pubdate", "bbox")


def score(record: dict) -> float:
    present = sum(1 for k in MANDATORY if record[k]["source"] != "gap")
    return round(present / len(MANDATORY), 3)


def to_csdgm(record: dict) -> bytes:
    root = etree.Element("metadata")
    idinfo = etree.SubElement(root, "idinfo")
    citeinfo = etree.SubElement(etree.SubElement(idinfo, "citation"), "citeinfo")
    etree.SubElement(citeinfo, "title").text = record["title"]["value"] or ""
    etree.SubElement(citeinfo, "pubdate").text = record["pubdate"]["value"] or "unknown"
    etree.SubElement(etree.SubElement(idinfo, "descript"), "abstract").text = (
        record["abstract"]["value"] or ""
    )
    return etree.tostring(root, pretty_print=True, xml_declaration=True, encoding="UTF-8")

Assembly rules:

• Score against the mandatory CSDGM element set only; optional gaps do not fail a record.
• Serialize with an explicit UTF-8 declaration so downstream lxml validation reads it cleanly.
• Route records below the configured threshold (for example 0.75) to the review queue.
• Keep the provenance map beside the XML so the report can show why a score is what it is.

Step 5: Batch-report and gate in CI Jump to heading#

Aggregate per-file scores into one report and fail the run if mandatory coverage across the batch falls below policy. Because a CSDGM record is the raw material for ISO 19115 lineage, feed this output into generating ISO 19115 lineage statements automatically, which consumes the provenance tags to record exactly how each element was derived.

# .github/workflows/fgdc-extract.yml
name: FGDC Extraction Batch
on: [pull_request]
jobs:
  extract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install "fiona>=1.9" "pyproj>=3.6" "lxml>=5.1" "PyYAML>=6.0"
      - name: Extract CSDGM from shapefile archive
        run: python -m pipeline.extract_fgdc data/archive/ --defaults config/defaults.yaml --min-score 0.75
      - name: Upload extraction report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: fgdc-extraction-report
          path: reports/extraction_*.json

CI rules:

• Fail the run when batch mandatory coverage drops below the configured floor.
• Emit the report even on failure so the review queue always has an actionable list.
• Pin the defaults-manifest commit in the report; a filled value is only auditable against a known policy.
• Wrap transient archive-read failures (locked file, mount hiccup) in retry logic, not a hard fail.

Verification Jump to heading#

Confirm the extractor actually distinguishes a documented shapefile from a bare one — a pipeline that scores everything 1.0 is silently inventing metadata. Assert on a known-good and a sidecar-less fixture:

rich = assemble_fields(harvest_sidecar(fx.shp_xml), defaults, fx.dbf)
assert rich["title"]["source"] == "harvested", "sidecar title not picked up"
assert score(rich) >= 0.75, f"documented fixture scored low: {score(rich)}"

bare = assemble_fields({}, {}, bare_fx.dbf)      # no sidecar, no defaults
assert bare["title"]["source"] == "gap", "invented a title from nothing"
assert bare["bbox"]["source"] == "computed", "bbox must always be derivable"
assert score(bare) < 0.75, "bare shapefile must not pass the completeness gate"

A healthy batch prints no CRSError for files that carry a valid .prj, and no failed to load external entity from the sidecar parser. On the CLI, fio info data/archive/parcels.shp --crs should agree with the harvested EPSG code, and xmllint --noout reports/records/parcels.xml should confirm the emitted CSDGM is well-formed before it reaches the mapping stage.

Troubleshooting Jump to heading#

Related Jump to heading#

• FGDC Metadata Mapping — the parent stage that maps the CSDGM records harvested here onto a target catalog schema
• Converting FGDC CSDGM to ISO 19115 automatically — translating the extracted CSDGM into ISO 19115 elements
• Generating ISO 19115 lineage statements automatically — consuming the provenance tags to record how each element was derived
• CRS Normalization & Sync — resolving legacy .prj well-known text to canonical EPSG codes
• Local Government Data Dictionaries — reconciling the DBF field names harvested here against a canonical vocabulary