Geospatial Schema Standardization & Mapping Automation

Handling Field Name Truncation When Writing Shapefiles

The DBF header allows field names of ten characters. That limit is older than the web, it is not going to change, and shapefiles are still what a great many recipients ask for. When a modern schema with names like ownership_status and owner_reference meets it, GDAL does the accommodating thing: it truncates to ten characters, notices that two fields now collide, and renames one of them to something like ownership_1. No exception is raised. The recipient receives a file whose columns mean something slightly different from what the schema says, and the mapping between the two exists only in the head of whoever ran the export. This procedure replaces that with a declared map, implementing a translation rule inside cross-platform schema translation, part of Geospatial Schema Architecture & Standards Mapping.

The steps map to configure (Steps 1–2, the failure and the map), execute (Step 3, collision detection), validate (Step 4, the shipped crosswalk), and log (Step 5, the CI assertion).

Prerequisites checklist

Step 1: Reproduce what the driver does unaided

python
# geopandas >= 0.14, fiona >= 1.9 — Python 3.10+
import geopandas as gpd
from shapely.geometry import Point

gdf = gpd.GeoDataFrame(
    {
        "ownership_status": ["public"],
        "ownership_reference": ["REF-1"],
        "land_use_code": ["1_1_1"],
        "geometry": [Point(500000, 5600000)],
    },
    crs="EPSG:25832",
)
gdf.to_file("out/parcels.shp")

print(list(gpd.read_file("out/parcels.shp").columns))
# ['ownership', 'ownershi_1', 'land_use_c', 'geometry']

Three things happened, none of them announced. ownership_status became ownership. ownership_reference would have become the same thing, so it became ownershi_1. And land_use_code became land_use_c, which is merely ugly rather than ambiguous. A recipient looking at ownership and ownershi_1 has no way to tell which is the status and which is the reference; a fifty-fifty guess is being asked of them, and about half the time the guess is wrong.

Step 2: Declare the abbreviation map

yaml
# translation_manifest.yaml — the shapefile profile of the target schema
formats:
  esri_shapefile:
    name_limit: 10                # MANDATORY: the DBF header limit
    encoding: "UTF-8"             # MANDATORY: written to the .cpg sidecar
    field_names:                  # MANDATORY: every field that exceeds the limit
      ownership_status:    "own_status"
      ownership_reference: "own_ref"
      land_use_code:       "landuse"
      last_survey_date:    "survey_dt"
      building_height_m:   "bldg_ht_m"
    unmapped_policy: reject       # MANDATORY: reject | truncate

Abbreviations chosen by a person beat abbreviations chosen by a prefix rule, because a person knows which part of the name carries the meaning. own_status and own_ref are instantly readable; ownership and ownershi_1 are not. Keeping them in the manifest also means the mapping is reviewed once and then stable — the same field gets the same short name in every export, forever, which matters enormously to recipients whose own scripts depend on it.

unmapped_policy: reject is the rule that keeps the map complete. A field added to the schema next year and not added to the map fails the export rather than being silently truncated, which is the whole point.

Step 3: Detect collisions before writing anything

python
# translate/shapefile_names.py — Python 3.10+
import logging

logger = logging.getLogger("translate.shapefile")


def resolve_names(columns: list[str], profile: dict) -> dict[str, str]:
    limit = profile["name_limit"]
    mapping = dict(profile.get("field_names", {}))
    resolved: dict[str, str] = {}

    for column in columns:
        if column == "geometry":
            continue
        short = mapping.get(column, column)
        if len(short) > limit:
            if profile.get("unmapped_policy", "reject") == "reject":
                raise ValueError(
                    f"'{column}' is {len(column)} characters and has no entry in "
                    f"field_names; add a reviewed abbreviation rather than truncating."
                )
            short = column[:limit]
        resolved[column] = short

    collisions: dict[str, list[str]] = {}
    for long_name, short in resolved.items():
        collisions.setdefault(short.lower(), []).append(long_name)
    clashing = {k: v for k, v in collisions.items() if len(v) > 1}
    if clashing:
        raise ValueError(f"abbreviations collide: {clashing} — fix the map, do not let GDAL rename")

    logger.info("shapefile field map: %s", resolved)
    return resolved

Note that the collision test is case-insensitive. DBF field names are conventionally upper-cased by some writers and lower-cased by others, so own_ref and OWN_REF are the same field to a recipient even when they are two keys in a Python dictionary — and a collision that only appears after the driver upper-cases everything is the hardest kind to diagnose.

Driver-Chosen Names Versus Declared Abbreviations The left column lists the schema's long field names: ownership_status, ownership_reference and land_use_code. The middle column shows what GDAL writes unaided: ownership, ownershi_1 and land_use_c, with a note that the first two are indistinguishable to a recipient. The right column shows what the declared map writes: own_status, own_ref and landuse, with a note that a crosswalk file ships alongside so the original names are recoverable. Both files are valid shapefiles; only one is interpretable. Schema field name ownership_status ownership_reference land_use_code Written unaided by the driver ownership ownershi_1 land_use_c which of the first two is the reference? no warning, no crosswalk, no way to tell Written from the declared map own_status own_ref landuse readable, stable across every export crosswalk ships beside the .shp

Step 4: Ship the crosswalk with the export

python
# translate/shapefile_names.py — geopandas >=0.14 — Python 3.10+
import csv
from pathlib import Path


def write_shapefile(gdf, path: Path, profile: dict) -> None:
    mapping = resolve_names(list(gdf.columns), profile)
    renamed = gdf.rename(columns=mapping)
    renamed.to_file(path, driver="ESRI Shapefile", encoding=profile["encoding"])

    # The .cpg sidecar tells the reader the encoding; without it, non-ASCII values
    # are decoded as latin-1 by most clients and every accented name is mangled.
    path.with_suffix(".cpg").write_text(profile["encoding"], encoding="ascii")

    crosswalk = path.with_name(path.stem + "_fields.csv")
    with crosswalk.open("w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(["shapefile_field", "schema_field"])
        for long_name, short in sorted(mapping.items(), key=lambda kv: kv[1]):
            writer.writerow([short, long_name])

The crosswalk is four lines of code and it is the difference between an export a recipient can use and one they have to ask about. Ship it in the same zip as the .shp, .dbf, .shx, .prj and .cpg. The .cpg file deserves its own mention: without it the encoding is a guess, and the guess is usually wrong for any dataset containing a street name with an accent.

Step 5: Gate the written header, not the intent

python
# tests/test_shapefile_names.py — pytest >=7, fiona >=1.9
import fiona
import pytest

from translate.shapefile_names import resolve_names, write_shapefile

PROFILE = load_manifest("translation_manifest.yaml")["formats"]["esri_shapefile"]


def test_written_header_matches_the_declared_map(tmp_path, sample_gdf):
    out = tmp_path / "parcels.shp"
    write_shapefile(sample_gdf, out, PROFILE)
    with fiona.open(out) as source:
        written = set(source.schema["properties"])
    assert written == set(PROFILE["field_names"].values()) | {"parcel_id"}


def test_unmapped_long_name_is_rejected(sample_gdf):
    # Negative control: a new long field must fail the export, not be truncated.
    columns = list(sample_gdf.columns) + ["conservation_area_reference"]
    with pytest.raises(ValueError, match="no entry in field_names"):
        resolve_names(columns, PROFILE)


def test_colliding_abbreviations_are_rejected():
    profile = dict(PROFILE, field_names={"owner_a": "own_ref", "owner_b": "OWN_REF"})
    with pytest.raises(ValueError, match="collide"):
        resolve_names(["owner_a", "owner_b"], profile)

Asserting against the written header rather than against the mapping dictionary is what makes this test meaningful: it is the only way to catch a driver that upper-cases names, trims differently than expected, or applies its own rule despite the rename.

Verification

bash
# GDAL >= 3.8 — read the DBF header of the exported file
ogrinfo -so out/parcels.shp parcels | sed -n '/Geometry Column/,$p'
# own_status: String (10.0)
# own_ref: String (10.0)
# landuse: String (10.0)

# The crosswalk that ships beside it
cat out/parcels_fields.csv
# shapefile_field,schema_field
# landuse,land_use_code
# own_ref,ownership_reference
# own_status,ownership_status

Troubleshooting

Symptom Likely cause Fix
Fields named FIELD_1, FIELD_2 in the output Names collided after truncation and GDAL renamed them Add explicit abbreviations; the collision test should have caught it before writing
Accented values arrive mangled No .cpg sidecar, so the reader guessed the encoding Write the .cpg file, as in Step 4
A date column arrives as text The DBF date type has no time component, so writers demote datetimes Split date and time into separate fields, or export a format that carries the type
Numeric values silently rounded DBF numeric fields have a fixed width and precision Declare width and precision in the profile, or export a format without the limit
The same field is abbreviated differently in two exports Two copies of the map, or a fallback truncation path There must be one manifest; set unmapped_policy: reject so no fallback exists
The recipient asks what a column means The crosswalk was not shipped Include it in the zip; it is four lines of code and saves a support cycle every time