Compliance Reporting and Audit Trails as an Engineering Discipline Jump to heading#

Every geospatial pipeline that publishes into a regulated environment eventually faces the same demand: prove it. A regulator, an internal auditor, or a downstream consumer asks not whether the data looks right but whether the organization can demonstrate — from records captured at processing time — that each dataset was transformed correctly, validated against a stated contract, and published with its provenance intact. Compliance reporting is the engineering practice that produces this proof as a deterministic byproduct of the pipeline rather than as a manual scramble before an audit deadline.

This discipline turns every transform and validation event from the other three practice areas on GeoSpatial Schema into queryable, regulator-ready evidence. Where CRS normalization and attribute ETL do the work of reshaping data, and schema-standards mapping defines the target contract, this fourth practice area proves the work was done correctly. It rests on three artifacts: append-only lineage manifests that record what happened to every feature, machine-readable dataset metadata (schema.org Dataset and DCAT-AP JSON-LD) that publishes provenance to open-data portals, and continuous-integration scorecards that gate merges on schema conformance before a regression can ever reach production. The primary drivers are statutory: government and open-data teams operating under the INSPIRE metadata regulation, FGDC records obligations, and ISO 19115 provenance and data-quality requirements cannot certify a dataset without this evidence, regardless of how accurate the geometry is.

“Deterministic” is again the operative word, carried forward from the transformation pillars. An evidence pipeline is deterministic when the same processing run over the same inputs produces the same manifest, the same derived metadata, and the same scorecard verdict. Non-determinism here is corrosive in a specific way: if the evidence can vary run to run, it stops being evidence. The three failure channels this discipline closes are manifests that lose events under load, metadata that drifts out of sync with the data it describes, and scorecards whose pass/fail threshold is a matter of opinion rather than a published contract. Each section below is engineered to close one of these channels — write the manifest append-only so events cannot be lost or rewritten, derive metadata mechanically so it cannot drift, and pin the scorecard thresholds so the verdict is reproducible.

The Evidence Pipeline Jump to heading#

The diagram below shows the four-stage evidence pipeline. Transform and validation events flow in from the upstream practice areas on the left; the pipeline appends them to an immutable lineage manifest, derives dataset metadata, publishes machine-readable JSON-LD, and computes a CI scorecard that gates the next merge.

Stage 1 — Capture and append. Every reprojection, cast, flatten, and rejection emits a structured event. The pipeline appends each event as an immutable row to a partitioned Parquet manifest, keyed by run_id and dataset_version. This stage owns the lineage manifest generation practice: it defines the event schema, the write path, and the append-only guarantee.

Stage 2 — Derive metadata. Manifest rows for a dataset roll up into dataset-level provenance: ISO 19115 LI_Lineage process steps and DQ_DataQuality measures. Derivation is mechanical — a query over the manifest, never a hand-edited document — so the metadata cannot drift from the events it describes.

Stage 3 — Publish machine-readable metadata. The derived provenance is emitted as schema.org Dataset and DCAT-AP JSON-LD for open-data portals and harvesters. This is the dataset metadata and JSON-LD publishing practice, which also validates the emitted graph against SHACL shapes before it ships.

Stage 4 — Score and gate. A columnar scan of the manifest computes a conformance scorecard — completeness, validity, conformance rate, drift — and fails the CI job when any metric crosses its threshold. This is the CI validation scorecard publishing practice, which turns the evidence into a merge gate.

Standards Alignment Matrix Jump to heading#

The following table maps regulatory provenance and data-quality requirements to the evidence-pipeline stages. Each standard demands specific fields or documents; the stage column shows where each requirement is satisfied, and the final column links to the practice that owns the implementation.

Regulatory conformance requires the complete chain. A dataset with immaculate geometry that lacks an LI_Lineage statement still fails an INSPIRE Directive schema compliance audit, and a dataset whose published JSON-LD disagrees with its manifest fails a data-quality check even when both artifacts are internally valid. Treat the table as a contract: every row must be satisfied before a dataset receives a conformant status flag. The provenance-record obligations these standards impose on element naming, cardinality, and crosswalks between profiles are defined one layer up in Geospatial Schema Architecture & Standards Mapping; this practice area consumes those definitions and produces the evidence that they were met.

Core Pattern: Appending an Immutable Lineage Record Jump to heading#

The code below is the kernel of Stage 1. It accepts a batch of transform events, validates them against a fixed schema, and appends them to a partitioned Parquet dataset without ever mutating an existing file. The append-only guarantee is enforced structurally: each call writes a new fragment into the run_id partition, and the writer refuses to touch a partition that a prior run already sealed.

# pyarrow >=14, geopandas >=0.14
from __future__ import annotations

import logging
import uuid
from dataclasses import dataclass, asdict
from datetime import datetime, timezone
from pathlib import Path

import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq

logger = logging.getLogger(__name__)

LINEAGE_SCHEMA = pa.schema([
    pa.field("run_id",           pa.string(),  nullable=False),
    pa.field("dataset_id",       pa.string(),  nullable=False),
    pa.field("dataset_version",  pa.string(),  nullable=False),  # git SHA or ETag
    pa.field("feature_id",       pa.string(),  nullable=False),
    pa.field("event",            pa.string(),  nullable=False),  # reproject|cast|reject
    pa.field("source_crs",       pa.string(),  nullable=True),
    pa.field("target_crs",       pa.string(),  nullable=True),
    pa.field("method",           pa.string(),  nullable=True),   # e.g. NTv2:...
    pa.field("accuracy_m",       pa.float64(), nullable=True),   # None if unknown
    pa.field("conformant",       pa.bool_(),   nullable=False),
    pa.field("rejection_code",   pa.string(),  nullable=True),   # None if conformant
    pa.field("operator",         pa.string(),  nullable=True),
    pa.field("timestamp_utc",    pa.timestamp("us", tz="UTC"), nullable=False),
])


@dataclass(frozen=True)
class LineageEvent:
    dataset_id: str
    dataset_version: str
    feature_id: str
    event: str
    conformant: bool
    source_crs: str | None = None
    target_crs: str | None = None
    method: str | None = None
    accuracy_m: float | None = None
    rejection_code: str | None = None
    operator: str | None = None


class SealedPartitionError(RuntimeError):
    """Raised when a run_id partition already exists and would be mutated."""


def append_lineage(
    events: list[LineageEvent],
    manifest_root: Path,
    run_id: str | None = None,
) -> str:
    """Append *events* to the append-only lineage manifest.

    Returns the run_id under which the events were written.

    Raises
    ------
    ValueError
        An event fails schema validation.
    SealedPartitionError
        The target run_id partition already contains data.
    """
    if not events:
        raise ValueError("Refusing to write an empty lineage batch.")

    run_id = run_id or str(uuid.uuid4())
    partition_dir = manifest_root / f"run_id={run_id}"
    if partition_dir.exists() and any(partition_dir.glob("*.parquet")):
        raise SealedPartitionError(
            f"run_id={run_id} is already sealed; appends must use a new run_id."
        )

    now = datetime.now(timezone.utc)
    rows = []
    for ev in events:
        record = asdict(ev)
        record["run_id"] = run_id
        record["timestamp_utc"] = now
        if not record["conformant"] and record["rejection_code"] is None:
            raise ValueError(
                f"feature {ev.feature_id}: non-conformant event needs a rejection_code."
            )
        rows.append(record)

    table = pa.Table.from_pylist(rows, schema=LINEAGE_SCHEMA)

    # write_to_dataset with a run_id partition creates a new fragment; it never
    # rewrites an existing partition, preserving the append-only invariant.
    pq.write_to_dataset(
        table,
        root_path=str(manifest_root),
        partition_cols=["run_id", "target_crs"],
        existing_data_behavior="error",  # fail rather than overwrite
    )
    logger.info("Appended %d lineage rows under run_id=%s", len(rows), run_id)
    return run_id

Key design decisions:

• The schema is a hard contract. pa.Table.from_pylist(rows, schema=LINEAGE_SCHEMA) raises if a field is missing or mistyped, so a malformed event fails at write time rather than surfacing as a gap during an audit.
• existing_data_behavior="error" makes the append-only guarantee enforced by the storage layer, not by convention. A run that tries to overwrite a sealed partition fails loudly instead of silently replacing evidence.
• Non-conformant events are required to carry a rejection_code. This mirrors the rejection contract in the ETL pillar’s error handling and retry logic: a quarantined feature is not an absence of evidence, it is evidence of a specific, coded failure that a later query can group and trend.
• Partitioning by run_id and target_crs is chosen for the queries an auditor actually runs — “everything from this run” and “everything published in this frame” — so the dashboard answers them with a columnar scan rather than a full-table replay.

The events consumed here are precisely the records the transformation pillars already emit. The source_crs, target_crs, method, and accuracy_m fields carry directly from the lineage record introduced in the CRS pillar’s Compliance & Audit Requirements section; the rejection_code values are the same codes produced by the ETL rejection path. This practice area does not re-derive that information — it standardizes the upstream events into one auditable trail.

Validation Gates & Thresholds Jump to heading#

Evidence is only trustworthy if it is complete and self-consistent. Every reporting run passes through a sequence of gates before its artifacts are accepted; a failure at any gate blocks publication and, in CI, fails the merge. The gates are measurable and boolean — each produces a pass/fail with a logged reason, never a subjective score.

Gate 1 — Manifest Completeness

• Pass: the manifest row count for a run equals the number of features the pipeline reports as processed (manifest_rows == processed_features).
• Fail: rejection code MANIFEST_INCOMPLETE; indicates events were dropped, usually a lost batch under load or an unhandled exception that skipped the append.

Gate 2 — Schema Validity

• Pass: every manifest row conforms to LINEAGE_SCHEMA with all nullable=False fields populated.
• Fail: rejection code MANIFEST_SCHEMA_ERROR; a producer emitted a malformed event.

Gate 3 — Rejection Coding

• Pass: every row with conformant == False carries a non-null rejection_code drawn from the registered code set.
• Fail: rejection code UNCODED_REJECTION; a failure was recorded without a machine-readable cause, making it un-triageable at audit time.

Gate 4 — JSON-LD Validity

• Pass: each published dataset node parses as valid JSON-LD and passes the SHACL shapes for schema.org Dataset and DCAT-AP.
• Fail: rejection code JSONLD_INVALID; the emitted metadata graph violates a shape constraint, e.g. a missing dct:license or an unresolvable dcat:distribution.

Gate 5 — Metadata Consistency

• Pass: the derived dataset metadata agrees with the manifest — the declared conformance status, feature count, and target CRS in the JSON-LD match the manifest rollup for that dataset_version.
• Fail: rejection code METADATA_DRIFT; the published summary disagrees with the underlying evidence, the single most damaging inconsistency in an audit.

Gate 6 — Scorecard Threshold

• Pass: the run’s conformance metrics meet the published thresholds (defined below).
• Fail: rejection code SCORECARD_BELOW_THRESHOLD; conformance regressed relative to the contract.

The scorecard thresholds are pinned values, versioned alongside the pipeline so a verdict is reproducible:

A conformance rate below 100 percent is expected and healthy — it reflects features legitimately quarantined by upstream gates — but a drop in the rate is the signal. The scorecard trends each metric against the prior run’s baseline, so a sustained decline in conformance or a rise in drift is caught as a regression rather than absorbed as noise.

Compliance & Audit Requirements Jump to heading#

The lineage manifest is the evidence artifact. Its value in an audit depends entirely on three properties — immutability, retention, and reconstructability — that must be engineered in from the first write, because none can be added retroactively without destroying the guarantee they provide.

Append-only immutability. No record may be updated or deleted after writing. Corrections are expressed as new rows that supersede prior ones, carrying a supersedes reference to the earlier run_id and feature_id, never as in-place edits. This is what lets a reviewer trust that a conformant flag was set at processing time rather than back-filled before the audit. The existing_data_behavior="error" guard in the core pattern above enforces this at the storage layer; a run that would overwrite a sealed partition fails rather than silently replacing history.

Retention keyed to the governing framework. Each pipeline run writes a manifest keyed by run_id (UUID4) and dataset_version (git SHA or object-store ETag). Retention is a per-dataset policy field carried in the manifest, not a global setting: a partition may be expired only when every applicable framework permits it. INSPIRE monitoring and reporting guidance points programmes toward multi-year retention, and many national implementations require at least six years of evidence for published datasets; FGDC and agency records schedules can require longer. Carrying the policy with the data means an expiry job can reason about each partition independently rather than applying one blanket window.

Version control and reconstructability. The manifest is not the only versioned artifact — the schema contract, the scorecard thresholds, and the SHACL shapes are all committed alongside the pipeline code. This is what makes a run reconstructable: given a dataset_version, a reviewer can check out the exact schema, thresholds, and shapes that were in force and re-derive the metadata and scorecard from the immutable manifest. The published metadata is therefore never authoritative on its own; it is a derived view whose provenance is the manifest plus the versioned rules that transformed it.

The four standing metrics published per run, trended over time, are the operational surface of these requirements:

• Completeness rate — manifest rows divided by processed features; anything below 100 percent is a lost-evidence incident, not a data-quality nuance.
• Conformance rate — conformant features divided by total; a sustained drop signals an upstream supplier or schema change.
• Rejection breakdown — non-conformant rows grouped by rejection_code, so the most common cause can be triaged at its source rather than repeatedly re-observed downstream.
• Metadata freshness — the lag between the newest manifest row for a dataset and its most recently published JSON-LD, which surfaces stale portal records before a harvester does.

The ISO 19115 lineage statements derived from this manifest, and the append-only Parquet write path that produces it, are documented in the lineage-generation practice. The relationship between the raw manifest and the published dataset node is illustrated below.

Dataset Metadata & JSON-LD Publishing Jump to heading#

Once provenance is derived from the manifest, it must be published in a form that machines can harvest. Dataset Metadata & JSON-LD Publishing covers the serialization and validation of that metadata: emitting a schema.org Dataset node so general-purpose search engines and dataset-search tools can index the resource, and a DCAT-AP profile so European and national open-data portals can harvest it into their catalogues.

The two representations serve different consumers and must stay consistent with each other and with the manifest. The publishing practice enforces this with SHACL shapes rather than ad-hoc checks: emitting schema.org Dataset JSON-LD for open-data portals defines the required properties and their provenance from manifest fields, and validating DCAT-AP metadata with SHACL shapes defines the constraint graph that Gate 4 evaluates before any node ships. Key operations covered there:

• Mapping manifest fields to schema.org Dataset properties — distribution, dateModified, spatialCoverage, and a provenance link back to the lineage record.
• Populating DCAT-AP mandatory properties (dct:title, dct:description, dct:license, dcat:distribution) and validating cardinality against the DCAT-AP SHACL shapes.
• Asserting the metadata-consistency gate: the feature count and conformance status in the published node must match the manifest rollup for the same dataset_version.

Publishing conformant metadata is the point at which this practice area meets the INSPIRE Directive schema compliance obligations owned by the standards pillar: the discoverable record it requires is exactly the node produced here, and the ISO 19115 lineage it references is exactly the provenance derived at Stage 2.

CI Validation Scorecards Jump to heading#

Evidence that is only inspected during an audit is evidence collected too late. The CI Validation Scorecard Publishing practice moves the gates into continuous integration, so a regression in conformance, completeness, or schema drift fails a pull request before it can reach production. The scorecard is the operational form of the thresholds table above: a single job that scans the manifest, evaluates each metric, and reports a pass/fail verdict with a per-metric breakdown.

Two concrete integrations sit under this practice. Publishing a schema conformance scorecard in GitHub Actions covers computing the metrics in a workflow, rendering them as a job summary and a status check, and trending them against the prior run’s baseline. Gating pull requests on geospatial schema drift covers the specific case of a published field changing shape — a renamed column, a widened type, a dropped attribute — and failing the merge unless the change is an explicit, reviewed decision.

The scorecard consumes the same manifest as every other stage, which is what keeps the whole practice area coherent: the number a regulator sees in an ISO 19115 data-quality statement, the number a portal harvests from the DCAT-AP node, and the number that gates a merge are all the same number, computed once from the append-only evidence. When the batch layer in Automated Attribute Transformation & ETL Workflows reshapes ten thousand datasets in a run, the scorecard is what proves — automatically, on every merge — that the reshaping stayed within contract.

Maintenance & Regression Strategy Jump to heading#

Evidence pipelines degrade in three predictable ways: the manifest schema evolves and old and new partitions diverge, the metadata vocabularies change (schema.org and DCAT-AP revise properties, SHACL shapes tighten), and the scorecard thresholds drift from the reality of the data they gate. A regression strategy addresses all three without breaking the reconstructability guarantee.

Schema-evolution discipline. The manifest schema is versioned, and every partition records the schema_version under which it was written. New fields are added as nullable; existing fields are never repurposed or narrowed. A migration that must change a field’s meaning writes a new field and leaves the old one intact, so a query over a multi-year manifest can reason about each partition under the schema that produced it. A golden-manifest test asserts that a fixed set of input events produces a byte-stable manifest and a byte-stable derived metadata document across pipeline versions.

# Example golden-manifest assertion (pytest)
# pyarrow >=14
import pyarrow.parquet as pq
import pyarrow.compute as pc


def test_golden_manifest_rollup(tmp_path):
    events = load_golden_events()  # fixed, versioned fixture
    run_id = append_lineage(events, tmp_path, run_id="golden-0001")

    table = pq.read_table(tmp_path)
    # Completeness gate: one row per input event, none dropped.
    assert table.num_rows == len(events), "manifest lost or duplicated events"

    # Every non-conformant row must carry a rejection_code.
    non_conf = table.filter(pc.equal(table["conformant"], False))
    codes = non_conf["rejection_code"].to_pylist()
    assert all(c is not None for c in codes), "uncoded rejection in manifest"

    # Rollup is deterministic across runs.
    conformant = pc.sum(pc.cast(table["conformant"], "int64")).as_py()
    assert conformant == EXPECTED_CONFORMANT_COUNT

Vocabulary-drift alerting. Subscribe to the release notes for the vocabularies the metadata depends on — the schema.org releases and the DCAT-AP specification revisions. When either changes, re-run the SHACL validation over a corpus of previously published nodes in CI and alert on any node that newly fails, so a tightened constraint is caught before it silently rejects a production publish.

Threshold review cadence. The scorecard thresholds are a contract, and a contract that never changes eventually lies. Review the conformance-rate and drift thresholds each reporting period against the trended metrics: a conformance rate that sits far above its threshold invites tightening, and one that repeatedly grazes the floor invites investigation of the upstream cause rather than a loosened bar. Any threshold change is a reviewed commit, versioned with the pipeline, so a past run’s verdict remains reproducible under the thresholds that were actually in force.

For the transformation context that produces the events this practice area records, see Automated Attribute Transformation & ETL Workflows for the batch, casting, and retry layers, and Coordinate Reference System Normalization & Sync for the reprojection events and accuracy figures that populate the manifest.

Frequently Asked Questions Jump to heading#

What is the difference between a lineage manifest and a metadata record? A lineage manifest is a row-per-event operational log: one immutable record for every transform, validation, or rejection, keyed by run_id and feature_id. A metadata record is a dataset-level summary a regulator reads — an ISO 19115 document or a schema.org Dataset node. The manifest is the raw evidence; the metadata record is the rolled-up statement derived from it. Keeping both, and deriving the second from the first, means the summary is never hand-maintained and can always be reconstructed from the append-only log.

Why must the lineage manifest be append-only rather than a mutable table? Auditability depends on the guarantee that no prior record was altered after the fact. A mutable table cannot prove that a conformant flag was set at processing time rather than back-filled before an audit. An append-only Parquet store, partitioned by run_id and written once, gives every reviewer the same history: corrections are new rows that supersede old ones, never in-place edits, so the full sequence of what was known and when stays reconstructable.

How long must geospatial audit evidence be retained? Retention is set by the governing framework, not the pipeline. INSPIRE monitoring and reporting guidance points programmes toward multi-year retention, and many national implementations require at least six years of evidence for published datasets; FGDC and agency records schedules can require longer. Treat retention as a per-dataset policy field carried in the manifest, so a partition can be expired only when every applicable framework permits it.

Does publishing schema.org Dataset JSON-LD satisfy INSPIRE metadata obligations? No. schema.org Dataset and DCAT-AP JSON-LD make a dataset discoverable to search engines and open-data harvesters, but INSPIRE conformance is defined against the INSPIRE metadata regulation and ISO 19115 profiles. The two serve different consumers, so the practice is to derive both from one canonical provenance record: the ISO 19115 document for the statutory audit and the JSON-LD for portal discovery, kept consistent because neither is authored by hand.

What should a CI scorecard fail a merge on? A scorecard fails on measurable regressions, not on style: manifest completeness below 100 percent of processed features, any dataset node that fails JSON-LD or SHACL validation, a conformance rate below the published threshold for the reporting period, or a rise in schema drift against the pinned target schema. Each is a boolean gate with a logged reason, so a failing pull request tells the author exactly which artifact regressed rather than reporting an opaque score.

Related Jump to heading#

• Lineage Manifest Generation — Stage 1–2: the append-only event schema, Parquet write path, and ISO 19115 rollup
• Dataset Metadata & JSON-LD Publishing — Stage 3: schema.org Dataset and DCAT-AP serialization with SHACL validation
• CI Validation Scorecard Publishing — Stage 4: conformance scorecards and schema-drift gates in continuous integration
• Coordinate Reference System Normalization & Sync — the reprojection events and accuracy figures that populate the lineage manifest
• Automated Attribute Transformation & ETL Workflows — the casting and rejection events, including the error-handling and retry logic that produces coded rejections
• Geospatial Schema Architecture & Standards Mapping — the ISO 19115, INSPIRE, and FGDC contracts this evidence proves conformance against