Publishing a Schema Conformance Scorecard in GitHub Actions Jump to heading#

This guide builds the concrete GitHub Actions workflow that turns a pipeline run into a visible, reviewable conformance record: it runs the validation suite, computes the scorecard, renders a Markdown summary into the job summary, commits an SVG badge, and posts a sticky pull-request comment. It is the deployment layer for CI Validation Scorecards, which defines the metrics and thresholds; here we wire those into an actual pull-request check. The scorecard engine, threshold manifest, and emit functions referenced below are the ones specified on that page — this guide assumes they already exist as scorecard/ modules in your repository.

Prerequisites checklist Jump to heading#

Confirm each of these before writing the workflow. The badge-commit and comment steps need write permissions that a default GITHUB_TOKEN does not grant, and the scorecard is meaningless if the gates never wrote their manifests.

• pyarrow >=14 and the scorecard/ package (engine, emit, gate) present in the repo and importable.
• geopandas >=0.14 and pyproj >=3.6 installed against the same PROJ build for the validation gates.
• The validation suite writes per-gate lineage manifests into a known run directory (for example runs/${GITHUB_SHA}/).
• scorecard_thresholds.yaml committed at the repo root and passing Pydantic validation.
• Workflow permissions will include contents: write (badge commit) and pull-requests: write (comment).
• A dedicated badges branch exists, or the workflow is allowed to create it, to hold the committed SVG without polluting history on the main branch.

Step 1: Scaffold the workflow and permissions Jump to heading#

Create .github/workflows/scorecard.yml. Scope it to pull requests and pushes to the default branch, and declare the two write scopes the later steps depend on. Set concurrency so a rapid succession of pushes to the same pull request cancels superseded runs rather than racing to commit the badge.

# .github/workflows/scorecard.yml
name: Conformance Scorecard
on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: write          # commit the SVG badge
  pull-requests: write     # post the scorecard comment

concurrency:
  group: scorecard-${{ github.ref }}
  cancel-in-progress: true

jobs:
  scorecard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

Step 2: Run the validation suite Jump to heading#

Install dependencies, sync the PROJ grids the CRS gate needs, and run the gates so their manifests land in the run directory. Use --tb=short and let a gate failure surface, but do not let it abort the job before the scorecard is computed — the scorecard must report on a failed run, not vanish with it. continue-on-error: true on the suite step preserves the failure for the scorecard to grade while keeping the job alive.

      - name: Install dependencies
        run: pip install "geopandas>=0.14" "pyproj>=3.6" "pyarrow>=14" "pydantic>=2.0" "pyyaml>=6.0" "pytest>=7"
      - name: Sync required PROJ grids
        run: pyproj sync --source-id us_noaa --source-id ca_nrc
      - name: Run validation suite
        id: validate
        continue-on-error: true
        run: |
          mkdir -p "runs/${GITHUB_SHA}"
          pytest tests/ -v --tb=short --run-dir "runs/${GITHUB_SHA}"

Step 3: Compute and render the scorecard Jump to heading#

Call the engine to reduce the manifests, then render all three artifacts from the single Scorecard object so the JSON, Markdown, and badge can never disagree. This CLI wrapper reads the threshold manifest, computes the scorecard, evaluates the gate, and writes the artifacts to scorecard-out/. Critically, it exits 0 even when the scorecard fails its thresholds — the exit code that blocks the merge is applied in a later, explicit step, so the comment and badge always publish first.

# scripts/run_scorecard.py  — pyarrow >=14, pyyaml >=6.0
import os
import sys
from pathlib import Path

import yaml

from scorecard.engine import compute_scorecard
from scorecard.emit import emit_json, emit_markdown, emit_badge
from scorecard.gate import evaluate
from schema_drift import count_drift            # see the schema-drift guide

RUN_ID = os.environ["GITHUB_SHA"]
RUN_DIR = Path("runs") / RUN_ID
OUT = Path("scorecard-out")
OUT.mkdir(exist_ok=True)

thresholds = yaml.safe_load(Path("scorecard_thresholds.yaml").read_text())["scorecard"]
gates = ["crs_normalization", "attribute_etl", "schema_compliance"]

drift = count_drift(current="schema/current.json", golden="schema/golden.json")
sc = compute_scorecard(
    run_id=RUN_ID,
    run_dir=RUN_DIR,
    gates=gates,
    min_features_for_rates=thresholds["min_features_for_rates"],
    schema_drift_count=drift,
)
passed = evaluate(sc, thresholds)

emit_json(sc, OUT / "scorecard.json")
emit_markdown(sc, passed, OUT / "scorecard.md")
emit_badge(sc, passed, OUT / "conformance.svg")

# Record the verdict for a later gating step; do not fail here.
Path(OUT / "verdict").write_text("pass" if passed else "fail")
print(f"scorecard computed for {RUN_ID}: {'PASS' if passed else 'FAIL'}")
sys.exit(0)

      - name: Compute scorecard
        run: python scripts/run_scorecard.py

Step 4: Write the job summary and commit the badge Jump to heading#

Append the rendered Markdown to $GITHUB_STEP_SUMMARY so the scorecard appears on the run’s summary page with no artifact download. Then commit the SVG to the dedicated badges branch using a worktree, which keeps the badge history isolated from your main branch. Guard the commit so it only runs when the badge actually changed, avoiding an empty commit on every run.

      - name: Write job summary
        run: cat scorecard-out/scorecard.md >> "$GITHUB_STEP_SUMMARY"

      - name: Commit badge to badges branch
        if: github.event_name == 'push'
        run: |
          git config user.name  "scorecard-bot"
          git config user.email "[email protected]"
          git fetch origin badges || git branch badges
          git worktree add ../badges badges 2>/dev/null || git worktree add -b badges ../badges
          cp scorecard-out/conformance.svg ../badges/conformance.svg
          cd ../badges
          if ! git diff --quiet; then
            git add conformance.svg
            git commit -m "scorecard: update conformance badge [skip ci]"
            git push origin badges
          else
            echo "badge unchanged; nothing to commit"
          fi

Reference the committed badge from your README with a raw URL to the badges branch: ![conformance](https://raw.githubusercontent.com/OWNER/REPO/badges/conformance.svg). Because it is committed rather than generated on demand, the badge renders even when the CI provider’s dynamic-badge service is down.

Step 5: Post the pull-request comment Jump to heading#

Post the scorecard as a single sticky comment that is updated in place on every push, so a busy pull request accumulates one living scorecard rather than a wall of stale ones. Find an existing bot comment by a hidden marker and update it; create one only if none exists. Use the github-script action so no external dependency is required.

      - name: Post scorecard comment
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const marker = '<!-- conformance-scorecard -->';
            const body = marker + '\n' + fs.readFileSync('scorecard-out/scorecard.md', 'utf8');
            const {owner, repo} = context.repo;
            const issue_number = context.issue.number;
            const {data: comments} = await github.rest.issues.listComments({owner, repo, issue_number});
            const existing = comments.find(c => c.body.includes(marker));
            if (existing) {
              await github.rest.issues.updateComment({owner, repo, comment_id: existing.id, body});
            } else {
              await github.rest.issues.createComment({owner, repo, issue_number, body});
            }

Finish the job with the explicit gating step. This is the only step allowed to fail the workflow, and it does so on the verdict the scorecard already computed — keeping the merge decision downstream of publication guarantees reviewers always see the scorecard even on a failing run.

      - name: Enforce conformance gate
        run: |
          verdict=$(cat scorecard-out/verdict)
          echo "scorecard verdict: $verdict"
          test "$verdict" = "pass"

Verification Jump to heading#

Confirm the workflow published the scorecard and gated correctly — do not trust a green check alone. Open a pull request that deliberately introduces one quarantined feature and confirm each surface reflects it:

# 1. The JSON artifact exists and carries the run id.
jq -e '.run_id and .total_features > 0' scorecard-out/scorecard.json

# 2. The verdict file matches the exit status of the gating step.
test "$(cat scorecard-out/verdict)" = "fail"   # for the deliberately-broken PR

# 3. The badge committed to the badges branch is valid SVG.
git show badges:conformance.svg | head -1 | grep -q "<svg"

On the pull request itself, the job summary must show the scorecard table, exactly one comment carrying the conformance-scorecard marker must be present (push twice and confirm it updates rather than duplicates), and the check must be red when verdict is fail. A healthy passing run shows a green check, a conformance rate at or above the manifest floor, and a badges branch commit only on pushes to main.

Troubleshooting Jump to heading#

Related Jump to heading#

• CI Validation Scorecards — the metric definitions, threshold manifest, and engine this workflow deploys
• Gating Pull Requests on Geospatial Schema Drift — the count_drift step that feeds schema_drift_count into this workflow
• Lineage Manifest Generation — the append-only gate manifests the compute step reduces
• Error Handling & Retry Logic — retry patterns for the transient PROJ-grid sync in Step 2