Skip to content

fix(ci): cache skill security scan results to avoid double LLM scans - #908

Merged
danbarr merged 1 commit into
mainfrom
cache-skill-security-scan-results
Aug 25, 2026
Merged

fix(ci): cache skill security scan results to avoid double LLM scans#908
danbarr merged 1 commit into
mainfrom
cache-skill-security-scan-results

Conversation

@danbarr

@danbarr danbarr commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • When Renovate opens a PR bumping a skill, skill-version-check's autofix job pushes a follow-up commit that only touches spec.version. That retriggers build-skills.yml and re-runs the LLM-based security scan on source identical to what was already scanned, doubling OpenRouter cost per affected PR.
  • Caches the raw scan output (skill-scan-<skill>.json, scanner-version.txt) in skill-security-scan, keyed on repository/ref/skill_path (what the scanner actually reads) plus a hash of scripts/skill-scan/requirements.txt. Check out skill source and Run skill security scan are skipped on a cache hit.
  • The allowlist evaluation (process_scan_results.py, driven by security.allowed_issues in spec.yaml) is split into its own step that always runs, cache hit or not, against the possibly-cached raw scan output. This matters because adding/editing an allowlist entry is a common follow-up commit that doesn't change repository/ref/path — it must still get a fresh blocking decision, not a stale cached one.
  • build-skill-artifacts's pre-flight scan gate still reads scan-summary.json from the uploaded artifact either way, so publish is still blocked on unallowlisted findings.

Test plan

  • Confirm skill-security-scan restores from cache (and skips the LLM scan step) on the second pull_request synchronize run after a spec.version-only autofix commit — verified via this PR's smoke test: first run was a cache miss (full ~9min scan), re-running the same job hit the cache and skipped Check out skill source / Run skill security scan entirely, finishing in 31s
  • Confirm a PR that only adds an allowed_issues entry still gets a fresh, correct pass/block decision even when the raw scan is served from cache
  • Confirm a cache miss (new repository/ref/path) still runs the scanner normally — verified on the first run of this PR's smoke test
  • Confirm build-skill-artifacts's pre-flight scan gate still blocks publish on a blocking finding — gate passed on both runs, reading the fresh scan-summary.json each time

🤖 Generated with Claude Code

When Renovate opens a PR bumping a skill, skill-version-check's autofix
job pushes a follow-up commit that only touches spec.version. That
retriggers build-skills.yml and re-runs the LLM-based security scan on
source that's identical to what was already scanned, doubling cost.

Cache the scan result keyed on repository/ref/path (what the scanner
actually reads) plus the scanner requirements hash, so a version-only
follow-up commit reuses the prior verdict instead of paying for another
scan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@toolhive-release-app

Copy link
Copy Markdown
Contributor

🛡️ Skill Security Scan Results

✅ agent-observability-eval-bootstrap

  • Status: Passed
  • Findings: 5
  • Allowed (not blocking): 2
    • LLM_PROMPT_INJECTION (Allowed: FP: same as policy_violation above — all pattern matches are on documentation, code examples, or attack pattern descriptions for detection purposes, not agent instructions.)
    • MANIFEST_MISSING_LICENSE (Allowed: datadog-labs/agent-skills is licensed MIT at the repository root; upstream does not embed an SPDX license identifier in per-skill SKILL.md frontmatter.)

Summary: Scanned 1 skill(s), all passed security checks. ✅

@danbarr
danbarr merged commit 11de8be into main Aug 25, 2026
14 checks passed
@danbarr
danbarr deleted the cache-skill-security-scan-results branch August 25, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants