feat: packagist transitive dependent counts

[epipav]

No description provided.

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

PR Summary

Medium Risk
Weekly full scan of ~1.5B-row package_dependencies and bulk updates to transitive_dependent_count affect shared DB load and criticality inputs; safeguards (empty-graph abort, merge guard, single-instance workflow id) limit worst-case data corruption.

Overview
Adds a fifth Packagist Temporal lane that materializes packages.transitive_dependent_count from stored direct edges (deps.dev has no Packagist data), so ranking can use the same depth≥2 transitive signal as other ecosystems.

Pipeline: snapshot packagist require edges into staging.packagist_transitive_edges → recursive reverse closure into staging.packagist_transitive_counts → keyset merge (10K batches, IS DISTINCT FROM, zero-fill leaves). Run state lives in new packagist_transitive_runs (pending → merging → done|failed), not per-purl state or osspckgs_ingest_jobs.

Orchestration: computePackagistTransitiveDependents runs prepare once (45m timeout, heartbeats), then merge rounds with continueAsNew; empty snapshot or empty counts table fail fast (non-retryable) to avoid zero-filling good data. ingestPackagistMetadata chain-starts this workflow on natural completion (skipped in STOP_AFTER_FIRST_PAGE); manual transitive trigger reuses fixed workflow id packagist-transitive-drain to prevent concurrent staging rebuilds.

DAL/worker: New transitiveDependents + packagistTransitiveRuns modules; activities wired through worker exports. findPendingJobByKind dedupes ranking ingest jobs on Temporal retry (used by rankPackages). Docs: ADR-0009 fifth-lane decision, Packagist README §5, tests (workflow/activity unit + opt-in destructive DB integration).

^{Reviewed by Cursor Bugbot for commit 0e14296. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Adds weekly Packagist transitive-dependent computation and integrates it with the packages worker.

Changes:

• Adds PostgreSQL snapshot, closure, merge, and run-ledger DAL operations.
• Adds Temporal workflow/activity orchestration, manual triggering, and metadata-drain chaining.
• Adds migrations, tests, ADR updates, and operational documentation.

Review notes: The PR has five unresolved findings. Its title also lacks the required JIRA key, and the diff exceeds the recommended 1,000-line target.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/tsconfig.json	Enables BigInt test syntax.
services/libs/data-access-layer/src/packages/transitiveDependents.ts	Implements snapshot, closure, and merge queries.
services/libs/data-access-layer/src/packages/transitiveDependents.integration.test.ts	Tests graph and ledger behavior.
services/libs/data-access-layer/src/packages/packagistTransitiveRuns.ts	Adds run-ledger operations.
services/libs/data-access-layer/src/packages/index.ts	Exports new DAL modules.
services/libs/data-access-layer/src/osspckgs/ingestJobs.ts	Adds pending-job lookup.
services/apps/packages_worker/src/workflows/index.ts	Exports the workflow.
services/apps/packages_worker/src/scripts/triggerPackagistSeed.ts	Adds manual transitive trigger.
services/apps/packages_worker/src/packagist/workflows.ts	Orchestrates preparation and merge draining.
services/apps/packages_worker/src/packagist/README.md	Documents the new lane.
services/apps/packages_worker/src/packagist/activities.ts	Implements Temporal activities.
services/apps/packages_worker/src/packagist/__tests__/wiring.test.ts	Verifies worker exports.
services/apps/packages_worker/src/packagist/__tests__/transitiveDependents.test.ts	Tests workflow orchestration.
services/apps/packages_worker/src/criticality/activities.ts	Reuses pending-job lookup.
services/apps/packages_worker/src/activities.ts	Registers new activities.
docs/adr/0009-packagist-worker-design-decisions.md	Records the architecture decision.
backend/src/osspckgs/migrations/V1785740540__packagist_transitive_runs.sql	Creates the run ledger.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

services/libs/data-access-layer/src/osspckgs/ingestJobs.ts:80

• This coerces a PostgreSQL bigserial identifier to a JavaScript number even though the packages-db connection intentionally leaves int8 values as strings. Once an ID exceeds Number.MAX_SAFE_INTEGER, the rounded value can update or query the wrong job. Keep the ID as a string and align createIngestJob, markJobStatus, and their callers with that representation.

  // id is bigserial (pg returns int8 as a string) — convert so the declared type is true.
  return row ? Number(row.id) : null

services/apps/packages_worker/src/packagist/workflows.ts:70

• batch.changed is not retry-stable. If the merge activity commits its UPDATE but Temporal loses the completion, the retry processes the same cursor after the rows already match and returns changed: 0; this accumulator then permanently under-reports changed_rows. Persist per-batch/cumulative change counts atomically with the merge (keyed by run/cursor), or otherwise make the returned count deterministic across activity retries.

      const batch = await acts.mergePackagistTransitiveBatch(cursor, TRANSITIVE_MERGE_BATCH)
      processed += batch.processed
      changed += batch.changed

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (3)

services/libs/data-access-layer/src/packages/transitiveDependents.ts:23

• statement_timeout applies to each statement, not the whole prepare activity. This helper runs CTAS, index creation, and ANALYZE twice sequentially, so a valid attempt can exceed the 45-minute start-to-close timeout while its transaction remains active. Temporal may then retry the full scan concurrently and contend on these global staging tables. Split the phases into separately timed activities or enforce an end-to-end database deadline that terminates the original attempt before Temporal retries it.

    await tx.result(`SET LOCAL statement_timeout = '40min'`)

docs/adr/0009-packagist-worker-design-decisions.md:191

• This edit leaves ADR-0009 outside the repository's mandatory ADR structure: it has ## Decisions rather than ## Decision and lacks ## Alternatives Considered plus the required ## Consequences subsections. Please bring the ADR into compliance with .claude/rules/adr-format.md:18-29 while updating it.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

services/apps/packages_worker/src/packagist/activities.ts:520

• This marks the ledger row failed after every failed activity attempt. Since Temporal retries prepare up to three times and findUnfinishedPackagistTransitiveRun excludes failed rows, the next attempt creates a new row; one workflow run can therefore produce several failed rows before succeeding, defeating the documented retry reuse and “one row per run” lifecycle. Keep the row unfinished for retryable, non-final attempts and mark it failed only for a non-retryable error or after retries are exhausted.

  } catch (err) {
    await failRunInLedger(qx, runId, (err as Error).message)
    throw err

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (2)

services/libs/data-access-layer/tsconfig.json:9

• Raising the shared DAL target to ES2020 for one test removes the ES2017 syntax check inherited from services/base.tsconfig.json:3, even though this comment says production sources must remain ES2017-compatible. Keep the library target at ES2017 and isolate or rewrite the test's BigInt literal instead of weakening checks for every DAL source file.

  // target/lib raised over base's es2017 for the BigInt literals in the packagist
  // transitive integration test (include pulls *.test.ts into tsc-check). Consumers
  // compile DAL sources under their own (es2017) configs, so src itself must stay
  // free of post-es2017 syntax.
  "compilerOptions": {
    "target": "es2020",
    "lib": ["es2020", "ES2021.String"]

docs/adr/0009-packagist-worker-design-decisions.md:191

• This adds another decision as a ### subsection, but .claude/rules/adr-format.md:18-29 requires edited ADRs to use the mandatory ## Decision, ## Alternatives Considered, and ## Consequences structure. This ADR still lacks those required sections and should be restructured before merging.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

[epipav]

@cursor review

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (3)

services/libs/data-access-layer/src/osspckgs/ingestJobs.ts:80

• osspckgs_ingest_jobs.id is bigserial, and the packages DB intentionally leaves int8 values as strings (services/libs/database/src/connection.ts:81-85). Converting it with Number loses identity once IDs exceed Number.MAX_SAFE_INTEGER, potentially updating the wrong job. Keep the ID as a string and align the existing create/mark job signatures and callers accordingly.

  return row ? Number(row.id) : null

docs/adr/0009-packagist-worker-design-decisions.md:191

• The repository ADR rule (.claude/rules/adr-format.md:18-28) requires edited numbered ADRs to contain ## Decision, ## Alternatives Considered, and ## Consequences in order. This adds another ### entry under ## Decisions while those mandatory sections remain absent, so the ADR still violates the enforced format. Restructure the living ADR or record this as a conforming separate ADR.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

services/libs/data-access-layer/src/packages/transitiveDependents.integration.test.ts:73

• This cleanup casts bigserial job IDs to int[], so it will fail with integer out of range once the sequence exceeds the 32-bit range. Cast to bigint[] to match the table schema.

        await qx.result(`DELETE FROM osspckgs_ingest_jobs WHERE id = ANY($(jobIds)::int[])`, {

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/adr/0009-packagist-worker-design-decisions.md:191

• This ADR update still does not follow the mandatory structure in .claude/rules/adr-format.md:18-28: the file has no ## Decision, ## Alternatives Considered, or ## Consequences sections in the required order. Please bring the edited ADR into that structure before adding this decision.

### Transitive dependent counts: weekly materialized reverse closure over our own edges

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0e14296. Configure here.}

[cursor]

Fail-mark masks non-retryable errors

Medium Severity

When prepare hits a non-retryable empty-edge abort, the catch awaits failRunInLedger before rethrowing. If that ledger write throws, the original ApplicationFailure.nonRetryable never propagates, so Temporal treats the failure as retryable and can rerun the full package_dependencies snapshot up to the prepare attempt limit.

 

^{Reviewed by Cursor Bugbot for commit 0e14296. Configure here.}