fix: remove 500-row cap that truncated viewer count displays

[chiefmojo]

Problem

clampLimit() in _helpers.ts hard-caps query limits at 500 rows. The viewer's trace and episode count displays fetch the first N rows to count totals, so any store with more than 500 items shows a truncated count.

Fix

Raise the cap from 500 to 100,000. This is a safety guard against pathological inputs, not a query-size policy — reasonable operators setting limit: 10_000 for bulk exports should not be silently truncated.

export function clampLimit(n: number): number {
  if (!Number.isFinite(n) || n <= 0) return 50;
  return Math.min(Math.trunc(n), 100_000);
}

[Memtensor-AI]

Automated Test Results: FAILED

Cloud test-engine rerun against dev-v2.0.22 failed.

• Run: tr-e1b6e067-ca4 on cloud test-engine 10011

• Verdict: fail_code_bug

• Summary: Tests failed. Failed cases: test -- tests/unit/pipeline/memory-core.test.ts tests/unit/server/http.test.ts

• memos_local_plugin/unit: 102 passed, 1 failed, 0 skipped

Failed cases:

• test -- tests/unit/pipeline/memory-core.test.ts tests/unit/server/http.test.ts: ry shape the viewer expects
  AssertionError: expected 500 to be 200 // Object.is equality

• Expected

• Received

• 200

• 500

❯ tests/unit/server/http.test.ts:834:22
832| it("GET /api/v1/overview returns the summary shape the viewer expect…
833| const r = await fetch(${handle.url}/api/v1/overview);
834| expect(r.status).toBe(200);
| ^
835| const body = (await r.json()) as {
836| ok?: boolean;

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

Do not merge until this is fixed and cloud tests pass.

[syzsunshine219]

Automated Test Results: PASSED

Cloud test-engine rerun against the current PR head and latest main completed successfully after fixing the test stub.

• Passing run: tr-783ba09b-a4a on cloud test-engine 10012
• Ref: refs/pull/1954/head merged with latest main before execution
• Scope: memos_local_plugin/unit
• Result: 107 passed, 0 failed, 0 skipped
• Duration: 12s

The preceding rerun (tr-3c29f91b-182) reproduced the prior failure: /api/v1/overview returned 500 because the HTTP test MemoryCore stub did not implement the newly used countPolicies() and countWorldModels() methods. Commit 27c916ed adds those two stubs; production code is unchanged by the test fix. Manual code review is still required before merge.

[Memtensor-AI]

🤖 Open Code Review

Target: PR #1954
Task: 6f9badc340848847
Base: dev-v2.0.25
Head: fix/viewer-count-cap

🔍 OpenCodeReview found 4 issue(s) in this PR.

---

1. apps/memos-local-plugin/core/pipeline/memory-core.ts (L3767-L3769)

Replacing one arbitrary hardcoded ceiling (5_000) with another (100_000) still silently under-counts when the actual number of skills exceeds the limit. countSkills is supposed to return an accurate total, but it will return at most 100_000 regardless of true data size, making the count unreliable at scale.

Additionally, the sibling function listSkills (just above, line 3753) still uses limit: 5_000, so the count returned by countSkills will exceed the number of rows actually retrievable via listSkills for large datasets — creating a visible pagination inconsistency for callers.

Consider two targeted fixes:

1. For countSkills: use a dedicated repository method that performs a true DB-level count (e.g., handle.repos.skills.count(...)) instead of fetching all rows into memory and calling .length. If no such method exists, at minimum paginate/loop until exhausted rather than relying on a single large limit.
2. For consistency: align the limit in listSkills with the actual maximum supported by the repo layer, or document clearly that both functions are bounded by their respective caps.

💡 Suggested Change

Before:

    return handle.repos.skills.list({ status: input?.status, limit: 100_000 }).filter((r) =>
      (input?.includeAllNamespaces || visibleToCurrent(r)) && matchesNamespaceFilter(r, input)
    ).length;

After:

    // TODO: replace with a DB-level count query to avoid loading all rows into memory
    // and to guarantee accuracy beyond any fixed limit.
    // e.g., return handle.repos.skills.count({ status: input?.status, ... });
    return handle.repos.skills.list({ status: input?.status, limit: 100_000 }).filter((r) =>
      (input?.includeAllNamespaces || visibleToCurrent(r)) && matchesNamespaceFilter(r, input)
    ).length;

---

2. apps/memos-local-plugin/core/pipeline/memory-core.ts (L3767)

Loading up to 100,000 full skill rows into memory on every call to countSkills purely to obtain a .length is a significant performance concern. At scale this causes large heap allocations and slows down every count operation. A DB-level COUNT query or a streaming approach should be used instead.

---

3. apps/memos-local-plugin/server/routes/overview.ts (L50-L73)

The countSkills implementation in memory-core.ts (revealed by the companion diff) still performs a full table scan under the hood: handle.repos.skills.list({ status, limit: 100_000 }).filter(...).length. By calling countSkills three times — once per status — this code issues three separate full scans of the skills table (up to 300,000 row fetches in the worst case), and similarly three scans for policies.

A single call without a status filter (i.e., countSkills({ includeAllNamespaces: true }) returning total, then iterating once) would be far cheaper. Even better, a single listSkills({ limit: 100_000, includeAllNamespaces: true }) call followed by in-memory grouping — analogous to the old approach — would read the table once and produce all three counts. The refactor inadvertently triples the I/O cost for skills and policies.

Consider consolidating into a single count call or adding a batched count API before landing this.

💡 Suggested Change

Before:

      deps.core.countSkills({
        status: "active",
        includeAllNamespaces: true,
      }),
      deps.core.countSkills({
        status: "candidate",
        includeAllNamespaces: true,
      }),
      deps.core.countSkills({
        status: "archived",
        includeAllNamespaces: true,
      }),
      deps.core.countPolicies({
        status: "active",
        includeAllNamespaces: true,
      }),
      deps.core.countPolicies({
        status: "candidate",
        includeAllNamespaces: true,
      }),
      deps.core.countPolicies({
        status: "archived",
        includeAllNamespaces: true,
      }),

After:

      // Single unfiltered count covers all statuses; then one filtered
      // scan for the per-status breakdown avoids triple table scans.
      deps.core.countSkills({ includeAllNamespaces: true }),
      deps.core.listSkills({ limit: 100_000, includeAllNamespaces: true }),
      deps.core.countPolicies({ includeAllNamespaces: true }),
      deps.core.listPolicies({ limit: 100_000, includeAllNamespaces: true }),

---

4. apps/memos-local-plugin/server/routes/overview.ts (L75-L77)

The comment reads "cheaper than a dedicated count RPC and already cached by the core", which was accurate in the old code where metrics() was the only count-like call and the surrounding calls were bulk list fetches. In the refactored code, every other call in the Promise.all IS a dedicated count RPC, so the contrast the comment draws no longer makes sense to a reader. The phrase "cheaper than a dedicated count RPC" is now actively misleading in this context.

Update the comment to describe only what metrics.total represents (the grand total of traces), without the now-false efficiency comparison.

💡 Suggested Change

Before:

      // `metrics.total` is the grand total of traces — cheaper than a
      // dedicated count RPC and already cached by the core.
      deps.core.metrics({ days: 1, includeAllNamespaces: true }),

After:

      // `metrics.total` is the grand total of trace turns shown on the
      // Overview page; `metrics()` also provides write-today / session
      // counts that may be used in the future.
      deps.core.metrics({ days: 1, includeAllNamespaces: true }),

Generated by cloud-assistant via Open Code Review.

[Memtensor-AI]

✅ Automated Test Results: PASSED

All tests passed (107/107 executed). memos_local_plugin/unit: 107/107. Duration: 11s

Branch: fix/viewer-count-cap

[Memtensor-AI]

⚠️ Automated Test Results: NO TEST SCOPE

Automated tests were not run because the changed files do not map to an executable test scope.

Details: No executable test scope maps to the changed files. Automated tests were not run; add env.yaml source_mapping + execution for this path family, then rerun. Changed files: (none detected)
Manual review or env.yaml source_mapping/execution coverage is required before merge.

Branch: fix/viewer-count-cap