Skip to content

fix(oce-report): align the noise gate to the report's rolling window, Fixes AB#3731628 - #459

Open
Shahzaib (shahzaibj) wants to merge 1 commit into
shjameel-microsoft-oce-authapp-reportfrom
shjameel-microsoft-oce-noise-gate-fix
Open

fix(oce-report): align the noise gate to the report's rolling window, Fixes AB#3731628#459
Shahzaib (shahzaibj) wants to merge 1 commit into
shjameel-microsoft-oce-authapp-reportfrom
shjameel-microsoft-oce-noise-gate-fix

Conversation

@shahzaibj

@shahzaibj Shahzaib (shahzaibj) commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

⚠️ Stacked on #455 — review that one first. This PR targets
shjameel-microsoft-oce-authapp-report, not master, and shows a clean 1-commit / 18-file diff
against it. GitHub will auto-retarget it to master when #455 merges.

The bug

classify-novelty.js — the noise gate that decides what lands in "Things that need attention this
week"
— graded Sunday-aligned calendar weeks cut off at startofweek(curEnd). But the report
displays a rolling 7-day window.

For the 2026-08-01 run:

window
What the report showed 2026-07-25 → 2026-08-01
What the classifier graded 2026-07-19 → 2026-07-26

One day of overlap. Anything that turned in the last ~6 days was structurally invisible to the
gate — exactly the period an on-call engineer cares about most.

Evidence

Code Report WoW (rolling) Classifier WoW (calendar) Gap Verdict it shipped with
authorization_pending +63.2% −37.1% 100 pts "ONGOING — do not re-triage"
expired_token +26.7% −51.0% 78 pts "ONGOING — do not re-triage"

Both were genuinely rising, both were told to be ignored.

The same defect surfaced on AuthApp as a red scoreboard pill (rolling-derived) sitting directly
above an empty "Needs attention" section (calendar-derived) — the two halves of the report
disagreeing because they were measuring different weeks.

The fix

Bucket the 60-day trend and sparklines with bin_at(<TIME>, 7d, datetime(<TREND_END>)) instead of
startofweek(<TIME>).

The final bucket then is the report's window, so classifier WoW == displayed WoW by
construction
— not by tuning. Every bucket is a complete 7 days, so --include-partial-end and
TREND_CLASS_END become obsolete and are removed. Because 60 isn't a multiple of 7 the oldest
bucket is the 4-day partial one (the safe end to be partial on); --start drops it, leaving 8 clean
rolling weeks.

This is not "more alerts"

Alignment adds real signal and removes phantom signal:

  • authorization_pending, expired_token → correctly promoted to ACCELERATING
  • access_denied → correctly demoted; it was flagged ACCELERATING but is actually −53.2%

Attention set goes 4 → 5 keys, not 4 → 15.

Second, independent defect (also fixed here)

A key whose 60-day peak falls under the peak-floor (Broker 10,000 / AuthApp 1,000 devices) is
excluded from classification entirely — so it can carry a red pill forever while the attention
section truthfully reports a quiet week. Passkey WebAuthN Registration peaks at 732 devices and
shipped tag-bad (worst delta in its table) directly above "Quiet week — 0 NEW or ACCELERATING".

No bucketing change fixes this. Added a mandatory reconciliation rule: every tag-bad /
tag-warn pill must be either promoted into attention or named in a muted reconcile-note
giving the reason, tested in order — (1) below the classification floor, (2) within its own normal
band, (3) ONGOING and flat. validate-report.ps1 check 19 now hard-fails an unreconciled pill.

Also

  • bucket-trends.js now warns if --end is omitted. Its partial-end auto-drop is guarded by
    if (!endArg …); under rolling alignment the newest bucket is genuinely complete, so omitting
    --end could silently discard a real 70% collapse as "looks partial".
  • Docs updated across SKILL.md, both playbooks, both cheatsheets, and all 5 KQL files, with an
    explicit "do not reintroduce startofweek()" note carrying the evidence above — this is the
    kind of bug that gets "helpfully" reverted by someone who thinks calendar weeks look tidier.

classify-novelty.js itself needed no code change — it already reads
d.classifyWeeks || d.weeks, which are equal under rolling alignment.

Testing

Re-ran both reports end-to-end against live Kusto for the 2026-08-01 window. Verified the two
previously-suppressed codes now appear in attention with their correct rolling deltas, that
access_denied no longer appears, and that the AuthApp red pill is now reconciled. The protected
2026-07-31 before-fixture still fires exactly 3 hard FAILs, confirming the validator did not go soft.

Fixes AB#3731628

…d pills

The novelty classifier graded Sunday-aligned calendar weeks cut off at
startofweek(curEnd), while the report displays a rolling 7-day window. On the
2026-08-01 run the gate's "current" week was 07/19-07/26 against a report window
of 07/25-08/01 -- one day of overlap -- so anything that turned in the final ~6
days was structurally invisible to the gate.

Two real risers were suppressed as "ONGOING, do not re-triage":

  authorization_pending   report +63.2%   classifier -37.1%
  expired_token           report +26.7%   classifier -51.0%

Both were codes the on-call engineer had already spotted by hand.

Fix: bucket with bin_at(t, 7d, datetime(curEnd)) everywhere. The newest bucket
becomes exactly [curEnd-7d, curEnd) -- the report's own window -- so classifier
WoW equals displayed WoW by construction. Every bucket is a complete 7 days; the
4-day stub moves to the oldest bucket and --start drops it, leaving 8 clean
rolling weeks. TREND_CLASS_END and --include-partial-end are obsolete
(--include-partial-end kept as a no-op).

Alignment removes phantom signal as well as adding real signal: access_denied
was ACCELERATING under calendar weeks but is actually -53.2%. The attention set
went 4 -> 5 keys, not 4 -> 15.

Second, independent defect: a key whose 60-day peak sits under the peak floor
(Broker 10,000 / AuthApp 1,000 devices) is excluded from classification outright,
so it can carry a red pill forever while attention says "quiet week". Passkey
WebAuthN Registration shipped tag-bad (-1.27 pts, worst in the table) directly
above "Quiet week - 0 NEW or ACCELERATING"; it peaks at 732 devices. No bucketing
change fixes this. Added a reconciliation rule -- every tag-bad/tag-warn row is
either promoted into attention or named in a muted .reconcile-note with its
reason -- enforced by new validator check 19.

Also reordered the Authenticator report so Needs attention sits above the
scenario scoreboard, matching the Broker report and the reading order an OCE
actually wants.

Verified end-to-end against live Kusto for 2026-08-01:
  - Broker: authorization_pending + expired_token now ACCELERATING in attention;
    access_denied correctly absent; 3 visible attention rows; check 19 green.
  - AuthApp: still a genuine quiet week (NEW=0, ACCELERATING=0); both red pills
    now explained by a .reconcile-note; section order confirmed.
  - Both: "All hard checks passed", 0 U+FFFD.
  - Protected 2026-07-31 before-fixture still fires exactly 3 hard FAILs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@shahzaibj
Shahzaib (shahzaibj) requested a review from a team as a code owner August 18, 2026 07:21
@github-actions

Copy link
Copy Markdown

❌ Work item link check failed. Description does not contain AB#{ID}.

Click here to Learn more.

@github-actions github-actions Bot changed the title fix(oce-report): align the noise gate to the report's rolling window fix(oce-report): align the noise gate to the report's rolling window, Fixes AB#3731628 Aug 18, 2026
@github-actions

Copy link
Copy Markdown

✅ Work item link check complete. Description contains link AB#3731628 to an Azure Boards work item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant