fix(oce-report): align the noise gate to the report's rolling window, Fixes AB#3731628 - #459
Open
Shahzaib (shahzaibj) wants to merge 1 commit into
Conversation
…d pills
The novelty classifier graded Sunday-aligned calendar weeks cut off at
startofweek(curEnd), while the report displays a rolling 7-day window. On the
2026-08-01 run the gate's "current" week was 07/19-07/26 against a report window
of 07/25-08/01 -- one day of overlap -- so anything that turned in the final ~6
days was structurally invisible to the gate.
Two real risers were suppressed as "ONGOING, do not re-triage":
authorization_pending report +63.2% classifier -37.1%
expired_token report +26.7% classifier -51.0%
Both were codes the on-call engineer had already spotted by hand.
Fix: bucket with bin_at(t, 7d, datetime(curEnd)) everywhere. The newest bucket
becomes exactly [curEnd-7d, curEnd) -- the report's own window -- so classifier
WoW equals displayed WoW by construction. Every bucket is a complete 7 days; the
4-day stub moves to the oldest bucket and --start drops it, leaving 8 clean
rolling weeks. TREND_CLASS_END and --include-partial-end are obsolete
(--include-partial-end kept as a no-op).
Alignment removes phantom signal as well as adding real signal: access_denied
was ACCELERATING under calendar weeks but is actually -53.2%. The attention set
went 4 -> 5 keys, not 4 -> 15.
Second, independent defect: a key whose 60-day peak sits under the peak floor
(Broker 10,000 / AuthApp 1,000 devices) is excluded from classification outright,
so it can carry a red pill forever while attention says "quiet week". Passkey
WebAuthN Registration shipped tag-bad (-1.27 pts, worst in the table) directly
above "Quiet week - 0 NEW or ACCELERATING"; it peaks at 732 devices. No bucketing
change fixes this. Added a reconciliation rule -- every tag-bad/tag-warn row is
either promoted into attention or named in a muted .reconcile-note with its
reason -- enforced by new validator check 19.
Also reordered the Authenticator report so Needs attention sits above the
scenario scoreboard, matching the Broker report and the reading order an OCE
actually wants.
Verified end-to-end against live Kusto for 2026-08-01:
- Broker: authorization_pending + expired_token now ACCELERATING in attention;
access_denied correctly absent; 3 visible attention rows; check 19 green.
- AuthApp: still a genuine quiet week (NEW=0, ACCELERATING=0); both red pills
now explained by a .reconcile-note; section order confirmed.
- Both: "All hard checks passed", 0 U+FFFD.
- Protected 2026-07-31 before-fixture still fires exactly 3 hard FAILs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
❌ Work item link check failed. Description does not contain AB#{ID}. Click here to Learn more. |
|
✅ Work item link check complete. Description contains link AB#3731628 to an Azure Boards work item. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
classify-novelty.js— the noise gate that decides what lands in "Things that need attention thisweek" — graded Sunday-aligned calendar weeks cut off at
startofweek(curEnd). But the reportdisplays a rolling 7-day window.
For the 2026-08-01 run:
2026-07-25 → 2026-08-012026-07-19 → 2026-07-26One day of overlap. Anything that turned in the last ~6 days was structurally invisible to the
gate — exactly the period an on-call engineer cares about most.
Evidence
authorization_pendingexpired_tokenBoth were genuinely rising, both were told to be ignored.
The same defect surfaced on AuthApp as a red scoreboard pill (rolling-derived) sitting directly
above an empty "Needs attention" section (calendar-derived) — the two halves of the report
disagreeing because they were measuring different weeks.
The fix
Bucket the 60-day trend and sparklines with
bin_at(<TIME>, 7d, datetime(<TREND_END>))instead ofstartofweek(<TIME>).The final bucket then is the report's window, so classifier WoW == displayed WoW by
construction — not by tuning. Every bucket is a complete 7 days, so
--include-partial-endandTREND_CLASS_ENDbecome obsolete and are removed. Because 60 isn't a multiple of 7 the oldestbucket is the 4-day partial one (the safe end to be partial on);
--startdrops it, leaving 8 cleanrolling weeks.
This is not "more alerts"
Alignment adds real signal and removes phantom signal:
authorization_pending,expired_token→ correctly promoted to ACCELERATINGaccess_denied→ correctly demoted; it was flagged ACCELERATING but is actually −53.2%Attention set goes 4 → 5 keys, not 4 → 15.
Second, independent defect (also fixed here)
A key whose 60-day peak falls under the peak-floor (Broker 10,000 / AuthApp 1,000 devices) is
excluded from classification entirely — so it can carry a red pill forever while the attention
section truthfully reports a quiet week.
Passkey WebAuthN Registrationpeaks at 732 devices andshipped
tag-bad(worst delta in its table) directly above "Quiet week — 0 NEW or ACCELERATING".No bucketing change fixes this. Added a mandatory reconciliation rule: every
tag-bad/tag-warnpill must be either promoted into attention or named in a mutedreconcile-notegiving the reason, tested in order — (1) below the classification floor, (2) within its own normal
band, (3) ONGOING and flat.
validate-report.ps1check 19 now hard-fails an unreconciled pill.Also
bucket-trends.jsnow warns if--endis omitted. Its partial-end auto-drop is guarded byif (!endArg …); under rolling alignment the newest bucket is genuinely complete, so omitting--endcould silently discard a real 70% collapse as "looks partial".SKILL.md, both playbooks, both cheatsheets, and all 5 KQL files, with anexplicit "do not reintroduce
startofweek()" note carrying the evidence above — this is thekind of bug that gets "helpfully" reverted by someone who thinks calendar weeks look tidier.
classify-novelty.jsitself needed no code change — it already readsd.classifyWeeks || d.weeks, which are equal under rolling alignment.Testing
Re-ran both reports end-to-end against live Kusto for the 2026-08-01 window. Verified the two
previously-suppressed codes now appear in attention with their correct rolling deltas, that
access_deniedno longer appears, and that the AuthApp red pill is now reconciled. The protected2026-07-31 before-fixture still fires exactly 3 hard FAILs, confirming the validator did not go soft.
Fixes AB#3731628