From 3b4036276b86ea2821769324c62440c287a0d13a Mon Sep 17 00:00:00 2001 From: Shahzaib Date: Fri, 31 Jul 2026 20:25:41 -0700 Subject: [PATCH] fix(oce-report): align noise gate to the rolling window; reconcile red pills The novelty classifier graded Sunday-aligned calendar weeks cut off at startofweek(curEnd), while the report displays a rolling 7-day window. On the 2026-08-01 run the gate's "current" week was 07/19-07/26 against a report window of 07/25-08/01 -- one day of overlap -- so anything that turned in the final ~6 days was structurally invisible to the gate. Two real risers were suppressed as "ONGOING, do not re-triage": authorization_pending report +63.2% classifier -37.1% expired_token report +26.7% classifier -51.0% Both were codes the on-call engineer had already spotted by hand. Fix: bucket with bin_at(t, 7d, datetime(curEnd)) everywhere. The newest bucket becomes exactly [curEnd-7d, curEnd) -- the report's own window -- so classifier WoW equals displayed WoW by construction. Every bucket is a complete 7 days; the 4-day stub moves to the oldest bucket and --start drops it, leaving 8 clean rolling weeks. TREND_CLASS_END and --include-partial-end are obsolete (--include-partial-end kept as a no-op). Alignment removes phantom signal as well as adding real signal: access_denied was ACCELERATING under calendar weeks but is actually -53.2%. The attention set went 4 -> 5 keys, not 4 -> 15. Second, independent defect: a key whose 60-day peak sits under the peak floor (Broker 10,000 / AuthApp 1,000 devices) is excluded from classification outright, so it can carry a red pill forever while attention says "quiet week". Passkey WebAuthN Registration shipped tag-bad (-1.27 pts, worst in the table) directly above "Quiet week - 0 NEW or ACCELERATING"; it peaks at 732 devices. No bucketing change fixes this. Added a reconciliation rule -- every tag-bad/tag-warn row is either promoted into attention or named in a muted .reconcile-note with its reason -- enforced by new validator check 19. Also reordered the Authenticator report so Needs attention sits above the scenario scoreboard, matching the Broker report and the reading order an OCE actually wants. Verified end-to-end against live Kusto for 2026-08-01: - Broker: authorization_pending + expired_token now ACCELERATING in attention; access_denied correctly absent; 3 visible attention rows; check 19 green. - AuthApp: still a genuine quiet week (NEW=0, ACCELERATING=0); both red pills now explained by a .reconcile-note; section order confirmed. - Both: "All hard checks passed", 0 U+FFFD. - Protected 2026-07-31 before-fixture still fires exactly 3 hard FAILs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../oncall-weekly-telemetry-report/SKILL.md | 56 ++++++-- .../assets/docs/authapp-kusto-cheatsheet.md | 26 ++-- .../assets/docs/kusto-cheatsheet.md | 62 +++++---- .../assets/playbooks/authapp.md | 118 ++++++++++++----- .../assets/playbooks/broker.md | 120 ++++++++++++------ .../assets/queries/60d-trend-codes.kql | 42 +++--- .../assets/queries/60d-trend-types.kql | 14 +- .../assets/queries/README.md | 15 +-- .../assets/queries/authapp/README.md | 6 +- .../queries/authapp/scenario-60d-trend.kql | 22 +++- .../authapp/scenario-sparkline-series.kql | 31 +++-- .../queries/wow-table-sparkline-series.kql | 37 +++--- .../assets/scripts/bootstrap-report.ps1 | 60 +++++---- .../assets/scripts/bucket-trends.js | 103 ++++++++------- .../assets/scripts/validate-report.ps1 | 58 +++++++++ .../templates/authapp-report-template.html | 90 +++++++++---- .../assets/templates/report-template.html | 45 ++++++- .../assets/templates/template-readme.md | 15 +-- 18 files changed, 613 insertions(+), 307 deletions(-) diff --git a/.github/skills/oncall-weekly-telemetry-report/SKILL.md b/.github/skills/oncall-weekly-telemetry-report/SKILL.md index 37d2654a..75dc6b2b 100644 --- a/.github/skills/oncall-weekly-telemetry-report/SKILL.md +++ b/.github/skills/oncall-weekly-telemetry-report/SKILL.md @@ -52,8 +52,10 @@ crash layer, which needs a secret), `--end YYYY-MM-DD` (see § Reporting window) Resolved reporting window (UTC): # example values for a run on 2026-07-15 Last 7 days: 2026-07-08 -> 2026-07-15 (exclusive upper bound) Baseline: 2026-07-01 -> 2026-07-08 - 60-day trend: 2026-05-16 -> 2026-07-15 (literal 60d ending today; chart includes current partial week) - Trend delta cutoff: weeks < 2026-07-12 (startofweek(curEnd); pass as bucket-trends.js --end) + 60-day trend: 2026-05-16 -> 2026-07-15 (literal 60d ending today; rolling 7d buckets anchored at curEnd) + Trend buckets: 8 complete rolling weeks; final bucket == the Last-7-days window above (classifier WoW == displayed WoW) + bucket-trends.js: --start=2026-05-16 --end=2026-07-15 (pass BOTH; --end disables the partial-end auto-drop heuristic) + Sparkline (8 rolling weeks): 2026-05-20 -> 2026-07-15 (SPARK_START -> SPARK_END, exclusive; no Sunday alignment needed) ``` These dates are stamped into each report's ``, `<div class="meta">`, and Generated banner during bootstrap — you do not hand-edit them. @@ -85,17 +87,32 @@ crash layer, which needs a secret), `--end YYYY-MM-DD` (see § Reporting window) (`[curEnd - 14d, curEnd - 7d)`). No user input. 3. **60-day trend window** — auto-computed as the **literal last 60 days ending today** - (`[curEnd - 60d, curEnd)`), so both bounds move with `-EndDate`. Trend sections are Sun-Sat - weekly-bucketed (Kusto `startofweek()` is Sunday-aligned) because the trend needs stable weekly - denominators, but the final bar is the **current in-progress (partial) week** — the chart ends - today. Regression/improvement **delta classification is still computed on complete weeks only** - (`bucket-trends.js --end=startofweek(curEnd) --include-partial-end`); a partial week as "last" - would read as a fake −99% improvement, so it is charted but excluded from the delta math. - -**Kusto note (weekly-bucketed queries only):** `startofweek()` is Sunday-aligned, so -`startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When authoring weekly-bucketed queries, verify -by printing the distinct week values from your first query. Off-by-one-week is the #1 silent error -in weekly-bucket queries. + (`[curEnd - 60d, curEnd)`), so both bounds move with `-EndDate`. Trend and sparkline sections are + bucketed into **rolling 7-day windows anchored at `curEnd`** (`bin_at(t, 7d, datetime(curEnd))`), + **not** Sun-Sat calendar weeks. The final bucket is therefore `[curEnd - 7d, curEnd)` — byte-for-byte + the same window as the headline WoW numbers — so **the novelty classifier's WoW equals the WoW the + report prints**, by construction. Every bucket is a complete 7 days; there is no partial end bar and + no separate classification cutoff. Invoke as + `bucket-trends.js --start=<curEnd-60d> --end=<curEnd>` (**pass both** — see the note below). + + Because 60 is not a multiple of 7, the **oldest** bucket (`curEnd - 63d`) covers 4 days and is the + partial one — the safe end to be partial on. `--start` drops it, leaving 8 complete rolling weeks. + +> **⚠️ Why not `startofweek()`.** Calendar-week bucketing cut off at `startofweek(curEnd)` lagged the +> report's rolling window by up to a full week, so anything that turned in the last ~6 days was +> structurally invisible to the noise gate — exactly the period an on-call engineer cares about most. +> On the 2026-08-01 run the gate's "current" week was 07/19–07/26 against a report window of +> 07/25–08/01 (**one day of overlap**): `authorization_pending` read **+63.2%** in the report and +> **−37.1%** to the classifier, and was filed *"ONGOING — do not re-triage"*. Same for `expired_token` +> (+26.7% vs −51.0%). Re-bucketing on `bin_at` promoted both to ACCELERATING **and** demoted +> `access_denied`, a former false positive (actually −53.2%). Alignment adds real signal *and* removes +> phantom signal — it is not merely "more alerts". **Do not reintroduce `startofweek()`.** + +> **⚠️ Always pass `--end=<curEnd>` to `bucket-trends.js`, not just `--start`.** Its partial-end +> auto-drop heuristic is guarded by `if (!endArg …)`. Under rolling alignment the newest bucket is +> genuinely complete, so omitting `--end` would let a **real** 70% collapse be silently discarded as +> "looks partial". Passing `--end` filters nothing (every bucket label is `< curEnd` by construction) +> and disables the heuristic. The script now warns if you omit it. --- @@ -210,6 +227,19 @@ Never carry a convention across the two playbooks. - **A moved metric is a question, not a verdict.** Never publish a regression verdict without the app's diagnostic ladder having been walked (Broker: Originator pre-check + dim slicing; Authenticator: volume floor + rate normalisation + error-reason decomposition). +- **Every red/amber table pill must be reconciled.** The scoreboard / WoW tables colour a row from + its own rolling delta; the attention section is populated from `classify-novelty.js`'s **novelty** + verdict. Those answer different questions, so a row can be legitimately red in the table and + legitimately absent from attention — but a reader who sees that mismatch with no explanation + concludes the report is broken. Precedent: `Passkey WebAuthN Registration` shipped carrying + `tag-bad` (−1.27 pts, worst delta in the table) directly above the words *"Quiet week — 0 NEW or + ACCELERATING"*. Both statements were true: the scenario peaks at ~732 bad-outcome devices, under + the 1,000-device peak-floor, so it is **structurally excluded** from classification and can never + appear in attention however sharply it moves. Every `tag-bad`/`tag-warn` row must therefore be + **either** promoted into attention **or** named in a muted `<div class="reconcile-note">` giving + the reason it is not escalated — test the reasons in this order: (1) below the classification + floor, (2) within its own normal band, (3) ONGOING and flat. `validate-report.ps1` check 19 + hard-fails an unreconciled pill. - **Filename collision rule.** If a report already exists for the same end-date, do not silently overwrite. Open it, list its top-3 findings, and explicitly state in chat what changed in the new data before regenerating. A second run on the same window without a delta is wasted work. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/docs/authapp-kusto-cheatsheet.md b/.github/skills/oncall-weekly-telemetry-report/assets/docs/authapp-kusto-cheatsheet.md index b2b2eddd..936b0a55 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/docs/authapp-kusto-cheatsheet.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/docs/authapp-kusto-cheatsheet.md @@ -47,8 +47,8 @@ same shape `bucket-trends.js` and `agg.js` already parse. Outcome columns each have a `…DCount` distinct-device twin. **Registration / Authentication MVs expose only `Initiated / Succeeded / Failed` (+`DCount`) and -`TotalUniqueDevices`. There is NO `Cancelled` and NO `PartiallySucceeded` column** — do not -invent one. PN MVs carry only an initiated counter; the terminal outcome lives in the paired +`TotalUniqueDevices`. There is NO `Cancelled` column and NO separate partly-succeeded outcome column** +— do not invent one. PN MVs carry only an initiated counter; the terminal outcome lives in the paired `_Results_MV_V1`. | Scenario | Outcome MV | Initiate column | @@ -223,13 +223,17 @@ estimating a crash rate from Kusto. There is no Kusto proxy for crash rate; do n ## 10. Weekly bucketing -`startofweek()` is **Sunday-aligned**, same as on the Broker side: -`startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. Print the distinct week values from the -first weekly query of the run and eyeball them. Off-by-one-week is the most common silent error -in weekly-bucketed KQL and it survives every other check in the pipeline. +Weekly trend queries are rolling 7-day buckets anchored at `curEnd`, not calendar weeks. Use the +view's time column with `bin_at(EventDate, 7d, datetime(<TREND_END>))` for scenario MVs (or the +appropriate raw-table time column for non-MV queries). The newest bucket is `[curEnd - 7d, curEnd)`, +which is exactly the same window as the report's displayed WoW numbers. -The 60-day trend deliberately **includes** the partial current week (it is the chart's final bar) -and excludes it from delta classification via -`bucket-trends.js --end=<startofweek(curEnd)> --include-partial-end`. -The 8-week sparkline series deliberately **excludes** it at the source. Both behaviours are -intentional and are not the same thing. +The 60-day trend filters `[curEnd - 60d, curEnd)`. Because 60 is not a multiple of 7, the **oldest** +bucket label falls before `<TREND_START>` and covers only 4 days of the filtered data. Invoke +`bucket-trends.js --start=<TREND_START> --end=<TREND_END>` so `--start` drops that oldest short +bucket and `--end` disables the legacy partial-end auto-drop heuristic. The 8-week sparkline series +uses the same `curEnd` anchor (`<SPARK_START> = curEnd - 56d`, `<SPARK_END> = curEnd`). + +> **⚠️ Historical note:** these queries used to use `startofweek()` calendar buckets and a separate +> classification cutoff. Do not restore that model; it can lag the rolling report window by up to a +> full week. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md b/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md index c09c2a6c..9e425a5f 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/docs/kusto-cheatsheet.md @@ -40,7 +40,7 @@ Time filter on materialized views is always **`EventInfo_Time`**. Use `PipelineI ## 3. Rolling 7-day WoW window (PRIMARY / attribution / latency) -**The report's primary window is a **rolling 7-day window** ending at start-of-day UTC on `-EndDate` (default: today), NOT a Sun→Sat calendar week.** Only the 60-day trend section (§ 7 below) still uses `startofweek()` bucketing. +**The report's primary window is a **rolling 7-day window** ending at start-of-day UTC on `-EndDate` (default: today), NOT a Sun→Sat calendar week.** The 60-day trend and sparkline sections use the same `curEnd` anchor with `bin_at(..., 7d, datetime(<TREND_END>))`, so classifier WoW and displayed WoW are aligned by construction. Canonical template — two rows per key (`week` in `{prevStart, curStart}`): @@ -69,11 +69,10 @@ Compute the placeholder values via `bootstrap-report.ps1` (which prints them to | `<CUR_END>` | `-EndDate` | `2026-07-09` | | `<CUR_START>` | `<CUR_END> - 7d` | `2026-07-02` | | `<PREV_START>` | `<CUR_END> - 14d` | `2026-06-25` | -| `<TREND_END>` | `<CUR_END>` (today) — literal 60d trend upper bound, exclusive | `2026-07-09` | | `<TREND_START>` | `<CUR_END> - 60d` | `2026-05-10` | -| `<TREND_CLASS_END>` | `startofweek(<CUR_END>)` — delta cutoff, `bucket-trends.js --end` | `2026-07-05` | -| `<SPARK_END>` | `startofweek(<CUR_END>)` — WoW-sparkline upper bound (8 complete weeks) | `2026-07-05` | -| `<SPARK_START>` | `<SPARK_END> - 56d` | `2026-05-10` | +| `<TREND_END>` | `<CUR_END>` (today) — literal 60d trend upper bound, exclusive, and the `bin_at` anchor | `2026-07-09` | +| `<SPARK_START>` | `<CUR_END> - 56d` — first label for the 8 rolling-week sparkline window | `2026-05-14` | +| `<SPARK_END>` | `<CUR_END>` — exclusive upper bound and `bin_at` anchor for sparklines | `2026-07-09` | --- @@ -113,14 +112,16 @@ The 7-dimension attribution slicing is **fully achievable from `ErrorStatsMetric Latency is stored as a TDigest sketch. **Percentiles are not additive** — averaging p95 across rows is meaningless. Always merge first: ```kql +let _startTime = datetime(<START>); +let _endTime = datetime(<END>); materialized_view('PerfStatsUpdated') -| where EventInfo_Time between ((_startTime) .. (_endTime)) +| where EventInfo_Time >= _startTime and EventInfo_Time < _endTime | where span_name in ('AcquireTokenSilent','GetAccounts','RemoveAccount','ProcessWebsiteRequest') | where span_status == 'OK' | summarize p50 = percentile_tdigest(tdigest_merge(responseTimeTDigest), 50, typeof(long)), p95 = percentile_tdigest(tdigest_merge(responseTimeTDigest), 95, typeof(long)), p99 = percentile_tdigest(tdigest_merge(responseTimeTDigest), 99, typeof(long)) - by week=startofweek(EventInfo_Time), span_name + by week = bin_at(EventInfo_Time, 7d, _endTime), span_name ``` **Note:** there is also a `PerfStatsMetrics` view, but it does **not** expose per-percentile columns directly — it has the merged TDigest. Use `PerfStatsUpdated` (preferred by the dashboard) and `percentile_tdigest(tdigest_merge(...), N, typeof(long))`. @@ -138,16 +139,18 @@ materialized_view('PerfStatsUpdated') --- -## 7. Week alignment — Kusto `startofweek()` is **Sunday-aligned** +## 7. Week alignment — use `bin_at()` anchored at `curEnd` -> **Scope:** only the 60-day trend section (§ 3 of the report / `bucket-trends.js` pipeline) still uses `startofweek()` weekly buckets. The primary/WoW section uses a rolling 7-day window — see § 3 of this cheatsheet. +All weekly-bucketed trend queries use `bin_at(EventInfo_Time, 7d, datetime(<TREND_END>))` (or the equivalent time column for the view) where `<TREND_END> = <CUR_END>`. The buckets step backward from `curEnd` in exact 7-day increments, so the newest bucket is `[curEnd - 7d, curEnd)` — exactly the same window as the report's headline WoW numbers. -If a user says "the week of May 2 → May 9", Kusto buckets it as `startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When writing weekly-bucketed queries (60-day trend, `wow-table-sparkline-series.kql`), **always confirm**: print the distinct `startofweek(EventInfo_Time)` values from your first query and verify the bucket labels match your intended range. Off-by-one-week is the #1 silent error in weekly-bucket queries. +For the literal-60-day trend window ending today (say `<CUR_END> = 2026-07-09`, so `<TREND_START> = 2026-05-10`), `bin_at(..., 7d, datetime(<TREND_END>))` produces labels: +`2026-05-07, 05-14, 05-21, 05-28, 06-04, 06-11, 06-18, 06-25, 07-02`. Because 60 is not a multiple of 7, the **oldest** bucket (`05-07`) only overlaps 4 days of the filtered window; pass `bucket-trends.js --start=<TREND_START>` to drop it, leaving 8 complete rolling weeks. The newest bucket (`07-02`) is complete and must stay in both the chart and the classifier. -For the literal-60-day trend window ending today (say `<CUR_END> = 2026-07-09`, so `<TREND_START> = 2026-05-10`), `startofweek()` produces buckets: -`2026-05-03, 05-10, 05-17, 05-24, 05-31, 06-07, 06-14, 06-21, 06-28, 07-05` — the first (`05-03`) is a partial start (window opens mid-week `05-10`) and is dropped by `bucket-trends.js --start`; the last (`07-05`) is the **partial current week** — it is charted as the final bar but excluded from delta classification via `--end=<TREND_CLASS_END>` (= `startofweek(today)` = `2026-07-05`) `--include-partial-end`. +Always invoke the bucketer with **both** `--start=<TREND_START> --end=<TREND_END>`. `--end` filters no rows because every bucket label is `< curEnd` by construction; its purpose is to disable the script's legacy partial-end auto-drop heuristic, which is wrong under rolling alignment. -The `wow-table-sparkline-series.kql` per-row sparklines are different: they stay on the last **8 complete** weeks (`<SPARK_END> = startofweek(today)`, exclusive, partial week filtered at the source) so no WoW row ends on a misleading partial dip. +The `wow-table-sparkline-series.kql` per-row sparklines use the same curEnd-anchored model: `<SPARK_START> = curEnd - 56d`, `<SPARK_END> = curEnd`, and 8 complete rolling weeks. There is no calendar-week alignment to reason about. + +> **⚠️ Historical note:** these queries used to rely on `startofweek()` calendar buckets. That lagged the rolling report window by up to a full week and made late-week regressions invisible to the noise gate. Do not reintroduce `startofweek()` here. --- @@ -156,16 +159,18 @@ The `wow-table-sparkline-series.kql` per-row sparklines are different: they stay ### 8a. Reliability (auth-only denominator) ```kql +let tEnd = datetime(<TREND_END>); +let tStart = tEnd - 56d; let all = materialized_view('SilentAuthStatsAllRequestsMetrics') - | where EventInfo_Time > ago(70d) + | where EventInfo_Time >= tStart and EventInfo_Time < tEnd | summarize allReq = sum(countRequests), allDev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time); + by week = bin_at(EventInfo_Time, 7d, tEnd); let ok = materialized_view('SilentAuthStatsRequestsWithoutExpectedErrorMetrics') - | where EventInfo_Time > ago(70d) + | where EventInfo_Time >= tStart and EventInfo_Time < tEnd | summarize okReq = sum(countRequests), okDev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time); + by week = bin_at(EventInfo_Time, 7d, tEnd); all | join kind=inner ok on week | project week, reqRel = round(100.0 * okReq / allReq, 3), @@ -176,22 +181,23 @@ all | join kind=inner ok on week **Auth-only device union** (Silent ∪ Interactive — what the report uses for the "real fleet" KPI). The natural reach for `hll_merge_array` to combine two pre-merged HLL sketches **does not exist in Kusto** (`SEM0260: Unknown function`). Instead, project the raw `countDevicesHll` rows from both views, `union` them, and `hll_merge` once at the end: ```kql +let tStart = datetime(<START>); +let tEnd = datetime(<END>); let s = materialized_view('SilentAuthStatsAllRequestsMetrics') - | where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) + | where EventInfo_Time >= tStart and EventInfo_Time < tEnd | project EventInfo_Time, countDevicesHll; let i = materialized_view('InteractiveAuthStatsAllRequestsMetrics') - | where EventInfo_Time between (datetime(<START>) .. datetime(<END>)) + | where EventInfo_Time >= tStart and EventInfo_Time < tEnd | project EventInfo_Time, countDevicesHll; union s, i | summarize authDev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time) -| where week < datetime(<END>) + by week = bin_at(EventInfo_Time, 7d, tEnd) | order by week asc ``` ### 8b. 60-day error trend (feeds `bucket-trends.js`) -Literal last 60 days ending today (`<TREND_START> = CUR_END - 60d`, `<TREND_END> = CUR_END`). Do NOT drop the partial current week here — it's the chart's final bar. Exclude it from deltas in the bucketer instead: `bucket-trends.js --end=<TREND_CLASS_END> --include-partial-end` where `<TREND_CLASS_END> = startofweek(today)`. +Literal last 60 days ending today (`<TREND_START> = CUR_END - 60d`, `<TREND_END> = CUR_END`). Bucket with `bin_at(EventInfo_Time, 7d, datetime(<TREND_END>))`, not calendar weeks. The newest bucket is `[CUR_END - 7d, CUR_END)` and is the same complete window used by the headline WoW numbers. Because the oldest bucket is the only short bucket, invoke the bucketer as `bucket-trends.js --start=<TREND_START> --end=<TREND_END>`. ```kql materialized_view('ErrorStatsMetrics') @@ -199,7 +205,7 @@ materialized_view('ErrorStatsMetrics') | where isnotempty(error_code) and error_code != 'success' | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), error_code + by week = bin_at(EventInfo_Time, 7d, datetime(<TREND_END>)), error_code | order by error_code asc, week asc ``` @@ -231,17 +237,19 @@ Run once each with the trailing dim set to: `span_name`, `calling_package_name`, ### 8e. Broker version share ```kql +let tEnd = datetime(<CUR_END>); +let tStart = tEnd - 21d; materialized_view('BrokerAdoptionStatsUpdated') -| where EventInfo_Time > ago(21d) +| where EventInfo_Time >= tStart and EventInfo_Time < tEnd | summarize req = sum(countRequests), dev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), broker_version + by week = bin_at(EventInfo_Time, 7d, tEnd), broker_version | order by week asc, req desc ``` > **⚠️ Share/snapshot views are window-parameterized — don't assume 60-day coverage.** The share queries — `BrokerAdoptionStatsUpdated` (version share, 8e), `AppStatsUpdated` (calling-app share), `SkuStatsUpdated` (SKU share), and the [`broker-version-share-wow.kql`](../queries/broker-version-share-wow.kql) / [`app-share.kql`](../queries/app-share.kql) templates — all take an explicit `<START>..<END>` (or `ago(Nd)`) window. They return **exactly the weeks you ask for, nothing more.** The adoption / app-share sections of the report typically only need a short **2–3 week** WoW window, so that's what these templates default to (`ago(21d)` above). > -> The trap: if you then try to draw a **9-week sparkline** for version/app/SKU adoption from that same short pull, you'll only have 2–3 real points and the rest will look flat or fabricated (the validator's low-peak `data-trend` heuristic may flag it). If you genuinely need a multi-week adoption sparkline, **re-run the share query with the full 60-day window** (`<START>` = reporting-Sunday − 56d) — don't pad a short result. If you don't need the sparkline, don't build one from a 2–3 week pull and pretend it's a trend. +> The trap: if you then try to draw an **8-week sparkline** for version/app/SKU adoption from that same short pull, you'll only have 2–3 real points and the rest will look flat or fabricated (the validator's low-peak `data-trend` heuristic may flag it). If you genuinely need a multi-week adoption sparkline, **re-run the share query with the full rolling sparkline window** (`<START>` = `curEnd - 56d`, `<END>` = `curEnd`) — don't pad a short result. If you don't need the sparkline, don't build one from a 2–3 week pull and pretend it's a trend. --- @@ -249,7 +257,7 @@ materialized_view('BrokerAdoptionStatsUpdated') | Script | Purpose | |---|---| -| [`bucket-trends.js`](bucket-trends.js) | Bucket every error code into regression / spike / improvement / flat across an N-week window. Pass `--end=YYYY-MM-DD` (= `startofweek(today)`, exclusive) to exclude the partial in-progress week from delta classification, plus `--include-partial-end` to still chart it as the final bar. | +| [`bucket-trends.js`](bucket-trends.js) | Bucket every error code into regression / spike / improvement / flat across the 8 complete rolling weeks ending at `curEnd`. Pass **both** `--start=<TREND_START> --end=<TREND_END>`; `--start` drops the 4-day oldest bucket, and `--end` disables the legacy partial-end auto-drop heuristic without filtering rows. | | [`agg.js`](agg.js) | Per-error per-dim top-N rollup with WoW deltas. Feeds spike-attribution dim blocks. | | [`summarize-attribution.js`](summarize-attribution.js) | Roll up 7-dim attribution slices per (error_code, week) — feeds the spike-attribution cards | | [`queries/`](queries/) | Canonical KQL templates, one per query — see [`queries/README.md`](queries/README.md) | diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/authapp.md b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/authapp.md index 0d6b108a..794d97bb 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/authapp.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/authapp.md @@ -109,21 +109,7 @@ both sides of the init↔results join or the two funnels quietly contaminate eac rate, overall **Unknown rate**, Broker-API success rate, crashes per 1,000 devices. WoW delta on each, inline SVG sparkline on each. -2. **Scenario scoreboard** — one table, **all 13 scenarios, every week, no exceptions**. Even - scenarios that did not move get a row; a silent scenario disappearing from the table is - indistinguishable from a scenario that was never checked. Columns: scenario, initiated, - success rate, Δ success (pts), failure rate, unknown rate, Δ unknown (pts), devices, 8-week - sparkline, status pill. Rows under the volume floor carry a `low-volume` tag. - - > **The 4 push-notification rows cannot fill the success/failure/unknown columns — that is - > expected, not a gap.** PN has no success/failure/Unknown model (see the outcome-model warning - > above; `Denied` is a healthy outcome, so a "failure rate" would be a lie). For the 4 PN rows - > put the **completion rate** in the success-rate column, the **error rate** in the failure - > column, and a literal `n/a` in the unknown-rate and Δ-unknown cells. Do **not** leave the cells - > blank (blank reads as "not measured") and do **not** synthesise an Unknown bucket for them. - > Footnote the table once: *"PN scenarios report completion/error; they have no Unknown state."* - -3. **Needs attention** — callouts using the `.item` flat-row pattern, ordered by **novelty, not +2. **Needs attention** — callouts using the `.item` flat-row pattern, ordered by **novelty, not volume** (see Step 4b). Render the classifier's `attention` set (`NEW` + `ACCELERATING`) at the top level, plus **at most 2** wins, and nothing else; `ONGOING` goes in a collapsed fold. Budget **≤ 8 visible rows total, wins included** (`validate-report.ps1` check 17 warns above it @@ -135,14 +121,16 @@ both sides of the init↔results join or the two funnels quietly contaminate eac the only multi-week bucket that stays visible, because "is it getting worse?" is the one question a known issue can still answer usefully. Delete the callout if the set is empty. - > **When the classifier and the headline delta disagree, keep the row here and show both.** - > The classifier's "not falling" gate runs on **complete Sun–Sat calendar weeks**; the headline - > percentage-point delta runs on the **rolling 7-day** window. Different bases, and they - > legitimately disagree. That is not a reason to demote the row, rename the group, or hedge the - > heading — keep it exactly **"Getting worse"** and resolve it *in the row body*: *"Down 2.1 pp - > across the last three complete weeks; the rolling window shows +0.4 pp as the slide flattens. - > Still 3.8 pp below its own 60-day median."* The sparkline settles it visually. Do **not** - > invent a "needs verification" group. + > **The classifier's WoW and the headline delta are the SAME basis** — both the rolling 7-day + > window, since the 60-day trend's final `bin_at` bucket *is* that window. They should agree in + > sign. If they don't, `--start`/`--end` were passed wrong; fix the invocation instead of + > hedging the report. (They used to run on different bases — calendar weeks vs rolling — and + > could disagree by 100 points. That was the bug this section used to describe as a feature.) + > + > A row can still be `ACCELERATING` while its **multi-week** slide outpaces this week's step. + > Keep the heading exactly **"Getting worse"** and resolve it *in the row body*: *"Down 2.1 pp + > over three weeks, −0.3 pp this week as the slide flattens. Still 3.8 pp below its own 60-day + > median."* The sparkline settles it visually. Do **not** invent a "needs verification" group. - **🔵 Ongoing / known** — label `ONGOING`: degraded but level or easing. **Collapse into a `<details class="fold">`** with a one-line summary ("N scenarios still below baseline, none accelerating") and each row's `weeksElevated`. Still in the report, no longer competing with @@ -179,6 +167,49 @@ both sides of the init↔results join or the two funnels quietly contaminate eac > and whether it's news. One generic sentence repeated across rows makes the section unreadable; > `validate-report.ps1` fails the report for it. +3. **Scenario scoreboard** — one table, **all 13 scenarios, every week, no exceptions**. Even + scenarios that did not move get a row; a silent scenario disappearing from the table is + indistinguishable from a scenario that was never checked. Columns: scenario, initiated, + success rate, Δ success (pts), failure rate, unknown rate, Δ unknown (pts), devices, 8-week + sparkline, status pill. Rows under the volume floor carry a `low-volume` tag. + + > **The 4 push-notification rows cannot fill the success/failure/unknown columns — that is + > expected, not a gap.** PN has no success/failure/Unknown model (see the outcome-model warning + > above; `Denied` is a healthy outcome, so a "failure rate" would be a lie). For the 4 PN rows + > put the **completion rate** in the success-rate column, the **error rate** in the failure + > column, and a literal `n/a` in the unknown-rate and Δ-unknown cells. Do **not** leave the cells + > blank (blank reads as "not measured") and do **not** synthesise an Unknown bucket for them. + > Footnote the table once: *"PN scenarios report completion/error; they have no Unknown state."* + + > **⚠️ MANDATORY — every red/amber pill must be reconciled.** The scoreboard colours a row from + > its own rolling delta; "Needs attention" is populated from the **novelty** classifier. Those + > answer different questions, so a row can be legitimately red here and legitimately absent + > there — but the reader sees a contradiction and concludes the report is broken. + > + > Real precedent from the 2026-08-01 run: **`Passkey WebAuthN Registration`** shipped carrying + > `tag-bad` (−1.27 pts, the worst delta in the table) while the attention section directly above + > read *"Quiet week — 0 NEW or ACCELERATING"*. Both were true. That scenario peaks at ~732 + > bad-outcome devices, **under the 1,000-device peak floor**, so it is structurally excluded from + > classification and can carry a red pill forever without ever being eligible for attention. No + > bucketing change fixes this — it has to be *explained*. + > + > So for each `tag-bad` / `tag-warn` row, either promote it into attention, **or** emit a muted + > `<div class="reconcile-note">` naming it and giving the reason. Test the reasons **in this + > order** and stop at the first that applies: + > 1. **Below the classification floor** — peak < 1,000 bad-outcome devices in the 60-day window. + > 2. **Within its own normal band** — the move is inside its 8-week variance (check the sparkline). + > 3. **ONGOING and flat** — already known and not accelerating; it lives in the fold. + > + > ```html + > <div class="reconcile-note">Flagged in the scoreboard but not escalated: <b>Passkey WebAuthN + > Registration</b> and <b>Entra PSI PN+CFA</b> peak below the 1,000-device classification floor, + > so novelty is not computed for them. Watch the sparkline; re-evaluate if volume clears the floor.</div> + > ``` + > + > Keep it visually muted — it is a footnote, not a finding, and must not compete with `.callout`. + > `validate-report.ps1` **check 19 hard-fails** any red/amber pill that is neither promoted nor + > named in a `.reconcile-note`. + 4. **60-day per-scenario trend** — weekly-bucketed sparkline per scenario, first→last delta, and a classification pill (regression / spike / improvement / flat) from `bucket-trends.js`. @@ -277,8 +308,8 @@ Run [`scenario-60d-trend.kql`](../queries/authapp/scenario-60d-trend.kql), then: ```pwsh node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js $data\scenario-60d.json ` - --key=scenario --metric=devs --end=<startofweek(curEnd)> --include-partial-end ` - --peak-floor=1000 --summary + --key=scenario --metric=devs --start=<TREND_START> --end=<TREND_END> ` + --family-sep=none --peak-floor=1000 --summary ``` The query maps `errs` / `devs` to **bad outcomes** (`Failed + Unknown`), so the bucketer's @@ -286,8 +317,21 @@ The query maps `errs` / `devs` to **bad outcomes** (`Failed + Unknown`), so the `--metric=devs` and `--metric=reqs` and report the union of what each flags — a scenario where device count is flat but event count explodes is a retry storm and only shows on one axis. -Do **not** filter the partial current week at the source; `--end` excludes it from the delta math -while `--include-partial-end` keeps it as the chart's final bar. +**Pass both `--start` and `--end`** (`<TREND_START>` = `curEnd − 60d`, `<TREND_END>` = `curEnd`; +`bootstrap-report.ps1` prints both). The query buckets with `bin_at(…, 7d, <TREND_END>)`, so the +newest bucket **is** the report's 7-day window and every bucket is a complete 7 days — there is no +partial end bar to exclude, and `--include-partial-end` / `TREND_CLASS_END` are obsolete. `--end` +filters no rows; its job is to **disable the partial-end auto-drop heuristic**, which would +otherwise be free to discard a genuine collapse in the newest bucket. The script warns if you omit +it — treat that warning as an error. `--start` drops the one genuinely partial bucket, which under +this anchoring is the **oldest** (`curEnd − 63d`, 4 days), leaving 8 clean rolling weeks. + +> **⚠️ `--peak-floor=1000` silently excludes low-volume scenarios from classification entirely** — a +> scenario peaking below it can never appear in "Needs attention" no matter how sharply it moves, +> while the scoreboard will still colour its pill red. That mismatch is not a bug in either +> component; it is why the **reconciliation rule** exists (see section 3 of § Required sections). +> After this step, cross-check the classifier's key list against the scoreboard and note which rows +> were dropped by the floor — you will need them for the `.reconcile-note`. ### Step 4b — Classify novelty (mandatory) @@ -327,17 +371,17 @@ classifier did not — that is the defect this step exists to prevent. > **Which series get classified: the outcome funnels only — the PN funnel is NOT run through the > classifier.** Feed `classify-novelty.js` the **9 outcome-funnel bad-outcome series** and nothing -> else. The 13 scenarios in Section 5's scoreboard are **9 outcome funnels + 4 push-notification +> else. The 13 scenarios in Section 3's scoreboard are **9 outcome funnels + 4 push-notification > families**; only the 9 are classifiable. The PN families (Section 7) are deliberately excluded for > two reasons: their `FinalResult` set has **two shapes** across the window so a weekly series is not > comparable week-to-week, and **`Denied` is a healthy outcome** — a rising `Denied` share is a user > correctly rejecting a prompt, which the classifier would read as a regression. Never let a PN -> family appear in Section 3's `attention` set. +> family appear in Section 2's `attention` set. > > PN still gets trend treatment, just not novelty classification: chart each family's **completion > rate** in Section 7 with its own sparkline and report the WoW delta there. If a PN family moves > enough to be this week's story, say so in Section 7 and, if it warrants top-level visibility, -> reference it from the Section 1 executive summary — not by inserting it into Section 3. +> reference it from the Section 1 executive summary — not by inserting it into Section 2. `weeksElevated` is **derived from the 9-week series, never persisted** — it counts consecutive recent weeks above the early-window baseline, so it is identical on any machine and needs no state file. @@ -496,11 +540,11 @@ Then verify by hand: > > The rules below are Authenticator-specific and do **not** transfer to the Broker playbook. -- **Novelty classification is mandatory, and Section 3 is ordered by it — never by volume.** Run +- **Novelty classification is mandatory, and Section 2 is ordered by it — never by volume.** Run [`classify-novelty.js`](../scripts/classify-novelty.js) (Step 4b) and lead with `NEW`. Volume-ranking the attention section is a known, reported defect — it puts flat-but-huge rows above real step-changes. If `NEW` is empty, write "nothing new this week"; do not backfill it with `ONGOING` scenarios. -- **Section 3's visible rows are the classifier's `attention` set plus at most 2 wins — nothing else.** +- **Section 2's visible rows are the classifier's `attention` set plus at most 2 wins — nothing else.** `NEW` + `ACCELERATING` visible with sparklines; `ONGOING` inside a collapsed `<details class="fold">`. Budget **≤ 8 visible rows total, wins included** (check 17 counts wins). The Broker report this replaces shipped 13 visible rows with zero charts while the section below it carried 38 — the reader @@ -522,7 +566,7 @@ Then verify by hand: shape on both apps. Report the absolute level and its position against the 60-day median instead. AuthApp scenarios normally sit at cv 0.02–0.2, so a genuinely `VOLATILE` scenario usually means flapping instrumentation — call that out rather than reporting it as a user-facing failure. -- **No boilerplate in Section 3.** Every row body must be specific to that scenario — what moved, from what +- **No boilerplate in Section 2.** Every row body must be specific to that scenario — what moved, from what to what, and whether it's news. One generic sentence repeated across rows makes the section unreadable and `validate-report.ps1` fails the report for it. - **Never `dcount_hll` / `hll_merge` / `percentile_tdigest` / `materialized_view('…')` / @@ -578,9 +622,13 @@ Then verify by hand: - [ ] Volume floor applied — `low-volume` rows tagged, excluded from the regression callout, and any scenario that *dropped into* low-volume is flagged as its own finding. - [ ] **60-day bucketing run on both axes** (`--metric=devs` AND `--metric=reqs`, `--key=scenario`), - union of regressions reported, partial week charted but excluded from delta classification. + union of regressions reported. Both `--start` and `--end` passed, so the newest `bin_at` bucket + is the report's own 7-day window and the classifier grades the period the tables print. +- [ ] **Every red/amber scoreboard pill reconciled** — each `tag-bad`/`tag-warn` row is either + promoted into "Needs attention" or named in a `.reconcile-note` with its reason (below the + 1,000-device floor / within its normal band / ONGOING-flat). Validator check 19 enforces this. - [ ] **Novelty classification run** ([`classify-novelty.js`](../scripts/classify-novelty.js), Step 4b, - `--family-sep=none`). Section 3 leads with `NEW`, `ACCELERATING` sits in the 🟠 Getting-worse + `--family-sep=none`). Section 2 leads with `NEW`, `ACCELERATING` sits in the 🟠 Getting-worse callout, `ONGOING` is inside a collapsed fold, and no `VOLATILE`/`RECOVERY` row headlines a percentage. Every row body is specific — no sentence repeats across rows. - [ ] **Attention section is short and charted.** Visible rows == the classifier's `attention` set diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/broker.md b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/broker.md index 9d7d8a46..00b9cf89 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/broker.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/playbooks/broker.md @@ -61,15 +61,17 @@ over the window, ≥ 10% above its own median, and not falling WoW). This is the "known issue is deteriorating" bucket and it is the *only* multi-week category that stays visible. - > **When the classifier and the headline WoW disagree, keep the row here and show both numbers.** - > The classifier's "not falling" gate runs on **complete Sun–Sat calendar weeks**; the report's - > headline `Δ WoW` runs on the **rolling 7-day** window. These are different bases and they - > legitimately disagree — a code can be `ACCELERATING` on calendar weeks while showing a small - > rolling-window decline. That is *not* a reason to demote it, rename the group, or hedge the - > heading. Keep the group heading exactly **"Getting worse"**, and resolve it *in the row body*: - > *"Up 18% over the last three complete weeks; the rolling 7-day window shows −4% as the ramp - > flattens. Still ~30% above its own 60-day median — watch, don't close."* The sparkline settles - > it visually, which is why the row has one. Do **not** invent a "needs verification" group. + > **The classifier's WoW and the headline WoW are the SAME number** — both are the rolling + > 7-day window, since the trend's final `bin_at` bucket is that window. They should agree to + > within HLL noise. A sign disagreement means `--start`/`--end` were passed wrong; go fix the + > invocation rather than writing a hedge into the report. (Before the `bin_at` fix the two ran + > on different bases and could disagree by 100 points — that was the bug, not a feature.) + > + > A row can still be `ACCELERATING` while its **multi-week** climb outpaces this week's step — + > e.g. up 18% over three weeks but +2% in the last one. Keep the group heading exactly + > **"Getting worse"** and resolve it *in the row body*: *"Up 18% over three weeks, +2% this + > week as the ramp flattens. Still ~30% above its own 60-day median — watch, don't close."* + > The sparkline settles it visually. Do **not** invent a "needs verification" group. 3. **🔵 Ongoing / known** — label `ONGOING`: elevated but flat. **These go inside a collapsed `<details class="fold">`**, summarised by one line ("N codes remain elevated, none accelerating"). They are still in the report — a reader can open the fold — but they no longer compete with the @@ -257,11 +259,20 @@ materialized_view('ErrorStatsMetrics') | where isnotempty(error_code) and error_code != 'success' | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), error_code + by week = bin_at(EventInfo_Time, 7d, datetime(<TREND_END>)), error_code | order by error_code asc, week asc ``` -**Do NOT filter the partial in-progress week here.** The chart wants it as the final bar (the window ends today). The partial week is excluded from the regression/improvement **delta math** by `bucket-trends.js` via `--end=<TREND_CLASS_END> --include-partial-end` (see 3c), not at the source — a partial week driving the delta would read as a fake −99% improvement, which is exactly why classification and display are split in the JS. +**`bin_at`, not `startofweek` — this is load-bearing.** Anchoring the 7-day bins at `<TREND_END>` +(= `curEnd`) makes the newest bucket exactly `[curEnd − 7d, curEnd)`, i.e. **the report's own WoW +window**, so the novelty classifier grades the same period the tables print. Under the old +`startofweek()` bucketing the classifier lagged by up to a week and silently suppressed real risers +(`authorization_pending` +63.2% in the report, −37.1% to the classifier). See the ⚠️ block in Step 4. + +**Every bucket here is a complete 7 days**, so there is no partial end bar and nothing to exclude +from the delta math — `--include-partial-end` and `TREND_CLASS_END` are obsolete. Because 60 isn't a +multiple of 7, the *oldest* bucket (`curEnd − 63d`) is the 4-day stub; `--start` drops it, leaving +8 clean rolling weeks. #### 3b. Per-error-type trend (same rigor) @@ -272,7 +283,7 @@ materialized_view('ErrorStatsMetrics') | where isnotempty(unified_error_type) | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), unified_error_type + by week = bin_at(EventInfo_Time, 7d, datetime(<TREND_END>)), unified_error_type | order by unified_error_type asc, week asc ``` @@ -284,18 +295,22 @@ materialized_view('ErrorStatsMetrics') ```pwsh # Error codes — by devices, then by requests. -# TREND_START = curEnd - 60d (literal 60d start) -# TREND_CLASS_END = startofweek(today) ("Trend delta cutoff" printed by bootstrap) -# --include-partial-end charts the current partial week while excluding it from deltas. -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --metric=reqs +# TREND_START = curEnd - 60d TREND_END = curEnd (both printed by bootstrap-report.ps1) +# Pass BOTH --start and --end. See the note below for why --end is not optional. +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_END> +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <codes.json> --start=<TREND_START> --end=<TREND_END> --metric=reqs # Error types — by devices, then by requests (note --key) -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --key=unified_error_type -node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end --key=unified_error_type --metric=reqs +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_END> --key=unified_error_type +node .github\skills\oncall-weekly-telemetry-report\assets\scripts\bucket-trends.js <types.json> --start=<TREND_START> --end=<TREND_END> --key=unified_error_type --metric=reqs ``` -`--end` is `<TREND_CLASS_END>` = `startofweek(today)` (exclusive) — the Sunday that opens the current in-progress week. Weeks at or after it (the partial current week) are excluded from delta classification; `--include-partial-end` keeps that week in the emitted `series` so the chart ends today. The script also auto-detects partial end-buckets and warns if `--end` is omitted, but passing it explicitly is safer. +**⚠️ `--end=<TREND_END>` is mandatory even though it filters nothing.** Every bucket label is +`< curEnd` by construction, so `--end` removes no data — its job is to **disable the partial-end +auto-drop heuristic**, which is guarded by `if (!endArg …)`. Under rolling alignment the newest +bucket is genuinely complete, so leaving the heuristic armed means a real 70% collapse could be +discarded as "looks partial". The script warns if you omit `--end`; treat that warning as an error. +`--include-partial-end` is a retained no-op — do not add it to new invocations. Take the **union** of all four regression sets. Both `error_code` and `error_type` regressions get a spike-attribution card in Step 5. @@ -412,21 +427,39 @@ Three codes, each flat for seven straight weeks, all stepping up in the *same* w **Families.** The classifier clusters keys sharing a prefix before `_` when ≥2 members share the same label. Report a family as ONE row. Error *types* are CamelCase and produce no families under `_` — that is correct, not a bug. -**⚠️ Two different WoW bases exist — do not conflate them.** The report headline ΔWoW is a **rolling 7-day** window (`[CUR_START, CUR_END)` vs the 7 days before). The classifier's `WoW` is **calendar Sun–Sat weeks**. They legitimately disagree — `authorization_pending` read **+3.5%** rolling and **−37.1%** weekly on the same data. Use novelty as *history and context* ("flat for seven weeks, first step this week"), **never** as a competing delta number, or the report will appear to contradict its own tables. +**⚠️ The classifier and the report now share ONE basis. This used to be a bug.** Both the headline +`Δ WoW` and the classifier's `WoW` are computed on the **same rolling 7-day window** +(`[CUR_START, CUR_END)` vs the 7 days before), because the 60-day trend is bucketed with +`bin_at(t, 7d, curEnd)` and its final bucket **is** that window. **If a classifier `WoW` ever +disagrees in sign or by more than HLL noise from the number in the table, the pipeline is +misconfigured — stop and check that `--start`/`--end` were passed as bootstrap printed them.** + +> **Why this warning exists.** Buckets used to be Sun–Sat calendar weeks cut off at +> `startofweek(curEnd)`, and the two bases were documented as "legitimately disagreeing". They did +> not legitimately disagree — the calendar basis lagged the report by up to a full week and was blind +> to anything that turned in the last ~6 days. On the 2026-08-01 run the classifier's current week +> was 07/19–07/26 against a report window of 07/25–08/01, **one day of overlap**: +> +> | code | report ΔWoW | old classifier WoW | old verdict | +> |---|---|---|---| +> | `authorization_pending` | **+63.2%** (171,897 → 280,572) | −37.1% | ONGOING — "do not re-triage" | +> | `expired_token` | **+26.7%** (86,255 → 109,251) | −51.0% | ONGOING — "do not re-triage" | +> +> Both were real risers the on-call engineer had already spotted by hand, and the report silently +> suppressed both. Re-bucketing on `bin_at` promoted them to ACCELERATING **and** demoted +> `access_denied` — an ONGOING-worthy code the old basis had wrongly promoted (actually −53.2%). +> The attention set went 4 → 5 keys, not 4 → 15. **Do not reintroduce `startofweek()` here.** -> **The division of labour, stated plainly so you do not have to derive it:** +> **What the classifier still contributes.** Selection and narrative, not numbers: > -> | Use the **rolling 7-day** numbers for… | Use the **calendar-week** classifier for… | +> | Take from the query results | Take from `classify-novelty.js` | > |---|---| -> | Every KPI tile, table cell, and Δ% chip | Which rows are promoted (`attention` set) | -> | Any number a reader can see | Which label a row carries (NEW / ACCELERATING / …) | -> | The sentence "X rose N% this week" | The sentence "…and it has been climbing for six weeks" | +> | Every KPI tile, table cell, and Δ% chip | Which rows are promoted (the `attention` set) | +> | The sentence "X rose N% this week" | Which label a row carries (NEW / ACCELERATING / …) | +> | | The sentence "…and it has been climbing for six weeks" | > -> **Rule: every *number* in the report comes from the rolling window; the classifier contributes -> *selection and narrative*, never a figure.** The one place the two meet is a row that is -> `ACCELERATING` on calendar weeks while the rolling delta is flat or negative — keep it in -> "Getting worse", keep the heading verbatim, and resolve it in the row body by stating both -> numbers and letting the sparkline settle it. Do not invent a hedged sub-group for these. +> The classifier's job is to answer *"is this new?"*, which a single delta cannot. It is no longer a +> second source of truth for *"how much did it move?"* — there is only one answer to that now. --- @@ -589,13 +622,16 @@ Do this section in three parts. Traffic changes (up *or* down) need the same lev ```kql materialized_view('BrokerAdoptionStatsUpdated') -| where EventInfo_Time > ago(70d) +| where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) | summarize totalReq = sum(countRequests), totalDev = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time) + by week = bin_at(EventInfo_Time, 7d, datetime(<TREND_END>)) | order by week asc ``` +Same `bin_at` anchoring as 3a/3b, so the final bucket is the report's WoW window and this traffic +series lines up bucket-for-bucket with the error trends you compare it against. + For each of the following, report direction + magnitude: - Total requests (WoW %, 60d %) - Total devices (WoW %, 60d %) @@ -680,7 +716,7 @@ The validator hard-fails on: 6. **Chartless KPI grid** — if more than half the `.kpi` tiles lack a `data-spark` element (catches the v7 regression where the body was rebuilt without sparklines). Also warns when total chart count (sparks + trends + inline svgs) is < 15. 7. **Code-attribution depth** — each `.attr-card`'s "Code attribution" block must contain an `Originator` row (proxy for the full 8-field structure: Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step). Catches the v7-third-pass regression where cards shipped with a `pr-list`-only stub. 8. **Attribution-card layout guards (v8)** — the CSS must define `.attr-card { margin-bottom: 16px }` AND `.dim-row` overflow rules (`text-overflow: ellipsis` + `min-width: 0`). Catches the "cards touching" and "text bleeding out of dim boxes" regressions from a stale `<head>` block. -9. **Fabricated-sparkline heuristic (v8)** — warns when a `data-trend` array's peak value is < 100 (almost certainly hand-rolled rather than sourced from real data). See [`assets/queries/wow-table-sparkline-series.kql`](../queries/wow-table-sparkline-series.kql) for the canonical KQL that pulls real 8-week series for every code in the WoW tables. Its `<SPARK_START>` / `<SPARK_END>` tokens are the last **8 complete** Sun-Sat weeks (`<SPARK_END>` = `startofweek(today)`, exclusive) — deliberately distinct from the trend-chart's `<TREND_START>` / `<TREND_END>` (literal last 60 days ending today). Per-row sparklines stay on complete weeks so a partial final point doesn't create a misleading dip in every WoW row. +9. **Fabricated-sparkline heuristic (v8)** — warns when a `data-trend` array's peak value is < 100 (almost certainly hand-rolled rather than sourced from real data). See [`assets/queries/wow-table-sparkline-series.kql`](../queries/wow-table-sparkline-series.kql) for the canonical KQL that pulls real 8-week series for every code in the WoW tables. Its `<SPARK_START>` / `<SPARK_END>` tokens are the last **8 complete rolling weeks** anchored at `curEnd` (`<SPARK_END>` = `curEnd`, exclusive; `<SPARK_START>` = `curEnd − 56d`) — the same `bin_at` basis as the trend chart, just a shorter span. Every point is a full 7 days, so no row can end on a misleading partial dip. Then: - **Run the visual smoke test (recommended)** — catches rendered-layout bugs that pure HTML/CSS validation can't see: @@ -706,8 +742,16 @@ Then: > They apply here too and are NOT repeated below: never carry a telemetry number forward between > runs · never hardcode the Generated date · never compose report HTML via a PowerShell `@'...'@` > heredoc (UTF-8 strip) · never bulk-regex-edit balanced HTML · no `devs`/`reqs` in user-facing -> text · same-end-date collision requires an explicit delta statement · no separate Markdown -> summary · never commit the report. **Read them before writing any HTML.** +> text · **every red/amber table pill is either promoted into attention or explained in a +> `.reconcile-note`** (validator check 19) · same-end-date collision requires an explicit delta +> statement · no separate Markdown summary · never commit the report. +> **Read them before writing any HTML.** +> +> The reconciliation rule bites on Broker too, for the same structural reason it does on the +> Authenticator: a code whose 60-day **peak** sits under the 10,000-device floor is excluded from +> classification outright, so it can hold a red pill indefinitely while attention says "quiet week". +> Now that both bases are rolling, disagreement on *direction* is a bug — what legitimately remains +> is disagreement on **classifiability**. Say which it is. > > The rules below are Broker-specific and do **not** transfer to the Authenticator playbook. @@ -717,7 +761,7 @@ Then: - **Never sum percentiles.** Latency is a TDigest sketch — `percentile_tdigest(tdigest_merge(responseTimeTDigest), N, typeof(long))` only. - **Always apply `MergeAccountType` / `MergeIsSharedDevice` / `MergeUiRequiredExceptions`** so this report agrees with the dashboard. - **Confirm the week bucket label matches the user's intent** before writing the rest of the queries (Sunday-aligned). -- **Do NOT filter the partial in-progress week at the source in the 60-day trend queries** — the chart ends today and wants that partial week as its final bar. Exclude it from the regression/improvement **delta math** instead by running `bucket-trends.js --end=<startofweek(today)> --include-partial-end`: the `--end` cutoff drops the partial week from first/last/delta classification while `--include-partial-end` keeps it in the emitted `series`. Skipping `--end` (or the cutoff) would make `bucket-trends.js` show every error as a fake −99% improvement. The per-row `wow-table-sparkline-series.kql` is the exception — it keeps 8 complete weeks (`<SPARK_END>` = `startofweek(today)`, with the partial week filtered at the source) so no WoW row ends on a misleading partial dip. +- **Bucket the 60-day trend with `bin_at(EventInfo_Time, 7d, datetime(<TREND_END>))`, never `startofweek()`.** Anchoring at `curEnd` makes the newest bucket exactly the report's WoW window, so the noise gate grades the period the tables print. `startofweek()` bucketing lagged by up to 6 days and structurally suppressed anything that turned late in the window — that is how `authorization_pending` shipped as "ONGOING, do not re-triage" while the report showed it up 63.2%. Every bucket is now a complete 7 days (the 4-day stub is the *oldest* bucket and `--start` drops it), so there is nothing to exclude from the delta math: `--include-partial-end` and `TREND_CLASS_END` are obsolete. **Always pass both `--start` and `--end` to `bucket-trends.js`** — `--end` filters no rows but disables the partial-end auto-drop heuristic, which would otherwise be free to discard a real collapse. `wow-table-sparkline-series.kql` uses the same `bin_at` basis over 8 weeks. - **Originator pre-check is mandatory.** A card cannot claim `Originator: Broker` without first running [`assets/queries/error-message-and-location.kql`](../queries/error-message-and-location.kql) and reading the throw site + top 3 `error_message` strings. If the throw site is in `common/ExceptionAdapter.{getExceptionFromTokenErrorResponse, exceptionFromAuthorizationResult}` AND the message starts with `AADSTS`, the originator is **eSTS, not broker** — see the AADSTS reference in [`assets/docs/kusto-cheatsheet.md`](../docs/kusto-cheatsheet.md). - **WoW-movers pass is mandatory.** The 60d bucketer's `--peak-floor` silently drops sub-10K-device codes, so [`assets/queries/wow-movers.kql`](../queries/wow-movers.kql) MUST be run as a separate pass for both `error_code` and `error_type` (per Step 3d). Its output is **merged into the single regression callout** and then grouped by Step 3e's novelty labels. Do not render a separate "emerging" callout. Skipping the pass is how the Apr 26 `Failed to parse JWT` spike (7 → 3,461 devs over 7 weeks) hid for two reports running. - **Novelty classification is mandatory, and Section 2 is ordered by it — never by volume.** Run [`classify-novelty.js`](../scripts/classify-novelty.js) (Step 3e) and lead with `NEW`. Ranking the attention list by device count is a known, reported defect: it put `IntuneAppProtectionPolicyRequiredException` (ΔWoW **+0.1%**, classifier says `ONGOING` and *falling*) at #1 while the genuinely new `ipc_*` family sat at #6/#9/#10. If the `NEW` bucket is empty, write "nothing new this week" — do not backfill it with `ONGOING` items. @@ -755,7 +799,7 @@ Then: - [ ] New `oncall-wow-report-YYYY-MM-DD.html` (where `YYYY-MM-DD` is the resolved `curEnd` — the end-date of the rolling 7-day window) exists at `$env:USERPROFILE\android-oce-reports\` (NOT at repo root). If a file for this end-date already existed, the chat session explicitly stated what changed before regenerating. - [ ] All sections present and populated (incl. 🚚 Traffic Attribution — even if “None this week”) -- [ ] **60-day trend bucketing run on the full cross-product** — `{error_code, error_type} × {devices, requests}` = 4 runs — union of regressions reported. Per-request retry storms (e.g. small device pool, exploding request count) are flagged on both axes. Source KQL spans the literal last 60 days ending today (no source-side partial-week filter); the partial current week is excluded from delta classification via `bucket-trends.js --end=<startofweek(today)> --include-partial-end` and charted as the final bar. +- [ ] **60-day trend bucketing run on the full cross-product** — `{error_code, error_type} × {devices, requests}` = 4 runs — union of regressions reported. Per-request retry storms (e.g. small device pool, exploding request count) are flagged on both axes. Source KQL spans the literal last 60 days ending today and buckets with `bin_at(…, 7d, <TREND_END>)`, so the newest bucket **is** the report's WoW window; every `bucket-trends.js` invocation passed **both** `--start` and `--end`. - [ ] **WoW-movers pass run** ([`wow-movers.kql`](../queries/wow-movers.kql)) for BOTH `error_code` and `error_type`. Its output rows are **merged into the single regression callout in Section 2**. Every row carries throw-site, dominant message, originator, and a next step. If the callout is empty (rare), render "None this week" rather than omit. - [ ] **Novelty classification run** ([`classify-novelty.js`](../scripts/classify-novelty.js)) on every `bucket-trends.js` sidecar. Section 2 is grouped 🆕 New → 🟠 Getting worse → 🔵 Ongoing (collapsed fold) → 🔁 Volatile → ↩️ Recovery, **not** sorted by device count. No `VOLATILE`/`RECOVERY` row headlines a percentage. Families are reported as one row. Every row body is specific — no sentence repeats across rows. - [ ] **Attention section is short and charted.** Visible rows == the classifier's `attention` set (`NEW` + `ACCELERATING`), ≤ 8 of them, each with an `.item-spark` 9-week sparkline. `ONGOING` rows live inside a collapsed fold. If `quietWeek: true`, the quiet-week banner is shown and nothing was promoted to fill the gap. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql index 1e2c6270..d6a3c204 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-codes.kql @@ -1,32 +1,38 @@ // 60-day per-error-code trend (LITERAL last 60 days ending today). // -// The 60-day trend spans the literal last 60 days ending at -// curEnd (today), so BOTH bounds move with the report date. The trend CHART -// includes the current in-progress week as its final (partial) bar. Per-week -// bucketing is still Sun-Sat aligned (Kusto startofweek() is Sunday-based), so the -// first and last buckets are partial by construction. +// ⚠️ BUCKETS ARE ROLLING 7-DAY WINDOWS ANCHORED AT <TREND_END>, **NOT** Sun-Sat +// calendar weeks. `bin_at(t, 7d, datetime(<TREND_END>))` walks backwards from +// curEnd in exact 7-day steps, so the FINAL bucket is [curEnd-7d, curEnd) -- +// byte-for-byte the same window the report's headline WoW numbers are computed +// over. Every bucket is a COMPLETE 7 days; there is no partial end bar. // -// IMPORTANT: do NOT drop the current in-progress week here -- the chart wants it. -// bucket-trends.js excludes the partial week from the regression/improvement DELTA -// math via `--end=<TREND_CLASS_END> --include-partial-end` (TREND_CLASS_END = -// startofweek(today)) while still charting it. A partial week driving the delta -// would read as a fake -99% improvement -- that's why classification and display -// are split in the JS, not filtered out here. +// This alignment is load-bearing, not cosmetic. When this query used +// startofweek() the trend/classifier basis lagged the report's rolling window by +// up to a full week (for a 2026-08-01 run the classifier's "current" week was +// 07/19-07/26 against a report window of 07/25-08/01 -- ONE day of overlap). +// Codes that turned in the final ~6 days were structurally invisible to the noise +// gate: authorization_pending read +63.2% in the report and -37.1% to the +// classifier, which duly filed it "ONGOING -- plateaued, do not re-triage". +// Aligning the bases makes classifier WoW == displayed WoW and removes that entire +// false-negative class. Do not "restore" startofweek() here. +// +// Because 60 is not a multiple of 7, the OLDEST bucket (curEnd-63d) covers only 4 +// days and IS partial. That is the safe end to be partial on, and it is dropped by +// `--start=<TREND_START>` (its label sorts before TREND_START). Net: 8 complete +// rolling weeks. There is no TREND_CLASS_END any more, and --include-partial-end +// is obsolete -- classification and display now see the identical bucket list. // // Inputs (replace before pasting; bootstrap-report.ps1 prints all values): // <TREND_START> = curEnd - 60d (first calendar day of the 60-day window) -// <TREND_END> = curEnd (today), EXCLUSIVE. Data is pulled up to but not -// including 00:00 UTC today, so the final Sun-Sat bucket holds -// Sunday..yesterday of the current week (the partial bar). +// <TREND_END> = curEnd (today), EXCLUSIVE, and the bin_at anchor. Data is +// pulled up to but not including 00:00 UTC today. // // Output: feed to assets/scripts/bucket-trends.js with -// --start=<TREND_START> --end=<TREND_CLASS_END> --include-partial-end -// where TREND_CLASS_END = startofweek(today) (bootstrap prints it as the -// "Trend delta cutoff" value). +// --start=<TREND_START> --end=<TREND_END> materialized_view('ErrorStatsMetrics') | where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) | where isnotempty(error_code) and error_code != 'success' | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), error_code + by week = bin_at(EventInfo_Time, 7d, datetime(<TREND_END>)), error_code | order by error_code asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql index 7f37fd56..fd1e5132 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/60d-trend-types.kql @@ -1,18 +1,20 @@ // 60-day per-error-type trend (with MergeUiRequiredExceptions to collapse variants). // -// LITERAL last 60 days ending today; Sun-Sat weekly bucketing kept (Kusto -// startofweek() is Sunday-aligned). The current in-progress week is charted as the -// final partial bar and is excluded from delta classification by bucket-trends.js, -// not filtered here. See 60d-trend-codes.kql for full semantics. +// LITERAL last 60 days ending today, bucketed into ROLLING 7-day windows anchored +// at <TREND_END> via bin_at() -- NOT Sun-Sat calendar weeks. The final bucket is +// exactly the report's rolling WoW window, so classifier WoW == displayed WoW. +// Every bucket is complete; there is no partial end bar and no TREND_CLASS_END. +// See 60d-trend-codes.kql for the full rationale (it documents the false-negative +// class this alignment fixes). Do not "restore" startofweek() here. // // Inputs (bootstrap-report.ps1 prints all values): // <TREND_START> = curEnd - 60d -// <TREND_END> = curEnd (today), EXCLUSIVE +// <TREND_END> = curEnd (today), EXCLUSIVE, and the bin_at anchor materialized_view('ErrorStatsMetrics') | extend unified_error_type = MergeUiRequiredExceptions(error_type) | where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) | where isnotempty(unified_error_type) | summarize errs = sum(countOverall), devs = dcount_hll(hll_merge(countDevicesHll)) - by week = startofweek(EventInfo_Time), unified_error_type + by week = bin_at(EventInfo_Time, 7d, datetime(<TREND_END>)), unified_error_type | order by unified_error_type asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md b/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md index f3cdbdd7..41c2e177 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/README.md @@ -15,17 +15,16 @@ rationale. | `<CUR_START>` | `<CUR_END> - 7d` (e.g. `2026-07-02`). Inclusive lower bound of the current window. | | `<PREV_START>` | `<CUR_END> - 14d` (e.g. `2026-06-25`). Inclusive lower bound of the prior 7-day baseline window. The baseline window is `[PREV_START, CUR_START)`. | | `<TREND_START>` | First calendar day of the 60-day trend chart window: `CUR_END - 60d` (literal 60 days ending today). | -| `<TREND_END>` | Exclusive upper bound of the 60-day trend chart window: `CUR_END` (today). The final Sun-Sat bucket is the current in-progress (partial) week — charted, but excluded from delta classification. | -| `<TREND_CLASS_END>` | Delta-classification cutoff = `startofweek(CUR_END)` (the Sunday that opens the current in-progress week). Passed to `bucket-trends.js` as `--end` (with `--include-partial-end`). On a `2026-07-09` (Thu) run, that's `2026-07-05`. Bootstrap prints it as "Trend delta cutoff". | -| `<SPARK_START>` | First Sunday of the WoW-table sparkline series: `startofweek(CUR_END) - 56d`. **Sunday-aligned; 8 complete weeks.** | -| `<SPARK_END>` | Sunday that OPENS the current in-progress week, exclusive: `startofweek(CUR_END)`. The `| where week < datetime(<SPARK_END>)` filter keeps the WoW-row sparklines on 8 complete weeks. | +| `<TREND_END>` | Exclusive upper bound of the 60-day trend chart window: `CUR_END` (today), and the `bin_at(..., 7d, datetime(<TREND_END>))` anchor. The newest bucket is `[CUR_END - 7d, CUR_END)` and is complete. | +| `<SPARK_START>` | First label of the WoW-table sparkline series: `CUR_END - 56d`. | +| `<SPARK_END>` | Exclusive upper bound and `bin_at` anchor for sparklines: `CUR_END`. The sparkline window is 8 complete rolling weeks. | | `<CODES_LIST>` | Comma-separated KQL string list, e.g. `'invalid_resource', 'null_pointer_error'` | | `<TYPES_LIST>` | Same shape but for `unified_error_type`. | | `<DIM>` | A single column name, replaced per dimension run. | **The primary/WoW queries emit two rows per key (bucket = `prevStart` or `curStart`).** The JS helpers (`agg.js`, `summarize-attribution.js`) sort the bucket label lexicographically and treat the smaller value as "prev" and the larger as "cur" — so any pair of sortable datetimes works. -**The 60-day trend queries emit Sun-Sat weekly buckets over the literal last 60 days ending today** (`[CUR_END - 60d, CUR_END)`). `startofweek()` is Sunday-aligned in Kusto. Do **not** filter the partial in-progress week at the source — the chart wants it as the final bar. Exclude it from the delta math via `bucket-trends.js --end=<TREND_CLASS_END> --include-partial-end` (`<TREND_CLASS_END>` = `startofweek(CUR_END)`); otherwise a partial "last" week reads as a fake −99% improvement on every code. The `wow-table-sparkline-series.kql` file is the exception: it keeps 8 complete weeks via `| where week < datetime(<SPARK_END>)` so no WoW row ends on a partial dip. +**The 60-day trend queries emit rolling 7-day buckets anchored at `<TREND_END>` over the literal last 60 days ending today** (`[CUR_END - 60d, CUR_END)`). Use `bin_at(<time column>, 7d, datetime(<TREND_END>))`. The newest bucket is exactly `[CUR_END - 7d, CUR_END)`, so classifier WoW equals the displayed WoW by construction. Because 60 is not a multiple of 7, the oldest bucket covers only 4 days and is dropped by `bucket-trends.js --start=<TREND_START>`. Always pass `--end=<TREND_END>` too; it disables the legacy partial-end auto-drop heuristic and filters no rows under rolling alignment. The `wow-table-sparkline-series.kql` file uses the same curEnd anchor and keeps 8 complete rolling weeks. ## File index @@ -34,8 +33,8 @@ rationale. | [`reliability-auth-only.kql`](reliability-auth-only.kql) | Auth-only requests/devices for the current + prior 7-day windows | Top-line health, denominator caveat | | [`broker-version-share.kql`](broker-version-share.kql) | Per-version share for the WoW window — **evidence for denominator caveat** | Denominator caveat callout, broker adoption | | [`broker-version-share-wow.kql`](broker-version-share-wow.kql) | Single WoW snapshot of version share — fastest evidence for cohort transitions | Denominator caveat callout | -| [`60d-trend-codes.kql`](60d-trend-codes.kql) | Feeds `bucket-trends.js` for codes (Sun-Sat weekly buckets over the literal last 60 days ending today; final bar = partial current week) | 60-day trend analysis | -| [`60d-trend-types.kql`](60d-trend-types.kql) | Feeds `bucket-trends.js` for types (Sun-Sat weekly buckets over the literal last 60 days ending today; final bar = partial current week) | 60-day trend analysis | +| [`60d-trend-codes.kql`](60d-trend-codes.kql) | Feeds `bucket-trends.js` for codes (curEnd-anchored rolling 7-day buckets; final bucket == displayed WoW window) | 60-day trend analysis | +| [`60d-trend-types.kql`](60d-trend-types.kql) | Feeds `bucket-trends.js` for types (curEnd-anchored rolling 7-day buckets; final bucket == displayed WoW window) | 60-day trend analysis | | [`wow-movers.kql`](wow-movers.kql) | **MANDATORY second pass** — catches small-base codes that spiked sharply in the current window (below the 60d bucketer's reporting threshold). Run for both `error_code` and `error_type`. **Merge its output rows into the same regression callout** alongside the standard WoW table, then group them by `classify-novelty.js` labels (🆕 New / 📈 Ongoing / 🔁 Volatile / ↩️ Recovery) — **never by device count**. Do not render a separate "emerging" callout. | Section 2 regression callouts | | [`attr-union-by-dim.kql`](attr-union-by-dim.kql) | **PREFERRED for WoW.** All 7 dims for N codes (or types) in ONE round-trip; pipe through `summarize-attribution.js --union`. | Spike attribution cards | | [`attr-codes-by-dim.kql`](attr-codes-by-dim.kql) | Per-dim form (run 7 times). Fall back to this only when the union exceeds payload size. | Spike attribution cards | @@ -45,4 +44,4 @@ rationale. | [`os-version-slice.kql`](os-version-slice.kql) | OS / OEM concentration (raw `android_spans`). **On-demand only** per Step 5 — don't slice every card. | OS-version dim in attribution cards (when applicable) | | [`latency.kql`](latency.kql) | p50/p95/p99 by hot span for the WoW window | Latency section | | [`app-share.kql`](app-share.kql) | Top calling apps for the WoW window | Traffic analysis | -| [`wow-table-sparkline-series.kql`](wow-table-sparkline-series.kql) | 8-week per-code/per-type sparkline series (Sun-Sat weekly buckets, `<SPARK_START>`/`<SPARK_END>` tokens — 8 complete weeks) for the WoW table `data-trend` arrays. **MANDATORY** — sparkline arrays must come from real data, never fabricated. | Section 6/7 tables | +| [`wow-table-sparkline-series.kql`](wow-table-sparkline-series.kql) | 8-week per-code/per-type sparkline series (curEnd-anchored rolling buckets, `<SPARK_START>`/`<SPARK_END>` tokens) for the WoW table `data-trend` arrays. **MANDATORY** — sparkline arrays must come from real data, never fabricated. | Section 6/7 tables | diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/README.md b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/README.md index 842a38fe..9056e261 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/README.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/README.md @@ -21,8 +21,8 @@ All files use angle-bracket tokens, replaced before execution. | `<CUR_START>` | `curEnd − 7d` — start of the reporting window | | `<CUR_END>` | `curEnd` — **exclusive** upper bound | | `<PREV_START>` | `curEnd − 14d` — start of the baseline window (its end is always `<CUR_START>`) | -| `<TREND_START>` / `<TREND_END>` | literal last 60 days ending today — partial final week **included** | -| `<SPARK_START>` / `<SPARK_END>` | last 8 **complete** Sun-Sat weeks — `<SPARK_END>` = `startofweek(curEnd)`, partial week **excluded at the source** | +| `<TREND_START>` / `<TREND_END>` | literal last 60 days ending today; `<TREND_END>` is also the `bin_at` anchor, so the newest bucket is the displayed WoW window | +| `<SPARK_START>` / `<SPARK_END>` | 8 complete rolling weeks ending at `curEnd` (`<SPARK_START> = curEnd - 56d`, `<SPARK_END> = curEnd`) | | `<ERRORS_MV>` | a `*_Errors_MV_V1` view name | | `<REASON_FILTER>` | optional `\| where Error in (...)` line, or blank | @@ -44,7 +44,7 @@ All files use angle-bracket tokens, replaced before execution. 1. `version-share-wow.kql` **first**. If the version mix moved materially, every downstream rate change has to be read against that. Running it last means re-reading every verdict. 2. `scenario-outcomes-wow.kql` + `pn-completion-wow.kql` — the scoreboard. -3. `scenario-60d-trend.kql` → `bucket-trends.js --key=scenario --end=<startofweek(curEnd)> --include-partial-end --peak-floor=1000`. +3. `scenario-60d-trend.kql` → `bucket-trends.js --key=scenario --start=<TREND_START> --end=<TREND_END> --peak-floor=1000`. 4. `scenario-sparkline-series.kql` — one pass, feeds every sparkline in the report. 5. For each scenario that regressed **and cleared the volume floor**: `scenario-errors-wow.kql`, then `scenario-errors-by-dim.kql` filtered to the reasons that actually moved. diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-60d-trend.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-60d-trend.kql index 5ccab3b3..af0b42a0 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-60d-trend.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-60d-trend.kql @@ -6,15 +6,27 @@ // script change. Extra columns (initiated, initiatedDevices, successRate) ride along for the // chart and are ignored by the bucketer. // +// ⚠️ BUCKETS ARE ROLLING 7-DAY WINDOWS ANCHORED AT <TREND_END>, **NOT** Sun-Sat calendar weeks. +// `bin_at(EventDate, 7d, tEnd)` steps back from curEnd in exact 7-day increments, so the FINAL +// bucket is [curEnd-7d, curEnd) — the identical window the scoreboard's WoW deltas are computed +// over. Every bucket is a COMPLETE 7 days; there is no partial end bar. +// +// This is load-bearing. Under the old startofweek() bucketing the novelty classifier ran on a +// window that lagged the scoreboard by up to a full week, which is how a scenario could carry a +// RED scoreboard pill (rolling-derived) while "Needs attention" simultaneously reported a quiet +// week (calendar-derived). Aligning the bases makes classifier WoW == scoreboard WoW and removes +// that contradiction. Do not "restore" startofweek() here. +// // node bucket-trends.js <file.json> --key=scenario --metric=devs \ -// --end=<startofweek(curEnd)> --include-partial-end --peak-floor=1000 +// --start=<TREND_START> --end=<TREND_END> --peak-floor=1000 // // Cluster: https://idsharedeus2.eastus2.kusto.windows.net // Database: d496be22d62a46b0a3cf67ea2e736fd8 // -// Tokens: <TREND_START> = curEnd - 60d <TREND_END> = curEnd (exclusive). -// Do NOT filter the partial in-progress week at the source — the chart wants it as its final bar. -// Exclude it from delta classification with bucket-trends.js --end=<startofweek(curEnd)>. +// Tokens: <TREND_START> = curEnd - 60d <TREND_END> = curEnd (exclusive, and the bin_at anchor). +// Because 60 is not a multiple of 7 the OLDEST bucket (curEnd-63d) covers only 4 days and IS +// partial; `--start=<TREND_START>` drops it, leaving 8 complete rolling weeks. --include-partial-end +// is obsolete — classification and display now see the identical bucket list. // See scenario-outcomes-wow.kql's "KQL GOTCHAS" header for why the lambda takes the tabular // parameter FIRST, why `P()` spells out every column instead of using `(T:(*))`, and why the // union argument carries its own parentheses. All four rules were learned against live Kusto. @@ -25,7 +37,7 @@ let outcome = (T:(EventDate:datetime, Ini:long, Suc:long, Fai:long, IniD:long, S | where EventDate >= tStart and EventDate < tEnd | summarize Initiated = sum(Ini), Succeeded = sum(Suc), Failed = sum(Fai), InitiatedD = sum(IniD), SucceededD = sum(SucD), FailedD = sum(FaiD) - by week = startofweek(EventDate) + by week = bin_at(EventDate, 7d, tEnd) | extend scenario = label }; let P = (T:(EventDate:datetime, Initiated:long, Succeeded:long, Failed:long, InitiatedDCount:long, SucceededDCount:long, FailedDCount:long)) { diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-sparkline-series.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-sparkline-series.kql index f1b93f40..b05134bd 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-sparkline-series.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/authapp/scenario-sparkline-series.kql @@ -1,21 +1,20 @@ -// Authenticator — 8-COMPLETE-WEEK series per scenario, for the per-row table sparklines. +// Authenticator — 8-ROLLING-WEEK series per scenario, for the per-row table sparklines. // -// Deliberately distinct from scenario-60d-trend.kql: -// * 60d trend = literal last 60 days ending TODAY, partial final week INCLUDED (chart bar). -// * this query = last 8 COMPLETE Sun-Sat weeks, partial week FILTERED AT THE SOURCE. -// Row sparklines must not end on a partial week or every row shows a fake final-week cliff. +// Same rolling 7-day bucketing as scenario-60d-trend.kql, just a shorter span: +// * 60d trend = literal last 60 days ending curEnd -> 8 complete rolling weeks (+ dropped stub). +// * this query = last 8 rolling weeks ending curEnd -> 8 complete rolling weeks, no stub. +// The LAST bucket is [curEnd-7d, curEnd) — identical to the scoreboard's current window — so a +// row reading -1.27 pts no longer draws a sparkline that disagrees with its own number. // -// <SPARK_END> = startofweek(curEnd), EXCLUSIVE. <SPARK_START> = <SPARK_END> - 56d. +// <SPARK_END> = curEnd, EXCLUSIVE, and the bin_at anchor. <SPARK_START> = <SPARK_END> - 56d. // -// ⚠️ <SPARK_START> MUST land exactly on a Sunday (a startofweek boundary). Because <SPARK_END> is -// itself startofweek(curEnd), subtracting exactly 56d always lands on one — so use that formula -// and nothing else. If <SPARK_START> is off by even one day, this query does NOT error: the -// first bucket silently becomes partial and its rate is subtly wrong. Verified live — -// a one-day-late start moved week 1 of Entra MFA Registration (No-QR) from 45.56% to 45.87%, -// which is exactly the kind of phantom "improvement" that would get written up as a finding. -// -// ✅ Verified live: with correct boundaries this returns exactly 8 buckets per scenario and the -// last bucket is a complete week (no trailing cliff). +// ✅ The old "⚠️ <SPARK_START> MUST land exactly on a Sunday" trap is GONE. Under startofweek() +// a one-day-late start silently made bucket 1 partial and subtly wrong (verified live: it moved +// week 1 of Entra MFA Registration (No-QR) from 45.56% to 45.87% — a phantom "improvement" +// that could have been written up as a finding). bin_at() defines buckets by 7-day steps back +// from the anchor, so ANY <SPARK_END> works and every bucket is complete by construction. +// Do not reintroduce startofweek(): it would desynchronise the sparkline from the number +// printed beside it AND resurrect the silent-partial-bucket trap. // // Emits one row per (scenario, week) with both the success-rate series (what the scoreboard // sparkline draws) and the bad-outcome volume series (what the regression rows draw). @@ -32,7 +31,7 @@ let outcome = (T:(EventDate:datetime, Ini:long, Suc:long, Fai:long, IniD:long, S | where EventDate >= sStart and EventDate < sEnd | summarize Initiated = sum(Ini), Succeeded = sum(Suc), Failed = sum(Fai), InitiatedD = sum(IniD), SucceededD = sum(SucD), FailedD = sum(FaiD) - by week = startofweek(EventDate) + by week = bin_at(EventDate, 7d, sEnd) | extend scenario = label }; let P = (T:(EventDate:datetime, Initiated:long, Succeeded:long, Failed:long, InitiatedDCount:long, SucceededDCount:long, FailedDCount:long)) { diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql b/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql index 9b561575..59ae552b 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql +++ b/.github/skills/oncall-weekly-telemetry-report/assets/queries/wow-table-sparkline-series.kql @@ -13,23 +13,23 @@ // the codes filter, once with the types filter -- and feed the result into the // WoW-table generator so every row has a real-data trend. // -// NOTE ON WEEK ALIGNMENT: even though the primary WoW section uses a rolling -// 7-day window AND the big 60-day trend chart now ends today (with a partial -// final bar), these per-ROW sparklines deliberately keep 8 COMPLETE Sun-Sat -// weeks. A partial final point in every WoW row would render as a misleading -// dip across the whole table; the decorative per-row trajectory reads cleaner -// on complete weeks. (The big 60d chart can afford one honest partial bar; 30+ -// table rows each ending on a partial point cannot.) +// NOTE ON WEEK ALIGNMENT: these sparklines use the SAME rolling 7-day buckets as +// the 60-day trend and the headline WoW numbers -- `bin_at(t, 7d, <SPARK_END>)` +// with <SPARK_END> = curEnd. That makes the LAST sparkline point identical to the +// row's own "current" value, so a row reading +63% no longer draws a falling +// trajectory. All 8 buckets are complete 7-day windows, so the old "keep complete +// Sun-Sat weeks to avoid a misleading partial dip in every row" workaround is no +// longer needed -- the partial-point problem cannot occur under rolling alignment. +// Do not reintroduce startofweek() here: it would desynchronise the sparkline from +// the number printed beside it. // -// Inputs (distinct from the trend-codes/types <TREND_*> tokens, which now mean -// the LITERAL last 60 days ending today -- these SPARK_* tokens still mean the -// last 8 COMPLETE weeks): -// <SPARK_START> Sunday of week-0 = startofweek(today) - 56d -// (e.g. 2026-05-10 for an 8-week window ending 2026-07-05) -// <SPARK_END> Sunday that OPENS the current in-progress week, EXCLUSIVE. -// Compute as startofweek(today). For a report run on 2026-07-09 -// (a Thu), use 2026-07-05. The `where week < datetime(<SPARK_END>)` -// drops the partial in-progress bucket at the source. +// Inputs (distinct from the trend-codes/types <TREND_*> tokens, which cover +// the LITERAL last 60 days ending today -- these SPARK_* tokens cover the +// last 8 rolling weeks): +// <SPARK_START> = curEnd - 56d (8 rolling 7-day buckets) +// <SPARK_END> = curEnd (today), EXCLUSIVE, and the bin_at anchor. No +// Sunday-alignment requirement -- any date works, because the +// buckets are defined by 7-day steps back from this anchor. // <CODES> Dynamic list of error_code values whose sparklines we need. // Build this from the union of: // * wow-movers-codes.json results @@ -39,10 +39,9 @@ let codes = dynamic([<CODES>]); materialized_view('ErrorStatsMetrics') -| where EventInfo_Time between (datetime(<SPARK_START>) .. datetime(<SPARK_END>)) +| where EventInfo_Time >= datetime(<SPARK_START>) and EventInfo_Time < datetime(<SPARK_END>) | where error_code in (codes) | summarize devs = dcount_hll(hll_merge(countDevicesHll)), errs = sum(countOverall) - by week = startofweek(EventInfo_Time), error_code -| where week < datetime(<SPARK_END>) + by week = bin_at(EventInfo_Time, 7d, datetime(<SPARK_END>)), error_code | order by error_code asc, week asc diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 index 687b5480..6fa59ba0 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bootstrap-report.ps1 @@ -155,29 +155,32 @@ $prevStart = $curEnd.AddDays(-14) $prevEnd = $curStart # 60-day trend spans the LITERAL last 60 days ending curEnd (today), so BOTH -# bounds move with -EndDate (the trend ends today rather than 3-6 days ago on -# the last complete Sunday). The trend CHART shows the -# current in-progress week as its final (partial) bar, but bucket-trends.js still -# computes regression/improvement DELTAS on complete Sun-Sat weeks only -- a -# partial week as "last" would read as a fake -99% improvement. So we resolve two -# distinct things: -# * data/chart window [sixtyDayStart, sixtyDayEnd) = [curEnd-60d, curEnd) -# -> queried and charted (final bucket is the partial week) -# * classification cutoff trendClassEnd = startofweek(curEnd) -# -> passed to bucket-trends.js as --end so the partial -# current week is excluded from the delta math while -# still being drawn. -# Per-week bucketing stays Sun-Sat aligned (Kusto startofweek() is Sunday-based), -# so the first and last buckets are partial by construction. -$curEndDow = [int]$curEnd.DayOfWeek # Sun=0 .. Sat=6 +# bounds move with -EndDate. +# +# ⚠️ Trend/sparkline buckets are ROLLING 7-day windows anchored at curEnd +# (`bin_at(t, 7d, curEnd)` in KQL), NOT Sun-Sat calendar weeks. The FINAL bucket is +# therefore [curEnd-7d, curEnd) -- byte-for-byte the same window as the headline WoW +# numbers -- so the novelty classifier and the report agree by construction. Every +# bucket is a COMPLETE 7 days, so there is no partial end bar, no separate +# classification cutoff, and no --include-partial-end. +# +# This replaced startofweek() bucketing, which lagged the report's rolling window by +# up to a full week and silently suppressed genuinely-rising keys (for a 2026-08-01 +# run the classifier's "current" week was 07/19-07/26 against a report window of +# 07/25-08/01: authorization_pending read +63.2% in the report and -37.1% to the +# classifier, and was filed "ONGOING -- do not re-triage"). Do not reintroduce a +# separate calendar-week cutoff here. +# +# Because 60 is not a multiple of 7, the OLDEST bucket (curEnd-63d) covers 4 days +# and IS partial -- the safe end to be partial on. It is dropped by passing +# --start=$sixtyDayStart to bucket-trends.js, leaving 8 complete rolling weeks. $sixtyDayStart = $curEnd.AddDays(-60) # literal 60 days ending today -$sixtyDayEnd = $curEnd # exclusive upper bound == today; chart includes the partial current week -$trendClassEnd = $curEnd.AddDays(-$curEndDow) # startofweek(curEnd): weeks >= this are the in-progress (partial) week, excluded from delta classification -# Sparkline window: last 8 COMPLETE Sun-Sat weeks. sparkEnd is exclusive and is by construction a -# Sunday, so sparkEnd-56d is also exactly a Sunday. These are emitted below so the author never -# hand-computes them: an off-by-one sparkStart does NOT error in Kusto, it silently makes the first -# sparkline bucket partial and skews its rate (verified live against Entra MFA Registration No-QR). -$sparkEnd = $trendClassEnd +$sixtyDayEnd = $curEnd # exclusive upper bound == today == the bin_at anchor +# Sparkline window: last 8 ROLLING weeks, same anchor. Because buckets are defined by +# 7-day steps back from curEnd, sparkStart has NO Sunday-alignment requirement -- the +# old "an off-by-one sparkStart silently makes bucket 1 partial" trap (verified live +# against Entra MFA Registration No-QR) cannot occur under bin_at. +$sparkEnd = $curEnd $sparkStart = $sparkEnd.AddDays(-56) # Sanity check: curEnd is an exclusive 00:00-UTC date boundary and must be today @@ -197,9 +200,10 @@ $prevEndStr = $prevEnd.ToString('yyyy-MM-dd') Write-Host "Resolved reporting window (UTC) for -App $App :" Write-Host " Last 7 days: $curStartStr -> $curEndStr (exclusive upper bound)" Write-Host " Baseline: $prevStartStr -> $prevEndStr" -Write-Host " 60-day trend: $($sixtyDayStart.ToString('yyyy-MM-dd')) -> $($sixtyDayEnd.ToString('yyyy-MM-dd')) (literal 60d ending today; chart includes current partial week)" -Write-Host " Trend delta cutoff: weeks < $($trendClassEnd.ToString('yyyy-MM-dd')) (startofweek(curEnd); pass as bucket-trends.js --end)" -Write-Host " Sparkline (8 complete weeks): $($sparkStart.ToString('yyyy-MM-dd')) -> $($sparkEnd.ToString('yyyy-MM-dd')) (SPARK_START -> SPARK_END, exclusive; both land on Sunday -- do not adjust by hand)" +Write-Host " 60-day trend: $($sixtyDayStart.ToString('yyyy-MM-dd')) -> $($sixtyDayEnd.ToString('yyyy-MM-dd')) (literal 60d ending today; rolling 7d buckets anchored at curEnd)" +Write-Host " Trend buckets: 8 complete rolling weeks; final bucket == the Last-7-days window above (classifier WoW == displayed WoW)" +Write-Host " bucket-trends.js: --start=$($sixtyDayStart.ToString('yyyy-MM-dd')) --end=$curEndStr (pass BOTH; --end disables the partial-end auto-drop heuristic)" +Write-Host " Sparkline (8 rolling weeks): $($sparkStart.ToString('yyyy-MM-dd')) -> $($sparkEnd.ToString('yyyy-MM-dd')) (SPARK_START -> SPARK_END, exclusive; no Sunday alignment needed)" # NOTE: Console output uses ASCII '->'; the HTML stamp below uses U+2192 arrows # and U+00B7 middle-dots to match the template's canonical visual style. This # is safe because $outText is written via [System.Text.UTF8Encoding]::new($false) @@ -303,8 +307,8 @@ $curLabel = "$(Format-DateHuman $curStart -IncludeYear:$false) $arrow $(F $prevLabel = "$(Format-DateHuman $prevStart -IncludeYear:$false) $arrow $(Format-DateHuman $prevEnd -IncludeYear:$false)" # 60-day trend: literal 60 days ending curEnd (today). curEnd is the exclusive # upper bound, so the last calendar day carrying data is curEnd - 1 (yesterday). -# The final weekly bucket is the current in-progress week (partial) -- see -# bootstrap's $trendClassEnd note and bucket-trends.js --include-partial-end. +# Buckets are rolling 7-day windows anchored at curEnd, so the FINAL bar is a +# complete week and is identical to the "Last 7 days" window above. $sixtyDayLabel = "$(Format-DateHuman $sixtyDayStart -IncludeYear:$false) $arrow $(Format-DateHuman ($sixtyDayEnd.AddDays(-1)) -IncludeYear:$true)" $todayStr = (Get-Date).ToUniversalTime().ToString('yyyy-MM-dd') @@ -319,7 +323,7 @@ $outText = [regex]::Replace($outText, '<title>[^<]*', "$newTitle $newMeta = @" <div class="meta"> <strong>Last 7 days: $curLabel</strong>  vs  <strong>$prevLabel</strong>  $dot  - 60-day trend: <strong>$sixtyDayLabel</strong> (last 60 days; final bar in progress)  $dot  + 60-day trend: <strong>$sixtyDayLabel</strong> (last 60 days; rolling 7-day buckets)  $dot  $($appProfile.SourceHtml)  $dot  Generated <strong>$todayStr</strong> </div> diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js index aaa0186d..3dac83c5 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/bucket-trends.js @@ -2,11 +2,23 @@ /** * bucket-trends.js -- Bucket every error code into 60-day trend categories. * - * This tool operates ONLY on the 60-day trend section, which still uses Sun-Sat - * weekly buckets (Kusto startofweek()-aligned). The primary/WoW section uses a - * rolling 7-day window and does NOT go through this script -- it consumes the - * two-bucket outputs of reliability-auth-only.kql / wow-movers.kql / etc. - * directly. + * This tool operates ONLY on the 60-day trend section. Its buckets are ROLLING + * 7-day windows anchored at curEnd (Kusto `bin_at(t, 7d, datetime(curEnd))`), NOT + * Sun-Sat calendar weeks. Consequently the FINAL bucket is [curEnd-7d, curEnd) -- + * exactly the same window the primary/WoW section reports on -- so a key's WoW here + * equals its WoW in the headline table by construction. + * + * ⚠️ HISTORY -- DO NOT REVERT TO startofweek(). Buckets used to be Sun-Sat calendar + * weeks cut off at startofweek(curEnd). That basis LAGGED the report's rolling window + * by up to a full week, so anything that turned in the last ~6 days was invisible to + * the novelty gate downstream. Real case (2026-08-01 run, classifier week 07/19-07/26 + * vs report window 07/25-08/01, ONE day of overlap): + * authorization_pending report +63.2% classifier -37.1% -> filed "ONGOING" + * expired_token report +26.7% classifier -51.0% -> filed "ONGOING" + * Both were real risers the OCE had already spotted by hand. Re-bucketing on bin_at + * promoted both to ACCELERATING and simultaneously DEMOTED access_denied (a former + * false positive, actually -53.2%). Alignment adds real signal and removes phantom + * signal; it is not merely "more alerts". * * Input: a Kusto MCP JSON result file from a query of the form: * @@ -15,39 +27,38 @@ * | where isnotempty(error_code) and error_code != 'success' * | summarize errs=sum(countOverall), * devs=dcount_hll(hll_merge(countDevicesHll)) - * by week=startofweek(EventInfo_Time), error_code + * by week=bin_at(EventInfo_Time, 7d, datetime(<trend_end>)), error_code * | order by error_code asc, week asc * * (Use dcount_hll on countDevicesHll, NOT sum(countDevices) -- see ../docs/kusto-cheatsheet.md.) * - * LITERAL 60-day window ending today: - * The 60d trend now spans [today-60d, today), so the query no longer filters the - * partial in-progress week at the source -- the CHART wants it as the final bar. - * Instead this script splits two week sets: - * * classification weeks (complete Sun-Sat only) drive first/last/delta/spike/ - * peak-floor -- a partial week as "last" would read as a fake -99% improvement. - * * display weeks (adds the partial current week) drive the emitted `series` - * arrays and the JSON sidecar, so the sparkline/chart ends today. - * Pass `--end=<startofweek(today)> --include-partial-end` for this behavior. - * Without --include-partial-end the script behaves as before (display == classify). + * WHICH BUCKET IS PARTIAL: + * 60 is not a multiple of 7, so stepping back from curEnd in 7-day strides leaves + * the OLDEST bucket (curEnd-63d) covering only 4 days. That is the safe end to be + * partial on. Pass `--start=<curEnd-60d>` to drop it, leaving 8 complete rolling + * weeks. The NEWEST bucket is always complete, so classify == display and there is + * no fake -99%-improvement failure mode to guard against. + * + * ⚠️ ALWAYS PASS --end=<curEnd>, NOT JUST --start. + * The partial-end auto-drop heuristic below is guarded by `if (!endArg ...)`. Under + * rolling alignment the last bucket is genuinely complete, so leaving --end off would + * let a REAL 70% collapse be silently discarded as "looks partial". Passing + * --end=<curEnd> disables the heuristic; it filters nothing, because every bucket + * label is < curEnd by construction. * - * <end> convention: startofweek(today) -- i.e. the Sunday that OPENS the currently - * in-progress week. Every complete week strictly before that Sunday is classified; - * with --include-partial-end the in-progress week (bucket == <end>) is still charted. - * See assets/scripts/bootstrap-report.ps1 for how this is computed ($trendClassEnd). + * LEGACY FLAGS (no-ops under rolling alignment, retained so old invocations don't crash): + * --include-partial-end There is no partial end bucket any more. classifyWeeks + * always equals weeks. Safe to omit; safe to pass. * * Usage: * node bucket-trends.js <mcp-output.json> - * [--start=YYYY-MM-DD] [--end=YYYY-MM-DD] # inclusive start, EXCLUSIVE end (week-bucket) - * [--include-partial-end] [--peak-floor=N] [--metric=devs|reqs] + * --start=YYYY-MM-DD --end=YYYY-MM-DD # inclusive start, EXCLUSIVE end (bucket label) + * [--peak-floor=N] [--metric=devs|reqs] * - * --start defaults to the second-earliest week in the data (drops partial start week). - * --end defaults to the most recent week, but the script will WARN-AND-DROP any week - * where (latest EventInfo_Time in the bucket - week-start) < 6 days, because that - * is a partial in-progress week and will turn every error into a fake -99% improvement. - * --include-partial-end keep the partial current week (bucket >= --end) in the emitted - * `series` arrays / JSON sidecar for charting, while still EXCLUDING it from the - * delta/spike/first/last classification. No-op without --end. + * --start drops the 4-day partial oldest bucket. Defaults to the second-earliest + * bucket in the data, which achieves the same thing -- but pass it explicitly. + * --end should be curEnd. See the warning above; omitting it re-enables a heuristic + * that is actively wrong for this bucketing scheme. * * --metric=devs (default) buckets on weekly device counts (catches errors hitting more users) * --metric=reqs buckets on weekly request counts (catches per-device retry storms) @@ -70,9 +81,9 @@ * sparkline-data-generator script). The sidecar shape is: * { * "metric": "devs" | "reqs", - * "weeks": [iso, ...], // DISPLAY weeks (incl. partial end) - * "classifyWeeks": [iso, ...], // complete weeks used for deltas - * "includePartialEnd": bool, + * "weeks": [iso, ...], // rolling 7d bucket starts + * "classifyWeeks": [iso, ...], // == weeks under rolling alignment + * "includePartialEnd": bool, // legacy; no effect * "buckets": { * "regression": [ { code, first, last, peak, delta, series: [N,N,...] }, ... ], * "spike": [...], @@ -140,12 +151,12 @@ const startISO = startArg ? `${startArg}T00:00:00Z` : weeks[1]; // drop partial const endISO = endArg ? `${endArg}T00:00:00Z` : null; // exclusive cutoff // --- Partial end-week detection --------------------------------------------- -// Compute the total devices/requests per bucket as a proxy for completeness. -// If the most recent bucket is < 30% of the median of the prior 3 buckets, it's -// almost certainly partial — drop it and warn. This catches the common case of -// running the report at 09:00 UTC Sunday and getting 9 hours of data in the -// "last week" bucket. (Caveat: real fleet collapses also look like this; warn, -// don't crash.) +// LEGACY GUARD. Under rolling bin_at(t, 7d, curEnd) bucketing the newest bucket is +// always a complete 7 days, so this heuristic should never fire in the normal skill +// workflow -- and it MUST NOT, because a real 70% fleet collapse looks identical to +// a partial bucket and would be silently discarded. It stays only to protect ad-hoc +// invocations that omit --end. Passing --end=<curEnd> disables it (see the warning +// emitted just below). function bucketTotal(w) { let t = 0; for (const wd of Object.values(series)) { @@ -157,21 +168,25 @@ function bucketTotal(w) { const totals = weeks.map(w => ({ w, t: bucketTotal(w) })); const medianOf = arr => { const s = [...arr].sort((a,b)=>a-b); return s[Math.floor(s.length/2)] || 0; }; let droppedPartial = null; +if (!endArg) { + console.warn('[bucket-trends] WARN: no --end given. Buckets are rolling 7-day windows anchored at curEnd, so the newest bucket is COMPLETE; the partial-end auto-drop heuristic below can therefore discard a genuine collapse. Pass --end=<curEnd> (it filters nothing) for the standard skill workflow.'); +} if (!endArg && weeks.length >= 4) { const last = totals[totals.length - 1]; const prevMedian = medianOf(totals.slice(-4, -1).map(x => x.t)); if (prevMedian > 0 && last.t < prevMedian * 0.3) { droppedPartial = last.w; - console.warn(`[bucket-trends] WARN: dropping likely-partial end bucket ${last.w} (total=${last.t.toLocaleString()} vs median-of-prior-3=${prevMedian.toLocaleString()}). Pass --end=YYYY-MM-DD to override or filter in KQL.`); + console.warn(`[bucket-trends] WARN: dropping likely-partial end bucket ${last.w} (total=${last.t.toLocaleString()} vs median-of-prior-3=${prevMedian.toLocaleString()}). If this is a REAL collapse, re-run with --end=<curEnd> to keep it.`); } } -// classKeep = complete Sun-Sat weeks used for delta/spike/first/last classification. -// A partial week here would produce a fake -99% improvement, so it is always excluded. +// classKeep = buckets used for delta/spike/first/last classification. +// Under rolling alignment every kept bucket is a complete 7 days; --start drops the +// 4-day partial OLDEST bucket (curEnd-63d). const classKeep = weeks.filter(w => w >= startISO && (endISO ? w < endISO : true) && w !== droppedPartial); -// displayKeep = weeks emitted in `series` / the JSON sidecar (what the chart draws). -// With --include-partial-end it adds the partial current week (bucket >= endISO up to -// endISO inclusive) so the chart ends today; otherwise it mirrors classKeep. +// displayKeep = buckets emitted in `series` / the JSON sidecar (what the chart draws). +// --include-partial-end is a legacy no-op under rolling alignment (there is no partial +// end bucket), so this normally mirrors classKeep exactly. const displayKeep = (includePartialEnd && endISO) ? weeks.filter(w => w >= startISO && w <= endISO) : classKeep; diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 index d3e1f411..fe7660d9 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 +++ b/.github/skills/oncall-weekly-telemetry-report/assets/scripts/validate-report.ps1 @@ -674,6 +674,64 @@ if ($t60Start -ge 0) { Add-Warn "Could not locate the 60-day section -- skipping the chart-catalog check." } +# ---- 19. Every red/amber table pill must be reconciled (HARD FAIL) ---- +# The scoreboard / WoW tables colour a row from its own rolling delta. The attention +# section is populated from classify-novelty.js's NOVELTY verdict. Those answer different +# questions, so a row can be legitimately red up there and legitimately absent from here. +# The reader cannot know that, and an unexplained red pill sitting above the words +# "Quiet week" reads as a broken report. +# +# Real case (2026-08-01 Authenticator run): `Passkey WebAuthN Registration` carried tag-bad +# (-1.27 pts, worst delta in the table) while the attention section said "Quiet week -- 0 NEW +# or ACCELERATING". Both statements were true. The scenario peaks at ~732 bad-outcome devices, +# UNDER the 1,000-device peak-floor, so it is structurally excluded from classification and +# can never appear in attention however sharply it moves. +# +# Rule: a key carrying tag-bad/tag-warn must appear EITHER in the attention section OR in a +# .reconcile-note that names it and gives the reason it is not being escalated. +$scoreStart = $content.IndexOf('id="scoreboard"') +if ($scoreStart -lt 0) { $scoreStart = $content.IndexOf('id="error-codes"') } +if ($scoreStart -ge 0 -and $attStart -ge 0 -and $attEnd -gt $attStart) { + $scoreEnd = $content.IndexOf('<h2', $scoreStart + 10) + if ($scoreEnd -lt 0) { $scoreEnd = $content.Length } + $scoreSec = $content.Substring($scoreStart, $scoreEnd - $scoreStart) + + # A flagged row = a <tr> whose markup contains tag-bad or tag-warn. The key is the first + # .code-cell in that row (both templates put the code/scenario name there). + $flagged = @() + foreach ($m in [regex]::Matches($scoreSec, '<tr\b.*?</tr>', 'Singleline')) { + $row = $m.Value + if ($row -notmatch 'tag-bad|tag-warn') { continue } + $k = [regex]::Match($row, '<td class="code-cell"[^>]*>(?:<[^>]+>)*([^<]+)') + if ($k.Success) { + $name = $k.Groups[1].Value.Trim() + if ($name -and $name -notmatch '^(EXAMPLE|CODE_|SCENARIO_)') { $flagged += $name } + } + } + $flagged = $flagged | Select-Object -Unique + + if ($flagged.Count -eq 0) { + Pass "No red/amber pills in the scoreboard/WoW table -- nothing to reconcile" + } else { + # Accounted for = named anywhere in the attention section (promoted as an item) or in + # any .reconcile-note anywhere in the report (explicitly dismissed). + $attSecFull = $content.Substring($attStart, $attEnd - $attStart) + $reconcileTxt = ([regex]::Matches($content, '<div class="reconcile-note".*?</div>', 'Singleline') | + ForEach-Object { $_.Value }) -join ' ' + $accountedIn = $attSecFull + ' ' + $reconcileTxt + + $unreconciled = $flagged | Where-Object { $accountedIn -notmatch [regex]::Escape($_) } + + if ($unreconciled.Count -gt 0) { + Add-Fail "$($unreconciled.Count) row(s) carry a red/amber status pill but are neither promoted into the attention section nor named in a .reconcile-note: $($unreconciled -join ', '). A reader who sees a red pill and an empty/quiet attention section concludes the report is broken. Either promote them, or add a muted <div class=""reconcile-note""> naming each one and why it is not escalated (below the classification floor / within its own normal band / ONGOING and flat)." + } else { + Pass "All $($flagged.Count) red/amber pill(s) are reconciled (promoted or explicitly dismissed)" + } + } +} else { + Add-Warn "Could not locate the scoreboard/WoW table or the attention section -- skipping the pill-reconciliation check." +} + Write-Host "" if ($failures.Count -eq 0) { Write-Host "All hard checks passed." -ForegroundColor Green diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/authapp-report-template.html b/.github/skills/oncall-weekly-telemetry-report/assets/templates/authapp-report-template.html index f4b6918e..6082c762 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/authapp-report-template.html +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/authapp-report-template.html @@ -395,6 +395,15 @@ border-radius: 6px; padding: 12px 14px; margin: 12px 0; font-size: 13px; } .quiet-banner strong { color: #1a7f37; } + /* Deliberately muted: a reconciliation note is a DISMISSAL, not a finding. If it + competes visually with .callout it re-creates the noise the noise gate removes. */ + .reconcile-note { + border-left: 3px solid #d0d7de; background: #f6f8fa; color: #57606a; + border-radius: 0 6px 6px 0; padding: 9px 12px; margin: 10px 0; font-size: 12.5px; + line-height: 1.55; + } + .reconcile-note strong { color: #424a53; font-weight: 600; } + .reconcile-note .mono { color: #24292f; } .metric { display: inline-flex; align-items: baseline; gap: 4px; font-size: 12px; color: #1f2328; font-variant-numeric: tabular-nums; @@ -443,7 +452,7 @@ <h1>Microsoft Authenticator (Android) · On-Call Report</h1> <div class="meta"> <strong>Last 7 days: Thu Jul 23 → Thu Jul 30, 2026</strong>  vs  <strong>Thu Jul 16 → Thu Jul 23</strong>  ·  - 60-day trend: <strong>Sun May 31 → Wed Jul 29, 2026</strong> (last 60 days; final bar in progress)  ·  + 60-day trend: <strong>Sun May 31 → Wed Jul 29, 2026</strong> (last 60 days; rolling 7-day buckets)  ·  Source: <code>Authenticator scenario materialized views</code>  ·  Generated <strong>2026-07-30</strong> </div> @@ -459,8 +468,8 @@ <h1>Microsoft Authenticator (Android) · On-Call Report</h1> <nav class="toc"> <ul> <li><a href="#health">📊 Top-line health</a></li> - <li><a href="#scoreboard">Scenario scoreboard</a></li> <li><a href="#attention">🚨 Needs attention</a></li> + <li><a href="#scoreboard">Scenario scoreboard</a></li> <li><a href="#trend60d">📈 60-day trend</a></li> <li><a href="#attribution">🔎 Error attribution</a></li> <li><a href="#unknown">Unknown analysis</a></li> @@ -483,27 +492,6 @@ <h2 id="health">📊 Top-line health <span class="count">— Authenticator scena <div class="kpi"><div class="label">Crash rate per 1k devices</div><div class="value">0.42</div><div class="delta delta-up">+0.05 WoW</div><div class="spark" data-spark='[0.31,0.32,0.29,0.34,0.33,0.36,0.35,0.37,0.42]' data-color="#cf222e"></div></div> </div> -<!-- ================ SCENARIO SCOREBOARD ================ --> -<h2 id="scoreboard">Scenario scoreboard <span class="count">— one row per Authenticator scenario family</span></h2> -<div class="table-card"><table> - <thead><tr><th>Scenario</th><th class="num">Initiated</th><th class="num">Success rate</th><th class="num">Δ success rate (pts)</th><th class="num">Failure rate</th><th class="num">Unknown rate</th><th class="num">Δ unknown (pts)</th><th class="num">Devices</th><th>8-week sparkline</th><th>Status</th></tr></thead> - <tbody> -<tr><td class="code-cell">Passkey WebAuthN Registration</td><td class="num">48,320</td><td class="num good">91.8%</td><td class="num good">+0.6</td><td class="num">4.7%</td><td class="num ">3.5%</td><td class="num muted">-0.3</td><td class="num">38,940</td><td class="spark-cell"><div class="trend" data-trend='[89.7,90.1,90.6,91,91.3,91.5,91.2,91.8]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> -<tr><td class="code-cell">Passkey InApp Registration</td><td class="num">22,410</td><td class="num ">88.2%</td><td class="num muted">-0.4</td><td class="num">6.9%</td><td class="num ">4.9%</td><td class="num muted">+0.2</td><td class="num">19,120</td><td class="spark-cell"><div class="trend" data-trend='[87.9,88,88.4,88.8,88.5,88.7,88.6,88.2]' data-color="#bf8700" data-w="160" data-h="28"></div></td><td><span class="tag tag-warn">watch</span></td></tr> -<tr><td class="code-cell">Passkey WebAuthN Authentication</td><td class="num">612,900</td><td class="num bad">94.1%</td><td class="num bad">-1.4</td><td class="num">3.2%</td><td class="num bad">2.7%</td><td class="num bad">+0.8</td><td class="num">402,100</td><td class="spark-cell"><div class="trend" data-trend='[96.4,96.2,95.8,95.5,95.3,95.1,95.5,94.1]' data-color="#cf222e" data-w="160" data-h="28"></div></td><td><span class="tag tag-bad">regressed</span></td></tr> -<tr><td class="code-cell">Entra MFA Registration (QR)</td><td class="num">1,245,000</td><td class="num ">97.6%</td><td class="num muted">+0.3</td><td class="num">1.2%</td><td class="num ">1.2%</td><td class="num muted">-0.2</td><td class="num">902,400</td><td class="spark-cell"><div class="trend" data-trend='[97.1,97,97.2,97.3,97.4,97.5,97.3,97.6]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> -<tr><td class="code-cell">Entra MFA Registration (No-QR)</td><td class="num">336,200</td><td class="num ">95.0%</td><td class="num muted">-0.1</td><td class="num">2.8%</td><td class="num ">2.2%</td><td class="num muted">0.0</td><td class="num">248,900</td><td class="spark-cell"><div class="trend" data-trend='[95.1,95.4,95.2,95.3,95,95.2,95.1,95]' data-color="#656d76" data-w="160" data-h="28"></div></td><td><span class="tag tag-muted">flat</span></td></tr> -<tr><td class="code-cell">Entra MFA PN+CFA</td><td class="num">42,800,000</td><td class="num bad">98.4%</td><td class="num bad">-0.9</td><td class="num">0.7%</td><td class="num bad">0.9%</td><td class="num muted">+0.5</td><td class="num">18,700,000</td><td class="spark-cell"><div class="trend" data-trend='[99.1,99,98.9,98.8,98.6,98.5,99,98.4]' data-color="#cf222e" data-w="160" data-h="28"></div></td><td><span class="tag tag-bad">regressed</span></td></tr> -<tr><td class="code-cell">Entra PSI Registration</td><td class="num">188,500</td><td class="num good">93.6%</td><td class="num good">+1.2</td><td class="num">4.4%</td><td class="num good">2.0%</td><td class="num good">-0.6</td><td class="num">131,800</td><td class="spark-cell"><div class="trend" data-trend='[91.9,92.2,92.6,92.8,93,93.2,92.4,93.6]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">win</span></td></tr> -<tr><td class="code-cell">Entra PSI PN Registration</td><td class="num">74,900</td><td class="num ">92.1%</td><td class="num muted">+0.4</td><td class="num">5.1%</td><td class="num ">2.8%</td><td class="num muted">-0.2</td><td class="num">58,600</td><td class="spark-cell"><div class="trend" data-trend='[91.2,91.5,91.4,91.8,91.7,92,91.8,92.1]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> -<tr><td class="code-cell">Entra PSI PN+CFA</td><td class="num">4,920,000</td><td class="num bad">96.8%</td><td class="num bad">-0.6</td><td class="num">1.4%</td><td class="num bad">1.8%</td><td class="num muted">+0.4</td><td class="num">2,860,000</td><td class="spark-cell"><div class="trend" data-trend='[97.4,97.3,97.2,97.1,97,97,97.4,96.8]' data-color="#bf8700" data-w="160" data-h="28"></div></td><td><span class="tag tag-warn">watch</span></td></tr> -<tr><td class="code-cell">MSA NGC Registration</td><td class="num">91,300</td><td class="num good">89.5%</td><td class="num good">+0.8</td><td class="num">7.2%</td><td class="num good">3.3%</td><td class="num muted">-0.4</td><td class="num">72,100</td><td class="spark-cell"><div class="trend" data-trend='[88.1,88.4,88.9,89,88.7,89.2,88.9,89.5]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> -<tr><td class="code-cell">MSA SA Registration</td><td class="num">19,800</td><td class="num ">86.4%</td><td class="num muted">+0.2</td><td class="num">8.4%</td><td class="num ">5.2%</td><td class="num muted">-0.1</td><td class="num">16,400</td><td class="spark-cell"><div class="trend" data-trend='[85.8,86,85.9,86.2,86.1,86.4,86.2,86.4]' data-color="#656d76" data-w="160" data-h="28"></div></td><td><span class="tag tag-muted">flat</span></td></tr> -<tr><td class="code-cell">MSA NGC PN+CFA</td><td class="num">3,760,000</td><td class="num ">97.2%</td><td class="num muted">+0.5</td><td class="num">1.1%</td><td class="num good">1.7%</td><td class="num muted">-0.4</td><td class="num">1,940,000</td><td class="spark-cell"><div class="trend" data-trend='[96.2,96.4,96.6,96.8,96.9,97,96.8,97.2]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">win</span></td></tr> -<tr><td class="code-cell">MSA SA PN+CFA</td><td class="num">1,185,000</td><td class="num ">95.4%</td><td class="num muted">-0.2</td><td class="num">2.2%</td><td class="num ">2.4%</td><td class="num muted">+0.1</td><td class="num">764,000</td><td class="spark-cell"><div class="trend" data-trend='[95.3,95.5,95.2,95.6,95.4,95.5,95.6,95.4]' data-color="#656d76" data-w="160" data-h="28"></div></td><td><span class="tag tag-muted">flat</span></td></tr> - </tbody> -</table></div> - <!-- ================ NEEDS ATTENTION ================ --> <!-- ORDERING IS BY NOVELTY, NOT BY VOLUME OR BY SEVERITY OF THE DELTA. @@ -530,6 +518,39 @@ <h2 id="attention">🚨 Needs attention</h2> </div> --> +<!-- + ⚠️ MANDATORY: RECONCILE EVERY RED/AMBER SCOREBOARD PILL. + The scoreboard pill is computed from the row's own WoW delta. This section is populated + from classify-novelty.js's novelty verdict. Those answer DIFFERENT questions, so a row can + legitimately be `tag-bad` up there and absent from here -- but the reader cannot know that, + and a red pill sitting above the words "Quiet week" reads as a broken report. + + Real case that motivated this rule (2026-08-01): `Passkey WebAuthN Registration` carried + tag-bad (-1.27 pts, worst delta in the table) while this section said "Quiet week -- 0 NEW or + ACCELERATING". Both statements were correct. The scenario peaks at ~732 bad-outcome devices, + UNDER the 1,000 peak-floor, so it is structurally excluded from classification and can never + appear here no matter how it moves. + + RULE: every scoreboard row carrying `tag-bad` or `tag-warn` must be accounted for in this + section -- either promoted as a real item, or named in the reconciliation note below with the + reason it is not being escalated. validate-report.ps1 hard-fails an unreconciled red pill. + + Reasons, in the order you should test them: + 1. below the classification floor -> "moved X -> Y devices; under the N-device floor, too + small to classify -- watch, do not page" + 2. within its own normal band -> "swings +/- N pts routinely; this week is inside that band" + 3. not novel (ONGOING) -> "elevated for K weeks and flat; tracked in the fold below" + + Render it as a plain reconciliation line, NOT a callout -- it must read as a dismissal, not a + finding, or it re-creates the noise this skill exists to remove: + + <div class="reconcile-note"> + <strong>Scoreboard reds not escalated:</strong> + <span class="mono">EXAMPLE_scenario</span> — 565 → 732 bad-outcome devices, under the + 1,000-device classification floor; too small to separate signal from noise. Watching. + </div> +--> + <div class="callout urgent"> <div class="callout-title"><span class="icon">🔴</span> New this week <span class="count">— clean step change off a flat baseline</span></div> <div class="item-list"> @@ -619,8 +640,29 @@ <h2 id="attention">🚨 Needs attention</h2> devices is noise, and promoting it here is exactly what trains an on-call engineer to skim. --> +<!-- ================ SCENARIO SCOREBOARD ================ --> +<h2 id="scoreboard">Scenario scoreboard <span class="count">— one row per Authenticator scenario family</span></h2> +<div class="table-card"><table> + <thead><tr><th>Scenario</th><th class="num">Initiated</th><th class="num">Success rate</th><th class="num">Δ success rate (pts)</th><th class="num">Failure rate</th><th class="num">Unknown rate</th><th class="num">Δ unknown (pts)</th><th class="num">Devices</th><th>8-week sparkline</th><th>Status</th></tr></thead> + <tbody> +<tr><td class="code-cell">Passkey WebAuthN Registration</td><td class="num">48,320</td><td class="num good">91.8%</td><td class="num good">+0.6</td><td class="num">4.7%</td><td class="num ">3.5%</td><td class="num muted">-0.3</td><td class="num">38,940</td><td class="spark-cell"><div class="trend" data-trend='[89.7,90.1,90.6,91,91.3,91.5,91.2,91.8]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> +<tr><td class="code-cell">Passkey InApp Registration</td><td class="num">22,410</td><td class="num ">88.2%</td><td class="num muted">-0.4</td><td class="num">6.9%</td><td class="num ">4.9%</td><td class="num muted">+0.2</td><td class="num">19,120</td><td class="spark-cell"><div class="trend" data-trend='[87.9,88,88.4,88.8,88.5,88.7,88.6,88.2]' data-color="#bf8700" data-w="160" data-h="28"></div></td><td><span class="tag tag-warn">watch</span></td></tr> +<tr><td class="code-cell">Passkey WebAuthN Authentication</td><td class="num">612,900</td><td class="num bad">94.1%</td><td class="num bad">-1.4</td><td class="num">3.2%</td><td class="num bad">2.7%</td><td class="num bad">+0.8</td><td class="num">402,100</td><td class="spark-cell"><div class="trend" data-trend='[96.4,96.2,95.8,95.5,95.3,95.1,95.5,94.1]' data-color="#cf222e" data-w="160" data-h="28"></div></td><td><span class="tag tag-bad">regressed</span></td></tr> +<tr><td class="code-cell">Entra MFA Registration (QR)</td><td class="num">1,245,000</td><td class="num ">97.6%</td><td class="num muted">+0.3</td><td class="num">1.2%</td><td class="num ">1.2%</td><td class="num muted">-0.2</td><td class="num">902,400</td><td class="spark-cell"><div class="trend" data-trend='[97.1,97,97.2,97.3,97.4,97.5,97.3,97.6]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> +<tr><td class="code-cell">Entra MFA Registration (No-QR)</td><td class="num">336,200</td><td class="num ">95.0%</td><td class="num muted">-0.1</td><td class="num">2.8%</td><td class="num ">2.2%</td><td class="num muted">0.0</td><td class="num">248,900</td><td class="spark-cell"><div class="trend" data-trend='[95.1,95.4,95.2,95.3,95,95.2,95.1,95]' data-color="#656d76" data-w="160" data-h="28"></div></td><td><span class="tag tag-muted">flat</span></td></tr> +<tr><td class="code-cell">Entra MFA PN+CFA</td><td class="num">42,800,000</td><td class="num bad">98.4%</td><td class="num bad">-0.9</td><td class="num">0.7%</td><td class="num bad">0.9%</td><td class="num muted">+0.5</td><td class="num">18,700,000</td><td class="spark-cell"><div class="trend" data-trend='[99.1,99,98.9,98.8,98.6,98.5,99,98.4]' data-color="#cf222e" data-w="160" data-h="28"></div></td><td><span class="tag tag-bad">regressed</span></td></tr> +<tr><td class="code-cell">Entra PSI Registration</td><td class="num">188,500</td><td class="num good">93.6%</td><td class="num good">+1.2</td><td class="num">4.4%</td><td class="num good">2.0%</td><td class="num good">-0.6</td><td class="num">131,800</td><td class="spark-cell"><div class="trend" data-trend='[91.9,92.2,92.6,92.8,93,93.2,92.4,93.6]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">win</span></td></tr> +<tr><td class="code-cell">Entra PSI PN Registration</td><td class="num">74,900</td><td class="num ">92.1%</td><td class="num muted">+0.4</td><td class="num">5.1%</td><td class="num ">2.8%</td><td class="num muted">-0.2</td><td class="num">58,600</td><td class="spark-cell"><div class="trend" data-trend='[91.2,91.5,91.4,91.8,91.7,92,91.8,92.1]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> +<tr><td class="code-cell">Entra PSI PN+CFA</td><td class="num">4,920,000</td><td class="num bad">96.8%</td><td class="num bad">-0.6</td><td class="num">1.4%</td><td class="num bad">1.8%</td><td class="num muted">+0.4</td><td class="num">2,860,000</td><td class="spark-cell"><div class="trend" data-trend='[97.4,97.3,97.2,97.1,97,97,97.4,96.8]' data-color="#bf8700" data-w="160" data-h="28"></div></td><td><span class="tag tag-warn">watch</span></td></tr> +<tr><td class="code-cell">MSA NGC Registration</td><td class="num">91,300</td><td class="num good">89.5%</td><td class="num good">+0.8</td><td class="num">7.2%</td><td class="num good">3.3%</td><td class="num muted">-0.4</td><td class="num">72,100</td><td class="spark-cell"><div class="trend" data-trend='[88.1,88.4,88.9,89,88.7,89.2,88.9,89.5]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">healthy</span></td></tr> +<tr><td class="code-cell">MSA SA Registration</td><td class="num">19,800</td><td class="num ">86.4%</td><td class="num muted">+0.2</td><td class="num">8.4%</td><td class="num ">5.2%</td><td class="num muted">-0.1</td><td class="num">16,400</td><td class="spark-cell"><div class="trend" data-trend='[85.8,86,85.9,86.2,86.1,86.4,86.2,86.4]' data-color="#656d76" data-w="160" data-h="28"></div></td><td><span class="tag tag-muted">flat</span></td></tr> +<tr><td class="code-cell">MSA NGC PN+CFA</td><td class="num">3,760,000</td><td class="num ">97.2%</td><td class="num muted">+0.5</td><td class="num">1.1%</td><td class="num good">1.7%</td><td class="num muted">-0.4</td><td class="num">1,940,000</td><td class="spark-cell"><div class="trend" data-trend='[96.2,96.4,96.6,96.8,96.9,97,96.8,97.2]' data-color="#1a7f37" data-w="160" data-h="28"></div></td><td><span class="tag" style="background:#dafbe1;color:#1a7f37;">win</span></td></tr> +<tr><td class="code-cell">MSA SA PN+CFA</td><td class="num">1,185,000</td><td class="num ">95.4%</td><td class="num muted">-0.2</td><td class="num">2.2%</td><td class="num ">2.4%</td><td class="num muted">+0.1</td><td class="num">764,000</td><td class="spark-cell"><div class="trend" data-trend='[95.3,95.5,95.2,95.6,95.4,95.5,95.6,95.4]' data-color="#656d76" data-w="160" data-h="28"></div></td><td><span class="tag tag-muted">flat</span></td></tr> + </tbody> +</table></div> + <!-- ================ 60-DAY TREND ================ --> -<h2 id="trend60d">📈 60-day per-scenario trend <span class="count">— weekly buckets; final bar in progress</span></h2> +<h2 id="trend60d">📈 60-day per-scenario trend <span class="count">— rolling 7-day buckets; final bar = this week's window</span></h2> <div class="table-card"><table> <thead><tr><th>Scenario</th><th class="num">First week</th><th class="num">Last week</th><th class="num">First → last delta</th><th>Weekly sparkline</th><th>Classification</th></tr></thead> <tbody> diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html index f3b563bc..10798747 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/report-template.html @@ -435,6 +435,15 @@ border-radius: 6px; padding: 12px 14px; margin: 12px 0; font-size: 13px; } .quiet-banner strong { color: #1a7f37; } + /* Deliberately muted: a reconciliation note is a DISMISSAL, not a finding. If it + competes visually with .callout it re-creates the noise the noise gate removes. */ + .reconcile-note { + border-left: 3px solid #d0d7de; background: #f6f8fa; color: #57606a; + border-radius: 0 6px 6px 0; padding: 9px 12px; margin: 10px 0; font-size: 12.5px; + line-height: 1.55; + } + .reconcile-note strong { color: #424a53; font-weight: 600; } + .reconcile-note .mono { color: #24292f; } </style> </head> <body> @@ -534,6 +543,34 @@ <h2 id="attention">🚨 Things that need attention this week</h2> </div> --> +<!-- + ⚠️ MANDATORY: RECONCILE EVERY RED/AMBER PILL IN THE WoW TABLES. + The WoW / error-code tables colour rows from their own rolling delta. This section is + populated from classify-novelty.js's novelty verdict. Both are correct, but they answer + DIFFERENT questions -- so a code can be red down there and legitimately absent from here. + A reader who spots that mismatch and is given no explanation concludes the report is broken. + + Since the trend buckets were realigned to the report's rolling window, the two bases agree on + DIRECTION and MAGNITUDE. What can still diverge is CLASSIFIABILITY: any code whose peak stays + under the peak-floor (10,000 devices) is excluded from classification entirely and can never + appear here, however sharply it moves. + + RULE: every `tag-bad` / `tag-warn` row in the WoW tables must be either promoted here or named + in a reconciliation note with the reason it is not escalated. validate-report.ps1 hard-fails an + unreconciled red pill. Test the reasons in this order: + 1. below the classification floor -> "moved X -> Y devices; under the 10,000-device floor" + 2. within its own normal band -> "swings +/- N% routinely; this week is inside that band" + 3. not novel (ONGOING) -> "elevated for K weeks and flat; tracked in the fold below" + + Render as a muted dismissal line, NOT a callout: + + <div class="reconcile-note"> + <strong>Table reds not escalated:</strong> + <span class="mono">EXAMPLE_code</span> — 6,100 → 8,400 devices, under the 10,000-device + classification floor; too small to separate signal from noise. Watching. + </div> +--> + <div class="callout watch"> <div class="callout-title"><span class="icon">ℹ️</span> Denominator caveat — read this first</div> <p>The headline <code>BrokerAdoptionStats</code> device count dropped <strong>−18.6% WoW</strong> (1.52 B → 1.24 B), but this <strong>is not a real fleet shrink</strong>. The drop is fully explained by three low-value spans deflating as the 16.0.1 rollout completes:</p> @@ -1210,11 +1247,11 @@ <h2 id="appendix">Appendix</h2> all | join kind=inner ok on week | project week, reqRel=round(100.0*okReq/allReq,3), devRel=round(100.0*okDev/allDev,3) | order by week asc</pre> - <p><strong>2. 60-day error trend (bucketed in post-processing):</strong></p> - <pre style="background:#f6f8fa;padding:10px;border-radius:6px;font-size:11.5px;overflow-x:auto;">ErrorStats | where EventInfo_Time > ago(70d) + <p><strong>2. 60-day error trend (rolling 7-day buckets; bucketed in post-processing):</strong></p> + <pre style="background:#f6f8fa;padding:10px;border-radius:6px;font-size:11.5px;overflow-x:auto;">ErrorStats | where EventInfo_Time >= datetime(<TREND_START>) and EventInfo_Time < datetime(<TREND_END>) | where isnotempty(error_code) and error_code != 'success' - | summarize errs=sum(countOverall), devs=sum(countDevices) - by week=startofweek(EventInfo_Time), error_code + | summarize errs=sum(countOverall), devs=dcount_hll(hll_merge(countDevicesHll)) + by week=bin_at(EventInfo_Time, 7d, datetime(<TREND_END>)), error_code | order by error_code asc, week asc</pre> <p><strong>3. Spike attribution (per error, per dimension):</strong></p> <pre style="background:#f6f8fa;padding:10px;border-radius:6px;font-size:11.5px;overflow-x:auto;">let codes = dynamic(['no_tokens_found','unauthorized_client','Code:-6', diff --git a/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md b/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md index 7da4d553..3b394731 100644 --- a/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md +++ b/.github/skills/oncall-weekly-telemetry-report/assets/templates/template-readme.md @@ -13,7 +13,7 @@ not a tokenized skeleton. The right mental model is: | Region | What to update | |---|---| | `<title>` and `<h1>` block | **All header dates (title, meta line, Generated) are auto-stamped by `bootstrap-report.ps1`** based on the resolved rolling 7-day window. You do not touch them. The `<h1>` text itself ("Android Broker · Weekly On-Call Report") is canonical prose — leave as-is. | -| KPI tiles (`.kpi-grid`) | Value, delta, `data-spark` array (8–9 numbers) per tile | +| KPI tiles (`.kpi-grid`) | Value, delta, `data-spark` array (8 rolling-week numbers) per tile | | 🚨 Needs-attention callouts (`.callout.urgent` / `.watch` / `.win`) | Replace bullet list with current-week findings; keep the 4 callout categories | | 📈 60-day trend tables | Rows + `.trend` sparkline arrays, generated by `bucket-trends.js` (4 runs, union of regressions) | | 🔎 Spike-attribution cards (`.attr-card`) | One card per regression. **Use [`templates/spike-card.html`](templates/spike-card.html) as the per-card skeleton.** Replace dim percentages, throw-site, PR list, etc. | @@ -31,7 +31,7 @@ not a tokenized skeleton. The right mental model is: | **`<title>`** | Skill identifier + end-date, e.g. `Android Broker · On-Call Report — Last 7 days ending 2026-07-09`. | Auto-stamped by bootstrap. | | **`<div class="meta">` line 1: `Last 7 days: <curStart> → <curEnd>`** | The rolling primary window the data covers. `curEnd` = start-of-day UTC on `-EndDate` (default: today). `curStart = curEnd − 7d`. Half-open — the end date is the exclusive upper bound. | Auto-stamped by bootstrap. | | **`<div class="meta">` line 1: `vs <prevStart> → <prevEnd>`** | The prior 7-day baseline the WoW deltas compare against. `prevStart = curEnd − 14d`, `prevEnd = curStart`. | Auto-stamped by bootstrap. | -| **`<div class="meta">` line 2: 60-day trend** | The literal last 60 days ending today (`sixtyDayStart = curEnd − 60d`, `sixtyDayEnd = curEnd`, exclusive). Sun-Sat weekly-bucketed; the final bar is the current in-progress (partial) week. Displayed as `sixtyDayStart → (sixtyDayEnd − 1)` (last full calendar day). Regression/improvement deltas are computed on complete weeks only (cutoff `startofweek(curEnd)`); the partial week is charted but excluded. | Auto-stamped by bootstrap. | +| **`<div class="meta">` line 2: 60-day trend** | The literal last 60 days ending today (`sixtyDayStart = curEnd − 60d`, `sixtyDayEnd = curEnd`, exclusive). Weekly buckets are rolling 7-day windows anchored at `curEnd` via `bin_at`; the newest bucket is `[curEnd − 7d, curEnd)`, identical to the displayed WoW window. Displayed as `sixtyDayStart → (sixtyDayEnd − 1)` (last full calendar day). Because 60 is not a multiple of 7, the oldest bucket is the short one and is dropped before classification, leaving 8 complete rolling weeks. | Auto-stamped by bootstrap. | | **`Generated <strong>…</strong>`** | The UTC date the report was produced. | Auto-stamped by bootstrap to `[datetime]::UtcNow.Date`. Do **not** hardcode it in any generator/assembler. The Generated date is UTC by contract, so if you rebuild the body programmatically re-derive it with a **UTC-date** formatter (`new Date().toISOString().slice(0,10)` in Node, `[datetime]::UtcNow.ToString('yyyy-MM-dd')` in PowerShell) so it always matches bootstrap's `[datetime]::UtcNow.Date` and the UTC window boundaries — never copy a literal from a prior file. Avoid local-timezone formatters (`toLocaleDateString()`, `(Get-Date).ToString(...)`); they can disagree with the UTC window by a day. | Manual override: `bootstrap-report.ps1 -EndDate YYYY-MM-DD` reproduces the report for any historic end-date (e.g. `-EndDate 2026-07-02` gives the window Jun 25 → Jul 2). The window computation is deterministic — the same `-EndDate` always resolves to the same start/prev boundaries. @@ -128,11 +128,11 @@ Required spark/trend coverage in every report: | Where | Attribute | Length | Color (see palette below) | |---|---|---|---| -| Every KPI tile in `.kpi-grid` (Top-line health) | `<div class="spark" data-spark='[...]' data-color="..."></div>` inside the tile | 8–9 weekly values | blue/green/dark-blue per metric semantic | -| **Every visible row in the Section 2 attention list** — including the wins | `<span class="item-spark trend" data-trend='[...]' data-color="..." data-w="120" data-h="22"></span>` right after `.item-name` | 9 weekly values | red worsening / orange accelerating / green improving / amber volatile | -| Broker 60-day section: **only the promoted slow burns** (rising on 60d, not already in Section 2 — often zero) | `<span class="trend" data-trend='[...]' data-color="..." data-w="160"></span>` in the trajectory cell | ~9 weekly values (incl. the current partial week as the final point) | red regression / amber spike / green improvement | -| Authenticator 60-day per-scenario table — **all ~13 scenarios** (it is the scoreboard, not an overflow list) | `<div class="trend" data-trend='[...]' data-color="..." data-w="220" data-h="32"></div>` in the sparkline cell | 8–9 weekly values | red regression / amber spike / green improvement / grey flat | -| Every row in the error-codes WoW table and error-types WoW table | `<span class="trend" data-trend='[...]' data-color="..."></span>` in the 60d-trend column | 8 complete weekly values (no partial week — see `wow-table-sparkline-series.kql`) | same palette | +| Every KPI tile in `.kpi-grid` (Top-line health) | `<div class="spark" data-spark='[...]' data-color="..."></div>` inside the tile | 8 rolling-week values | blue/green/dark-blue per metric semantic | +| **Every visible row in the Section 2 attention list** — including the wins | `<span class="item-spark trend" data-trend='[...]' data-color="..." data-w="120" data-h="22"></span>` right after `.item-name` | 8 rolling-week values | red worsening / orange accelerating / green improving / amber volatile | +| Broker 60-day section: **only the promoted slow burns** (rising on 60d, not already in Section 2 — often zero) | `<span class="trend" data-trend='[...]' data-color="..." data-w="160"></span>` in the trajectory cell | 8 complete rolling-week values | red regression / amber spike / green improvement | +| Authenticator 60-day per-scenario table — **all ~13 scenarios** (it is the scoreboard, not an overflow list) | `<div class="trend" data-trend='[...]' data-color="..." data-w="220" data-h="32"></div>` in the sparkline cell | 8 complete rolling-week values | red regression / amber spike / green improvement / grey flat | +| Every row in the error-codes WoW table and error-types WoW table | `<span class="trend" data-trend='[...]' data-color="..."></span>` in the 60d-trend column | 8 curEnd-anchored rolling-week values | same palette | > **📌 The two WoW reference tables keep a sparkline on every row — deliberate exemption, decided > explicitly. Do not strip them when reducing noise.** They are **lookup tables**: the reader arrives @@ -306,4 +306,3 @@ Render pattern: If your table has zero `.pill-bad` rows the week was unusually quiet — double-check the WoW-movers and 60d bucketing passes ran. If every row is `.pill-bad` you've mis-categorized. -