Dashboard setup sources: run hidden setup queries before filters and panels

[BorisTyshkevich]

Dashboard setup sources: run hidden setup queries before filters and panels

Problem

Some dashboards need to compute temporary shared state once before running their visible panels.

Example:

CREATE TEMPORARY TABLE daily AS
SELECT
    toDate(t) AS d,
    count() AS c,
    avg(v) AS a
FROM big.events
WHERE d >= {from:Date}
GROUP BY d

Visible dashboard queries can then read daily instead of repeating the heavy aggregation.

The previous design modeled this as a setup panel type. That is conceptually wrong: setup has no visual representation and does not belong in the panel renderer registry.

Goal

Allow a favorited saved query to participate in the dashboard as a Setup source:

dashboard: {
  role: 'setup'
}

A Setup source:

• runs before Filter sources and displayed Panels;
• never occupies a grid cell;
• discards output;
• surfaces errors prominently;
• uses a shared ClickHouse HTTP session so temporary tables and settings are visible to later dashboard queries.

Relationship to Panels

Panels answer:

How should this result be displayed?

Setup answers:

What must run before the dashboard's displayed queries?

Therefore:

panel      // visualization configuration
dashboard  // execution role

A Setup query may retain a dormant panel config, but the dashboard does not render it while the role is Setup.

Library authoring UX

Extend the saved-query Dashboard role selector introduced by #160:

Dashboard
[✓] Include in dashboard

Use as
( ) Panel
( ) Filter source
(o) Setup

Description:

Run this query before filter sources and panels.
Its result is not displayed.

The Library badge is:

[Setup]

The workbench title badge is:

Setup source

Changing to Setup:

• does not automatically favorite the query;
• does not delete panel configuration;
• does not change normal SQL Browser Run behavior.

Changing back to Panel restores the previous panel configuration.

Saved format

dashboard: {
  role: 'setup'
}

Unknown fields remain preserved under the same additive compatibility rules as the other dashboard roles.

Dashboard partition

Partition favorited queries before execution:

1. Setup sources;
2. Filter sources from Dashboard: query-backed filter options — single-select MVP #160;
3. displayed Panels.

Setup and Filter sources never get grid slots and do not count as “not shown.”

Dashboards with no Setup sources retain the current parallel, sessionless behavior.

Session constraint

ClickHouse HTTP sessions are exclusive: two concurrent requests using the same session_id can fail with SESSION_IS_LOCKED.

Temporary tables are visible only inside that session.

Therefore v1 uses one serialized session wave:

Setup sources, Library order
→ Filter-source option queries and selection reconciliation
→ displayed Panels

All requests in this wave use the same session.

The dashboard's ordinary bounded parallel execution remains unchanged when there are no Setup sources.

Safety envelope

Every automatically executed Setup statement uses:

readonly = 2
FORMAT Null

Allowed statement classes:

• SET;
• CREATE TEMPORARY TABLE ... AS SELECT;
• CREATE OR REPLACE TEMPORARY TABLE ... AS SELECT, when supported;
• row-returning read statements, with output discarded.

Reject client-side:

• permanent CREATE;
• DROP;
• ALTER;
• INSERT;
• mutations and other write classes.

The first rejected statement stops that Setup source and blocks the wave before any visible Panel runs.

Compatibility gates

Before release, verify on every supported ClickHouse version:

1. CREATE [OR REPLACE] TEMPORARY TABLE ... AS SELECT works under readonly = 2;
2. {name:Type} HTTP params bind inside that statement.

If either gate fails on a supported version, redesign the feature rather than weakening the safety boundary.

Multi-statement behavior

A Setup saved query may contain multiple statements.

Rules:

• split with the shared SQL statement splitter;
• execute sequentially;
• bind parameters in every allowed statement;
• use the parameter pipeline with bindPolicy: 'all';
• discard all outputs at the protocol level;
• stop on first error;
• continue to the next Setup source only after the current one succeeds.

Multiple Setup sources run in Library order.

No dependency graph between Setup sources in v1.

Parameters

Setup SQL participates in shared dashboard parameter analysis:

• its {name:Type} fields appear in the common filter bar;
• optional SQL blocks work normally;
• validation, relative time, activation, serialization, and persisted values use the shared pipeline.

v1 cycle prohibition

A Setup source may not reference a parameter whose choices are supplied by a query-backed Filter source.

Reason:

• Setup must run before a Filter source that reads a temporary table;
• the Setup source would simultaneously need the Filter source's selected value first.

Detect this configuration before execution and show a clear diagnostic naming both saved queries and the parameter.

Plain, Enum, recent, relative-time, or manually entered shared parameters are allowed.

Session lifecycle

Use a fresh session whenever setup state must be recomputed:

• dashboard open;
• Dashboard Refresh;
• any committed change to a parameter referenced by a Setup source;
• one recovery attempt after session expiry or unknown-temporary-table failure.

Fresh-session wave:

1. generate a new session_id;
2. prepare one parameter batch with one wallNowMs;
3. run all Setup sources in order;
4. run Filter sources and reconcile selections;
5. run displayed Panels;
6. abandon the previous session.

No explicit DROP/recreate cleanup is required.

Non-setup filter changes

A parameter change that affects no Setup source:

• reuses the current live session;
• follows the ordinary affected-panel rerun behavior;
• does not rerun Setup sources.

Conservative v1 rule:

• any Setup recomputation reruns every displayed Panel.

Errors and status

Setup is hidden, but its state must be visible.

Dashboard header indicator:

2 setup sources · 340 ms

Expanding it shows:

• saved-query name;
• statement number;
• status;
• elapsed time;
• error text.

If a Setup source fails:

• show a prominent setup-error banner;
• stop the current wave;
• do not run Filter sources or Panels against incomplete setup state;
• show Setup failed in every blocked Panel slot.

Session expiry recovery

Use a generous session_timeout.

If a Filter source or Panel fails with an unknown-temporary-table/session-expiry-class error:

1. start one fresh-session setup wave;
2. retry the complete wave once;
3. surface a second failure normally.

Never retry indefinitely.

Workbench behavior

No special Setup editor is required.

The ordinary SQL Browser Run path remains the place to develop and debug the script:

• sticky session behavior;
• sequential statements;
• normal per-statement results and errors.

The Dashboard role controls dashboard orchestration only.

Network seam

Provide setup-specific execution that accepts:

{
  sql,
  args,
  sessionId,
  sessionTimeout,
  signal
}

It must:

• enforce readonly: 2;
• append FORMAT Null safely or select an equivalent null-output request path;
• return timing and structured error data;
• never download result rows;
• support abort.

Thread session information through Filter-source and Panel query helpers when session mode is active.

Files

Expected changes:

• src/core/saved-io.js
  • add dashboard.role = 'setup'.
• src/ui/saved-history.js
  • add Setup to the Dashboard role editor and badges.
• src/core/dashboard.js
  • setup-source partitioning, validation, parameter dependency check, wave planning.
• src/core/sql-split.js
  • reuse canonical statement classification.
• src/ui/dashboard.js
  • serialized session-mode orchestration, status indicator, banner and blocked slots.
• src/net/ch-client.js
  • setup request path and session threading.
• src/styles.css
  • setup status and error UI.
• tests for all affected modules.

No new runtime dependency.

Tests

Role and partition

• Setup role round-trips through every saved-query ingress;
• unstarred Setup source is absent;
• Setup never gets a grid slot;
• panel config survives role changes;
• dashboard without Setup sources remains parallel and sessionless.

Validation and safety

• allowed statement classes pass;
• permanent/write classes are rejected before network;
• every setup request has readonly:2;
• output is discarded with FORMAT Null;
• parameter binding uses bindPolicy:'all';
• Filter-source target dependency cycle is diagnosed.

Execution

• Setup sources run in Library order;
• statements within one source run sequentially;
• first error stops the source and wave;
• no two shared-session requests are in flight simultaneously;
• order is Setup → Filter sources/reconciliation → Panels;
• one wallNowMs is used for the whole wave;
• setup-affecting parameter changes create a fresh session and rerun all panels;
• unrelated parameter changes reuse the live session and normal affected set.

Errors and recovery

• setup failure produces banner, detail and blocked Panel state;
• output rows are never parsed/downloaded;
• session-expiry/unknown-table failure triggers one fresh-session retry;
• second failure is surfaced without another retry.

Live compatibility

On every supported server:

• temporary CREATE AS SELECT works with readonly=2;
• params bind inside temporary CREATE AS SELECT.

Acceptance criteria

• A saved query can be assigned the Setup dashboard role in the Library.
• Favorited Setup sources run before Filter sources and Panels and never render as Panels.
• Dashboards with Setup sources use one serialized shared session.
• Dashboards without Setup sources retain current parallel behavior.
• Setup output is discarded at the protocol level.
• Setup errors block the wave and are clearly visible.
• Automatic Setup execution is restricted to the documented safe statement allowlist under readonly=2.
• Setup parameters use the shared pipeline and bindPolicy:'all'.
• A Setup source depending on a query-backed Filter-source target is rejected in v1.
• Setup-affecting changes use a fresh session and rerun all Panels.
• Session expiry gets one full recovery attempt.
• Compatibility gates pass; coverage gates hold; no new runtime dependency.

Non-goals

• A setup panel renderer or Panel-registry arm.
• Permanent DDL or writes.
• Parallel queries inside one ClickHouse session.
• Dependency analysis between multiple Setup sources.
• Detecting exactly which Panels use which temporary tables.
• Session pools.
• Setup sources depending on query-backed Filter-source selections.
• Automatic refresh/polling.

[BorisTyshkevich]

Body rewritten to model setup as dashboard.role='setup', not a Panel registry arm.
Execution/session semantics remain the feature's responsibility.