[BUG] BigQueryAgentAnalyticsPlugin re-runs all view DDL on every re-initialisation, on the request path (2.3.0 -> 2.8.0 regression)

[lupuletic]

TL;DR — Since 2.8.0, BigQueryAgentAnalyticsPlugin re-runs its full table/view readiness pass on every re-initialisation instead of once per process. Each pass issues one CREATE OR REPLACE VIEW statement per entry in _EVENT_VIEW_DEFS (25 at HEAD), and it is awaited from before_run_callback, so the DDL sits on the agent request path. Any deployment that shares one plugin instance across short-lived Runners re-runs all 25 statements per request. In our production service this produced ~19,000 view-DDL jobs/hour, exhausted the BigQuery per-table daily quota in 2h18m, and added ~25s to median response latency.

---

For humans

What we saw

We upgraded a production Google ADK service from 2.3.0 to 2.8.0. Within five minutes of the rollout, median server-side stream duration went from about 10s to about 35s, and the tail pinned near 60s. About two hours later our error logs filled with BigQuery 403s.

The 403s were quotaExceeded / load_job_per_table.long, raised from _create_analytics_views. They were a symptom, not the cause. The cause was the sheer number of view-creation statements being issued.

Job counts from INFORMATION_SCHEMA.JOBS_BY_PROJECT, filtered to statement_type = "CREATE_VIEW" on the plugin's dataset:

Period	View-DDL jobs per hour
ADK 2.3.0 (previous 3 days)	21 to 337
ADK 2.8.0, first full hour	18,881
ADK 2.8.0, second full hour	18,917

Nothing else changed. Same traffic, same config, same service account, same dataset. Only the ADK version.

Why it happens

In 2.3.0, _lazy_setup gated the readiness pass behind the schema cache:

if not self._schema:
    self._schema = _get_events_schema()
    await loop.run_in_executor(self._executor, self._ensure_schema_exists)

_schema is pure data and survives close(), so readiness ran at most once per process no matter how often the plugin cycled.

At HEAD the gate is gone, deliberately:

if self._schema is None:
    self._schema = _project_schema(_get_events_schema(), self._denied_columns)
# Run table readiness on EVERY setup attempt until one succeeds: the
# cached _schema must not gate it, or a failed first attempt would skip
# the table check on retry and mark the plugin started against a
# missing/unready table. Once _started is True,
# _lazy_setup returns early above, so the steady state pays no extra RPC.
await loop.run_in_executor(executor, self._ensure_schema_exists)

The intent is right: a failed first attempt should not be skipped on retry. But the final sentence is the load-bearing assumption, and it only holds if the plugin is initialised once per process. It is false whenever the plugin is closed and reused, because close() clears _started.

That is not a hypothetical. Runner.close() calls PluginManager.close(), which calls plugin.close() on every registered plugin, including plugins the caller owns and passed in via App. A host that builds a short-lived Runner per request over one shared App therefore shuts the plugin down after every request, and the next request pays a full readiness pass.

ADK already knows this hazard. PluginManager.set_skip_closing_plugins(True) exists for exactly this case, and tools/agent_tool.py uses it when it creates a nested Runner over shared plugins. It is simply not applied on the other paths that do the same thing. We have filed the corresponding fix against the AG-UI middleware, but we think the plugin should also be robust to being re-initialised, because "readiness already verified" is cheap to remember and expensive to redo.

Why this is worse than an extra RPC

Three things compound:

• It is on the request path. before_run_callback awaits _ensure_started(). The 25 statements run serially, ahead of the first token, every time setup re-runs. That is the latency regression, not just wasted quota.
• It consumes a hard daily quota. CREATE OR REPLACE VIEW counts against a per-table daily limit. At ~790 passes/hour we exhausted it in 2h18m, after which the plugin logged a stack trace per view per attempt. Analytics ingestion dropped about 20%.
• It fails silently until then. When the generation changes mid-setup, the success branch declines to set _started and logs nothing at all. Across 2h44m we saw only 12 Failed to initialize BigQuery Plugin lines while setup actually ran roughly 2,000 times. There is no signal that the plugin is looping until the quota dies.

What we would like

Remember that readiness succeeded, in a flag that survives close(), and skip the pass when it did. That keeps the 2.8.0 intent (a failed attempt still retries) while restoring the 2.3.0 cost profile (a successful one is not repeated). We are happy to open the PR; a draft is linked below.

---

For AI agents

defect:
  id: bqaa-readiness-reruns-per-init
  component: google.adk.plugins.bigquery_agent_analytics_plugin.BigQueryAgentAnalyticsPlugin
  class: [performance-regression, quota-exhaustion, latency-on-request-path, silent-failure]
  introduced_in: "2.8.0"
  last_good: "2.3.0"
  status_at_head: present
  head_verified: 852b575e

symptom:
  - "403 quotaExceeded (load_job_per_table.long) raised from _create_analytics_views"
  - "median agent latency +25s; p95/p99 pinned near 60s"
  - "CREATE_VIEW job rate rises ~90x with no traffic or config change"

call_chain:
  - file: src/google/adk/plugins/bigquery_agent_analytics_plugin.py
    symbol: before_run_callback
    line: 6970
    note: "request-path entry point; awaits _ensure_started()"
  - symbol: _ensure_started
    line: 5636
    note: "returns early only when self._started is True"
  - symbol: _lazy_setup
    line: 4716
  - symbol: _lazy_setup
    line: 4786
    note: "DEFECT SITE — _ensure_schema_exists runs unconditionally on every setup attempt"
  - symbol: _ensure_schema_exists
    line: 4861
  - symbol: _create_analytics_views
    line: 5094
    note: "emits one CREATE OR REPLACE VIEW per _EVENT_VIEW_DEFS entry (25 at HEAD, defined line 3912)"

state_reset_path:
  - file: src/google/adk/runners.py
    line: 2149
    note: "Runner.close() -> PluginManager.close()"
  - file: src/google/adk/plugins/plugin_manager.py
    line: 404
    note: "PluginManager.close() -> plugin.close() for every registered plugin, including caller-owned ones"
  - file: src/google/adk/plugins/bigquery_agent_analytics_plugin.py
    line: 5612
    note: "plugin.close() -> shutdown()"
  - line: 5196
    note: "shutdown() bumps _generation and clears _started (line 5217)"

existing_guard_not_applied:
  api: PluginManager.set_skip_closing_plugins
  defined: src/google/adk/plugins/plugin_manager.py:95
  honoured: src/google/adk/plugins/plugin_manager.py:392
  only_caller: src/google/adk/tools/agent_tool.py:278
  note: "guard exists for exactly this shared-plugin case but is applied on one path only"

secondary_defect:
  id: bqaa-silent-aborted-setup
  file: src/google/adk/plugins/bigquery_agent_analytics_plugin.py
  line: 5801
  note: >-
    On the success path, a _generation mismatch skips `self._started = True`
    and returns without logging. A plugin looping through full setup emits no
    diagnostic. Observed 12 failure logs against ~2000 actual setup runs.

reproduce:
  minimal: |
    plugin = BigQueryAgentAnalyticsPlugin(project_id=..., dataset_id=..., table_id=...)
    app = App(root_agent=agent, plugins=[plugin])
    for _ in range(N):
        runner = Runner(app=app.model_copy(update={"root_agent": agent}), ...)
        async for _ in runner.run_async(...):
            pass
        await runner.close()          # clears plugin._started
  observe: |
    SELECT COUNT(*) FROM `region-<loc>`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
    WHERE statement_type = "CREATE_VIEW" AND destination_table.dataset_id = "<dataset>";
  expected: "O(1) in N (25 total, one pass)"
  actual: "O(N) in N (25 * N)"

production_evidence:
  window: "2h44m on one Cloud Run service, ~6-9 instances"
  create_view_jobs_per_hour_before: "21-337"
  create_view_jobs_per_hour_after: "~18900"
  setup_passes_per_hour_derived: "~756 (18881 / 25)"
  runner_close_events_per_10min: 325
  plugin_register_events_per_10min: 329
  batch_writer_drains_per_10min: 101
  plugin_constructor_invocations_per_10min: 0   # proves a single shared instance
  fork_events_per_10min: 0
  instance_starts_in_window: 0
  quota_exhausted_after: "2h18m"
  daily_quota: "1500 table-modification jobs per view per day"

proposed_fix:
  primary:
    site: "src/google/adk/plugins/bigquery_agent_analytics_plugin.py:4786"
    change: >-
      Memoise readiness success in an attribute preserved across close()
      (alongside _schema, which already survives), and gate the
      _ensure_schema_exists call on it. Clear it only when readiness fails.
    preserves: "failed attempts still retry on the next setup (the 2.8.0 intent)"
    fixes: "successful readiness is not repeated per re-initialisation (the 2.3.0 cost)"
  secondary:
    site: "src/google/adk/plugins/bigquery_agent_analytics_plugin.py:5801"
    change: "log at WARNING when a generation mismatch aborts an otherwise successful setup"

acceptance_criteria:
  - "N close/re-init cycles produce 25 CREATE OR REPLACE VIEW statements total, not 25*N"
  - "a readiness attempt that raises is still retried on the next setup attempt"
  - "no new RPC is added to the steady-state path"
  - "a setup aborted by generation mismatch emits exactly one log line"

non_goals:
  - "changing _EVENT_VIEW_DEFS or the view SQL"
  - "changing the backoff schedule in _ensure_started"