Fixes #31582: Prevent asynchronous database pool exhaustion - #31583
Fixes #31582: Prevent asynchronous database pool exhaustion#31583harshach wants to merge 3 commits into
Conversation
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
✅ PR checks passedThe linked issue has a description and all required Shipping project fields set. Thanks! |
There was a problem hiding this comment.
Pull request overview
This PR addresses issue #31582 by preventing database-heavy background workloads from exhausting the Hikari connection pool and starving request traffic. It introduces bounded execution lanes for DB-consuming async work, improves cluster-wide scheduling behavior for Data Insights, batches high-fan-out cleanup/writes, bounds RDF indexing concurrency/caches, and tightens DB timeout defaults.
Changes:
- Introduce a DB-bounded async executor lane with per-operation backlog metrics, and route DB-heavy background tasks through it.
- Reduce DB fan-out by batching test-case result cleanup and CSV change-event persistence; bound RDF write concurrency and caches.
- Add cluster-wide Data Insights job locking with heartbeat, plus configurable concurrency budgets and tighter DB timeout defaults.
Reviewed changes
Copilot reviewed 38 out of 38 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-service/src/test/java/org/openmetadata/service/util/jdbi/JdbiUtilsTest.java | Adds unit coverage for applying configured JDBI statement timeout. |
| openmetadata-service/src/test/java/org/openmetadata/service/util/BoundedAsyncExecutorTest.java | Adds concurrency/cancellation/ordering tests for the bounded executor wrapper. |
| openmetadata-service/src/test/java/org/openmetadata/service/util/AsyncServiceTest.java | Adds tests for DB-bounded async lane behavior and cancellation semantics. |
| openmetadata-service/src/test/java/org/openmetadata/service/jdbi3/TestCaseRepositoryTest.java | Verifies test-case cleanup is batched (single delete) during bulk hard delete. |
| openmetadata-service/src/test/java/org/openmetadata/service/config/AsyncOperationsConfigurationTest.java | Validates default async-operation limits and stable config defaulting behavior. |
| openmetadata-service/src/test/java/org/openmetadata/service/apps/bundles/insights/workflows/dataAssets/DataAssetsWorkflowConcurrencyTest.java | Covers Data Insights concurrency budget calculation behavior. |
| openmetadata-service/src/main/java/org/openmetadata/service/util/OpenMetadataOperations.java | Fixes a JDBI handle leak in connection checking and makes validity explicit. |
| openmetadata-service/src/main/java/org/openmetadata/service/util/jdbi/JdbiUtils.java | Centralizes SqlStatements configuration and applies query timeout when using Hikari config. |
| openmetadata-service/src/main/java/org/openmetadata/service/util/BoundedAsyncExecutor.java | Introduces an ExecutorService wrapper to cap concurrent task execution. |
| openmetadata-service/src/main/java/org/openmetadata/service/util/AsyncService.java | Splits raw vs DB-bounded async execution, adds per-operation backlog stats + metrics. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java | Routes reindex work through cancellable DB-bounded async lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/glossary/GlossaryTermResource.java | Runs glossary move async work through DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/EntityResource.java | Moves async delete/restore and bulk-tag ops onto the DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/columns/ColumnResource.java | Routes CSV import and bulk column updates through the DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/apps/AppResource.java | Runs async app deletion/cleanup through the DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/ai/AuditPackGenerator.java | Routes audit pack generation through DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/rdf/translator/RdfPropertyMapper.java | Replaces unbounded concurrent maps with bounded Caffeine caches for RDF lookups. |
| openmetadata-service/src/main/java/org/openmetadata/service/rdf/RdfUpdater.java | Adds RDF write concurrency limits and routes RDF DB work through DB-bounded async lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplicationConfig.java | Adds asyncOperations configuration block to application config. |
| openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplication.java | Initializes AsyncService from config and passes async config into RDF init. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/UserRepository.java | Routes post-delete cleanup work through DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestSuiteRepository.java | Routes async test-suite deletion through DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestCaseResultRepository.java | Adds batch-delete support for test-case result cleanup and updates search delete script params. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestCaseRepository.java | Batches test-case result cleanup and routes cleanup work through DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestCaseDimensionResultRepository.java | Adds batch-delete API for test-case dimension results time series. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/HikariCPDataSourceFactory.java | Adds configurable query timeout seconds to DB factory config. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java | Adds chunked batch-delete DAO methods and tightens stale-lock deletion semantics. |
| openmetadata-service/src/main/java/org/openmetadata/service/governance/workflows/elements/nodes/userTask/CreateTask.java | Routes workflow termination dispatch through DB-bounded async lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/config/AsyncOperationsConfiguration.java | Introduces configurable limits for DB-heavy async work, RDF writes, and Data Insights DB budget. |
| openmetadata-service/src/main/java/org/openmetadata/service/audit/AuditLogRepository.java | Routes auth audit event writes through DB-bounded lane. |
| openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/insights/workflows/dataAssets/DataAssetsWorkflow.java | Computes Data Insights concurrency budget using CPU, pool-size fraction, and configured cap. |
| openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/insights/DataInsightsApp.java | Adds cross-server lease/heartbeat lock so only one server runs Data Insights at a time. |
| openmetadata-service/src/main/java/org/openmetadata/csv/EntityCsv.java | Batches/persists CSV change events asynchronously via DB-bounded lane. |
| docker/development/distributed-test/local/server1.yaml | Adds asyncOperations config and tightens DB timeout defaults for distributed tests. |
| docker/development/distributed-test/local/server2.yaml | Adds asyncOperations config and tightens DB timeout defaults for distributed tests. |
| docker/development/distributed-test/local/server3.yaml | Adds asyncOperations config and tightens DB timeout defaults for distributed tests. |
| conf/openmetadata.yaml | Adds asyncOperations config and tightens DB timeout defaults in default config. |
| conf/openmetadata-h2-test.yaml | Adds asyncOperations defaults for H2 test config and tightens DB timeout defaults. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.
Suppressed comments (2)
openmetadata-service/src/main/java/org/openmetadata/service/util/AsyncService.java:151
submitCancellableDatabaseTaskincrementsOperationStats.queuedinrecordSubmission(), but if the returnedFutureis cancelled before it starts running,FutureTask.run()becomes a no-op andstats.call(task)never executes (soqueuedis never decremented). This can permanently inflate queued metrics and generate false backlog warnings.
Make the cancellable task decrement queued on successful cancellation-before-start, and only increment/decrement active when the task actually begins executing.
public <T> Future<T> submitCancellableDatabaseTask(
DatabaseOperation operation, String context, Callable<T> task) {
final OperationStats stats = recordSubmission(operation, context);
final FutureTask<T> future = new FutureTask<>(() -> stats.call(task));
try {
databaseExecutorService.execute(future, () -> cancelDatabaseTask(stats, future));
} catch (RuntimeException e) {
stats.cancelSubmission();
throw e;
}
return future;
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TestCaseResultRepository.java:356
SearchRepository.deleteByScriptuses an Elasticsearch/OpenSearch script query (boolean predicate). The current script uses anif { ... }statement, which does not evaluate to a boolean value; other call sites pass a plain boolean expression (e.g.,ReportDataRepository). This risks delete-by-script failing or deleting nothing.
Use a boolean expression that safely handles missing fields instead of an if statement.
Map<String, Object> params = Map.of("fqns", testCaseFQNs);
searchRepository.deleteByScript(
TEST_CASE_RESULT,
"if (!(doc['testCaseFQN.keyword'].empty)) { params.fqns.contains(doc['testCaseFQN.keyword'].value) }",
params);
✅ Playwright Results — workflow succeededValidated commit ✅ 1467 passed · ❌ 0 failed · 🟡 2 flaky · ⏭️ 1 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 28m 4s ⏱️ Max setup 3m 7s · max shard execution 19m 15s · max shard-job elapsed before upload 22m 42s · reporting 9s 🌐 199.08 requests/attempt · 2.11 app boots/UI scenario · 29.26% common-shard skew Optimization targets still in progress:
🟡 2 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Code Review ✅ Approved 2 resolved / 2 findingsBounds asynchronous database work and adds request-pool protections to prevent pool exhaustion, addressing the rdfWritePermits static field volatility and DB-task metric leak findings. ✅ 2 resolved✅ Bug: rdfWritePermits static field is non-volatile and reassigned
✅ Bug: DB-task queued/active metrics can leak on shutdownNow
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.
Suppressed comments (4)
openmetadata-service/src/test/java/org/openmetadata/service/util/BoundedAsyncExecutorTest.java:114
- This test relies on an exact queued count via Semaphore#getQueueLength(), which is only an estimate. Waiting for
== 1can hang intermittently; prefer asserting the queue has at least one waiter.
openmetadata-service/src/test/java/org/openmetadata/service/util/BoundedAsyncExecutorTest.java:168 - Semaphore#getQueueLength() is an estimate, so waiting for the queued count to equal an exact value can be flaky. For ordering verification, it’s sufficient to wait until the queued count reaches at least the expected minimum before releasing the first task.
openmetadata-service/src/test/java/org/openmetadata/service/util/BoundedAsyncExecutorTest.java:54 - BoundedAsyncExecutor.getQueuedCount() is derived from Semaphore#getQueueLength(), which is documented as an estimate and may not be exact. Asserting an exact queued count here can make this test flaky under timing/VM differences; it’s enough to assert the queue is non-empty once concurrency is saturated.
This issue also appears in the following locations of the same file:
- line 112
- line 165
openmetadata-service/src/main/java/org/openmetadata/service/util/AsyncService.java:90
- AsyncService.getInstance() will eagerly initialize the singleton with default limits if it’s accessed before OpenMetadataApplication calls AsyncService.initialize(...). Any later initialize(config) call is then silently ignored, which can lead to running with unintended async DB limits (and misleadingly “configured” YAML). Consider logging when initialize is called after instantiation (or allowing re-init/update) so misconfiguration is detectable.
public static synchronized void initialize(AsyncOperationsConfiguration config) {
if (instance == null) {
instance = new AsyncService(Objects.requireNonNull(config));
instance.registerMetrics();
}
}
|
Describe your changes:
Fixes #31582
I bounded database-heavy asynchronous work, serialized Data Insights across servers, batched fan-out cleanup and CSV event writes, bounded RDF indexing, fixed a JDBI handle leak, and shortened database failure timeouts so background workloads cannot starve login and API requests.
Type of change:
High-level design:
Growing the Hikari pool was rejected because unbounded background fan-out would consume any larger pool and continue starving request traffic.
Tests:
Use cases covered
Unit tests
AsyncServiceTest,BoundedAsyncExecutorTest,AsyncOperationsConfigurationTest,DataAssetsWorkflowConcurrencyTest,JdbiUtilsTest, andTestCaseRepositoryTest.Backend integration tests
Ingestion integration tests
Playwright (UI) tests
Manual testing performed
Not performed; validation used focused Maven tests, Spotless, and
git diff --check.UI screen recording / screenshots:
Not applicable.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.