Skip to content

Fix SQLite "database is locked" errors under concurrent queue workers#128

Open
simonhamp wants to merge 1 commit into
NativePHP:mainfrom
simonhamp:fix/sqlite-database-locking
Open

Fix SQLite "database is locked" errors under concurrent queue workers#128
simonhamp wants to merge 1 commit into
NativePHP:mainfrom
simonhamp:fix/sqlite-database-locking

Conversation

@simonhamp

Copy link
Copy Markdown
Member

Problem

Apps that run background queue workers against the bundled SQLite database hit frequent SQLite... database is locked failures, even with only one or two workers and even though WAL is enabled.

Two issues in NativeServiceProvider::rewriteDatabase():

  1. The nativephp connection relies on one-shot PRAGMAs. journal_mode and busy_timeout are set with DB::statement(...) right after the config rewrite, but they aren't part of the connection config array. Laravel's SQLiteConnector only issues PRAGMAs it finds in the config, so on any reconnect — and for the lazily-created connections in each queue worker process — these settings are silently dropped.

  2. The connection falls back to DEFERRED transactions (the real culprit). The database queue pops a job in a read-then-write transaction (getNextAvailableJob()lockForUpdate() → mark reserved). Under DEFERRED, BEGIN starts as a read and upgrades to a write on the first write statement; if another worker has committed in between, SQLite returns SQLITE_BUSY_SNAPSHOT immediately and the busy handler does not retry it (retrying could deadlock). WAL and a large busy_timeout cannot help here.

transaction_mode has no PRAGMA equivalent — it controls how Laravel emits BEGIN — so it can only be fixed in the connection config.

Fix

Build the nativephp connection from the app's existing sqlite connection config (so any developer tuning carries over) and set the concurrency-critical keys explicitly:

  • transaction_modeIMMEDIATE — takes the write lock at BEGIN, so pop transactions serialise and wait on the busy timeout instead of erroring with SQLITE_BUSY_SNAPSHOT.
  • busy_timeout, journal_mode (WAL), synchronous (NORMAL) — now declared in config so the connector re-applies them on every connect in every process, not just the one alive at boot.

All are env-overridable and default to the same effective behaviour as before (WAL, 5s busy timeout), plus the IMMEDIATE and NORMAL improvements.

The trailing PRAGMA journal_mode=WAL / busy_timeout statements are now redundant but left in place to keep the diff minimal (WAL persists at the file level regardless).

Notes

  • Verified the current code on main still exhibits this — the connection array has no transaction_mode/busy_timeout/journal_mode/synchronous.
  • No behaviour change for single-process/non-queue usage; this only affects concurrent writers.

🤖 Generated with Claude Code

The runtime `nativephp` connection was built with no `transaction_mode`,
`busy_timeout`, `journal_mode` or `synchronous`, relying instead on a
one-shot `PRAGMA` after the config rewrite. Those PRAGMAs only affect the
connection alive at boot; they are lost on reconnect and never reach the
lazily-created connections in queue worker processes.

More importantly, the connection fell back to Laravel's default DEFERRED
transaction mode. The database queue pops jobs in a read-then-write
transaction (lockForUpdate then mark reserved), and under DEFERRED the
write-upgrade fails immediately with SQLITE_BUSY_SNAPSHOT when another
worker commits in between — an error the busy timeout does not retry.

Build the connection from the app's existing `sqlite` config and set the
concurrency-critical keys so Laravel's SQLiteConnector applies them on
every connect in every process. transaction_mode in particular has no
PRAGMA equivalent and can only be set here.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant