fix #9872: refresh repository lock of an idle mount#9894
Merged
Conversation
dbd4200 to
70ef3dc
Compare
An idle FUSE mount makes no repository calls, so its lock was never refreshed and expired as stale after <stale> seconds (30 min by default). A compact could then run and rewrite packs, invalidating the mount's cached pack locations, so subsequent reads failed with ObjectNotFound/PackNotFound and the next lock refresh aborted the mount with LockTimeout. Start a background thread (ThreadRunner, same approach as `borg with-lock`) that periodically calls repository.info() to keep the lock alive. The FUSE code is otherwise single-threaded and borgstore connections are not thread-safe, so all repository access - from the FUSE handlers and the refresh thread - is serialized via a new self._repo_lock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mfusepy backend (hlfuse.py) is a parallel implementation with the same idle mount lock issue, so apply the same background lock-refresh there: a ThreadRunner calling repository.info(), with all repository access serialized via a new self._repo_lock (read(), _process_archive()). Also fix the regression test to import the FuseBackend of the active FUSE implementation - under BORG_FUSE_IMPL=mfusepy, fuse.py is not importable (llfuse is None) and hlfuse.py is used instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
threading.RLock.locked() only exists on Python 3.14+, so on 3.11/3.12 the test raised AttributeError inside FakeRepository.info(). LockRefresher._run() caught it as a transient error and retried forever, and since the test drove _run() synchronously the whole test process hung (py311-llfuse / py312-pyfuse3 CI jobs got stuck; py314-mfusepy passed because 3.14 has RLock.locked()). Rewrite the test to use the real LockRefresher start()/terminate() API in a background thread (bounded by an Event wait + join, so it can never hang) and a plain threading.Lock, whose .locked() is available on all supported versions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5019a3d to
1040e60
Compare
…t stall shutdown If refresh_callable() (repository.info -> lock refresh) blocks indefinitely, e.g. on a hung remote-repo request, an unbounded join() in terminate() would stall whatever waits for it - notably a FUSE unmount. Make the refresh thread a daemon and join with a 5s timeout: shutdown proceeds even if the thread is still wedged, and the daemon thread is torn down at interpreter exit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #9872.
An idle FUSE mount makes no repository calls, so its lock's refresh (which currently only happens on read activity) never runs. After
<stale>seconds (30 min by default) the lock expires as stale. Acompactcan then run and its pack rewrites invalidate the mount's cached pack locations, so subsequent reads fail withObjectNotFound/PackNotFound(even for chunks that still exist, just moved to a new pack) and the next lock refresh aborts the mount withLockTimeout.Fix
While mounted, run a background thread that periodically calls
repository.info()to keep the lock alive (info()only touches the store every<stale>/2seconds; the rest are cheap no-ops). This is the same ideaborg with-lockalready used.The thread logic is factored into a small reusable
LockRefresherclass instorelocking.py:It runs a daemon thread that periodically calls
refresh_callable(optionally under a caller-suppliedlock), stops onLockTimeout(the lock was killed — nothing safe left to do), and onterminate()joins with a 5s timeout so a wedged refresh (e.g. a hung remote-repo request) can't stall shutdown / a FUSE unmount.borg with-locknow usesLockRefreshertoo, replacing its ad-hoc thread runner.The refresh thread is started after the possible daemonizing
fork()(threads do not survivefork()) and terminated in afinallyaround the FUSE main loop.This is applied to both FUSE implementations, which are parallel and selected by
BORG_FUSE_IMPL:fuse.py(llfuse / pyfuse3)hlfuse.py(mfusepy)Thread safety
The FUSE code is otherwise single-threaded, and borgstore connections are not thread-safe. To keep the refresh thread from racing with the FUSE handlers on the store connection, each mount gets a
self._repo_lockthat serializes all repository access. It is a reentrantRLock, because a FUSE handler can re-enter it on the same thread:check_pending_archive()holds it and calls_process_archive()→_process_leaf()(hardlink branch) →get_item(), which acquires it again. TheLockRefresheris given this same lock.Guarded repository-access points:
fuse.py:get_item()→ItemCache.get(),check_pending_archive()→_process_archive(),read()→repository_uncached.get(), and the background refresh →repository.info().hlfuse.py:check_pending_archive()→_process_archive()(covers theget_many()in_iter_archive_items()),read()→repository.get(), and the background refresh →repository.info().fuse.py's module docstring is updated to document this single deliberate exception to the single-threaded invariant.Tests
mount_cmds_test.pyFUSE tests pass on all backends (they exercise the now-serialized read / lookup / archive-loading paths, and thus the realLockRefresherstart()/terminate()lifecycle via the mount daemon).test_fuse_lock_refresh_calls_repository_info: drivesLockRefresherthrough its realstart()/terminate()API in a background thread (bounded by anEventwait + join, so it can't hang) and assertsrepository.info()is invoked while the serialization lock is held. It uses a plainthreading.Lock(whose.locked()exists on all supported Python versions —RLock.locked()is 3.14+ only).🤖 Generated with Claude Code