Skip to content

fix #9872: refresh repository lock of an idle mount#9894

Merged
ThomasWaldmann merged 6 commits into
masterfrom
mount-idle-lock-refresh-9872
Jul 11, 2026
Merged

fix #9872: refresh repository lock of an idle mount#9894
ThomasWaldmann merged 6 commits into
masterfrom
mount-idle-lock-refresh-9872

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Jul 10, 2026

Copy link
Copy Markdown
Member

Problem

Fixes #9872.

An idle FUSE mount makes no repository calls, so its lock's refresh (which currently only happens on read activity) never runs. After <stale> seconds (30 min by default) the lock expires as stale. A compact can then run and its pack rewrites invalidate the mount's cached pack locations, so subsequent reads fail with ObjectNotFound/PackNotFound (even for chunks that still exist, just moved to a new pack) and the next lock refresh aborts the mount with LockTimeout.

Fix

While mounted, run a background thread that periodically calls repository.info() to keep the lock alive (info() only touches the store every <stale>/2 seconds; the rest are cheap no-ops). This is the same idea borg with-lock already used.

The thread logic is factored into a small reusable LockRefresher class in storelocking.py:

LockRefresher(refresh_callable, sleep_interval=60, lock=None)

It runs a daemon thread that periodically calls refresh_callable (optionally under a caller-supplied lock), stops on LockTimeout (the lock was killed — nothing safe left to do), and on terminate() joins with a 5s timeout so a wedged refresh (e.g. a hung remote-repo request) can't stall shutdown / a FUSE unmount. borg with-lock now uses LockRefresher too, replacing its ad-hoc thread runner.

The refresh thread is started after the possible daemonizing fork() (threads do not survive fork()) and terminated in a finally around the FUSE main loop.

This is applied to both FUSE implementations, which are parallel and selected by BORG_FUSE_IMPL:

  • fuse.py (llfuse / pyfuse3)
  • hlfuse.py (mfusepy)

Thread safety

The FUSE code is otherwise single-threaded, and borgstore connections are not thread-safe. To keep the refresh thread from racing with the FUSE handlers on the store connection, each mount gets a self._repo_lock that serializes all repository access. It is a reentrant RLock, because a FUSE handler can re-enter it on the same thread: check_pending_archive() holds it and calls _process_archive()_process_leaf() (hardlink branch) → get_item(), which acquires it again. The LockRefresher is given this same lock.

Guarded repository-access points:

  • fuse.py: get_item()ItemCache.get(), check_pending_archive()_process_archive(), read()repository_uncached.get(), and the background refresh → repository.info().
  • hlfuse.py: check_pending_archive()_process_archive() (covers the get_many() in _iter_archive_items()), read()repository.get(), and the background refresh → repository.info().

fuse.py's module docstring is updated to document this single deliberate exception to the single-threaded invariant.

Tests

  • All existing mount_cmds_test.py FUSE tests pass on all backends (they exercise the now-serialized read / lookup / archive-loading paths, and thus the real LockRefresher start()/terminate() lifecycle via the mount daemon).
  • Added test_fuse_lock_refresh_calls_repository_info: drives LockRefresher through its real start()/terminate() API in a background thread (bounded by an Event wait + join, so it can't hang) and asserts repository.info() is invoked while the serialization lock is held. It uses a plain threading.Lock (whose .locked() exists on all supported Python versions — RLock.locked() is 3.14+ only).

🤖 Generated with Claude Code

@ThomasWaldmann ThomasWaldmann force-pushed the mount-idle-lock-refresh-9872 branch from dbd4200 to 70ef3dc Compare July 10, 2026 20:13
ThomasWaldmann and others added 5 commits July 11, 2026 00:19
An idle FUSE mount makes no repository calls, so its lock was never refreshed
and expired as stale after <stale> seconds (30 min by default). A compact could
then run and rewrite packs, invalidating the mount's cached pack locations, so
subsequent reads failed with ObjectNotFound/PackNotFound and the next lock
refresh aborted the mount with LockTimeout.

Start a background thread (ThreadRunner, same approach as `borg with-lock`) that
periodically calls repository.info() to keep the lock alive. The FUSE code is
otherwise single-threaded and borgstore connections are not thread-safe, so all
repository access - from the FUSE handlers and the refresh thread - is serialized
via a new self._repo_lock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mfusepy backend (hlfuse.py) is a parallel implementation with the same idle
mount lock issue, so apply the same background lock-refresh there: a ThreadRunner
calling repository.info(), with all repository access serialized via a new
self._repo_lock (read(), _process_archive()).

Also fix the regression test to import the FuseBackend of the active FUSE
implementation - under BORG_FUSE_IMPL=mfusepy, fuse.py is not importable (llfuse
is None) and hlfuse.py is used instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
threading.RLock.locked() only exists on Python 3.14+, so on 3.11/3.12 the
test raised AttributeError inside FakeRepository.info(). LockRefresher._run()
caught it as a transient error and retried forever, and since the test drove
_run() synchronously the whole test process hung (py311-llfuse / py312-pyfuse3
CI jobs got stuck; py314-mfusepy passed because 3.14 has RLock.locked()).

Rewrite the test to use the real LockRefresher start()/terminate() API in a
background thread (bounded by an Event wait + join, so it can never hang) and a
plain threading.Lock, whose .locked() is available on all supported versions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ThomasWaldmann ThomasWaldmann force-pushed the mount-idle-lock-refresh-9872 branch from 5019a3d to 1040e60 Compare July 10, 2026 22:21
…t stall shutdown

If refresh_callable() (repository.info -> lock refresh) blocks indefinitely, e.g.
on a hung remote-repo request, an unbounded join() in terminate() would stall
whatever waits for it - notably a FUSE unmount. Make the refresh thread a daemon
and join with a 5s timeout: shutdown proceeds even if the thread is still wedged,
and the daemon thread is torn down at interpreter exit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ThomasWaldmann ThomasWaldmann merged commit 2463960 into master Jul 11, 2026
19 checks passed
@ThomasWaldmann ThomasWaldmann deleted the mount-idle-lock-refresh-9872 branch July 11, 2026 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

borg2: idle mount does not refresh lock

1 participant