Skip to content

fix: guard shared instance maps with a RWMutex#127

Open
FlavioPulli wants to merge 1 commit into
evolution-foundation:mainfrom
FlavioPulli:fix/instance-map-race
Open

fix: guard shared instance maps with a RWMutex#127
FlavioPulli wants to merge 1 commit into
evolution-foundation:mainfrom
FlavioPulli:fix/instance-map-race

Conversation

@FlavioPulli

Copy link
Copy Markdown

killChannel, clientPointer and myClientPointer are created in main.go and shared by reference across all service packages and every MyClient. ReconnectClient deletes from all three without a lock, StartClient writes to them, the Disconnected handler triggers ReconnectClient from a goroutine, and every HTTP handler reads them concurrently. Go fatals on concurrent map access — we hit this in production as fatal error: concurrent map writes / internal/runtime/maps.fatal during reconnection bursts (stack through whatsmeowService.StartClient/ReconnectClient), which kills the entire process and every connected instance with it.

This adds a single exported sync.RWMutex in the whatsmeow service package guarding the three maps everywhere they are touched. Lock scopes are kept minimal: values are copied out under RLock and the lock is released before any long call (Disconnect, Connect, StartInstance, channel sends), so no lock is ever held across network operations. As a side effect the killChannel check-and-delete paths became atomic, closing a latent double-close race.

go build ./... and go vet ./... clean.

Three maps are created once in cmd/evolution-go/main.go — killChannel
(map[string]chan bool), clientPointer (map[string]*whatsmeow.Client) and
myClientPointer (map[string]*MyClient) — and shared by reference across
every service package (whatsmeow, instance, sendMessage, chat, call,
community, group, label, message, newsletter, user) and every *MyClient
value. None of the reads, writes or deletes on them were synchronized.

In production this fatals the process: ReconnectClient (whatsmeow.go)
deletes from all three maps without a lock, StartClient writes to
clientPointer/myClientPointer while its select loop and other goroutines
(schedulePresenceUpdates, teardownQR, myEventHandler's LoggedOut/poll-vote
paths) read killChannel/clientPointer, and every HTTP handler in the
peripheral packages reads clientPointer via ensureClientConnected — all
concurrently, since events.Disconnected fires ReconnectClient in its own
goroutine while requests keep flowing in. Go's runtime treats concurrent
map writes, or a read racing a write, as fatal — unrecoverable, not a
panic — and that is exactly what showed up in production as
internal/runtime/maps.fatal during bursts of reconnects.

The fix adds one exported sync.RWMutex (ClientMapsMu, in the new
pkg/whatsmeow/service/client_maps.go) shared by every package that touches
these maps. Every indexed read takes RLock/RUnlock, every write (assign or
delete) takes Lock/Unlock, and every guarded scope is kept as short as
possible: the client/channel value is copied into a local variable while
holding the lock, the lock is released, and only then is the value used
for anything that can block or take a while (IsConnected/Disconnect,
channel sends inside select, StartInstance/ReconnectClient/StartClient,
network calls). This also closes a couple of adjacent bugs for free:
StartClient's kill-signal select and schedulePresenceUpdates now
re-resolve the channel from the map on each loop iteration instead of
evaluating a stale expression, and ClearInstanceCache/Delete's
check-then-delete on killChannel is now atomic, so two concurrent cleanups
can no longer double-close the same channel.

No lock is ever held across a call into another guarded function
(ReconnectClient -> StartInstance -> StartClient all close their lock
scopes before calling the next), so there is no new deadlock risk.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @FlavioPulli, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant