fix: scheduler can no longer yield if protobuf mutex is held

[chris-olszewski]

What was changed

Scheduler#run_until_all_yielded will no longer return if the Google protobuf mutex is held.

This is done by inspecting block calls to see if the blocker is a mutex with google/protobuf in the path. This is more fragile than desired (as shown by how we test this behavior), but is already how we allowlist this mutex usage in IllegalWorkflowCallValidator so we are at least consistent in what we allow.

These block calls that come from the protobuf mutex will be tracked additionally and any Fiber that is blocked because of this will prevent us from considering that all ready fibers have been resumed.

Why?

We special case allow mutex usage from protobuf in the workflow as it is "safe" usage (as in it cannot cause non-determinism, it guards a cache to avoid unnecessary allocations). This is still not fully safe as if the workflow future is blocked waiting for this call, Scheduler#run_until_all_yielded can return even if there is additional work to do for this activation once the protobuf mutex is acquired. This resulted in "partial" workflow activations that could not be replayed consistently.

The issue is that the mutex used by protobuf is process wide so a worker that executes both activities and workflows will have the workflow threads fight with the activity threads for the mutex. This along with general CPU pressure could increase the chance of this mutex having contention.

We cannot warm this cache as it is backed by WeakMap.

Checklist

1. Closes [Bug] Sync primitives can lead to un-replayable workflow history. #464

2. How was this tested:

Added some fairly forced regression tests. Added some unit tests for our scheduler directly since those are the most worrisome changes.

3. Any docs updates needed?
   N/A

[GregoryTravis]

Are there any conditions where this could return something else? Just making sure I understand.

[chris-olszewski]

Since we're holding onto our fake protobuf mutex, we want to verify that this never emits any "commands". If this was ever non-empty that would mean we emitted commands even under the scenario we're trying to prevent.