Skip to content

Cache reference timestamps behind an IVsAsyncFileChangeEx2 watcher for snapshot reuse - #20457

Open
xperiandri wants to merge 20 commits into
dotnet:mainfrom
xperiandri:vs/file-change-watcher
Open

Cache reference timestamps behind an IVsAsyncFileChangeEx2 watcher for snapshot reuse#20457
xperiandri wants to merge 20 commits into
dotnet:mainfrom
xperiandri:vs/file-change-watcher

Conversation

@xperiandri

@xperiandri xperiandri commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Description

The transparent compiler's snapshot-reuse guard (WorkspaceExtensions.createProjectSnapshot) decided whether an old snapshot could be reused by stat'ing every -r: reference of the project (getOnDiskReferences) — on every new Roslyn Project instance, i.e. on every workspace fork, and before the cheap same-version fast path. For a project with a few hundred references that is a few hundred GetLastWriteTimeUtc calls per keystroke-driven typecheck.

This PR replaces that stat with a cached stamp that an IVsAsyncFileChangeEx2 watcher keeps fresh.

  • FileChangeWatcher.fs — the watcher, the F# counterpart of Roslyn's FileChangeWatcher / ReferenceFileChangeTracker (internal to Microsoft.VisualStudio.LanguageServices, not reachable through ExternalAccess.FSharp): advise/unadvise batched on a single-consumer queue with a 500 ms window, coalesced per kind and per sink; recursive .dll directory watches over the reference roots (DOTNET_ROOT/packs, machine dotnet/packs, .NET Framework reference assemblies, the NuGet cache honouring NUGET_PACKAGES); ref-counted per-file watches for everything else; free-threaded sink; batch application as a cancellableTask under a token the watcher owns.
  • FSharpReferenceChangeTracker keeps each watched path's last-write stamp inside its watch entry and drops it on the raw change notification. IReferenceStamps.GetLastWriteTimeUtc serves the cached value only while the path is watched — a stamp is only ever served while a notification can still reach it — and stats unwatched paths directly. The 2 s debounce stays for consumers that do expensive work per change.
  • FSharpProjectOptionsReactor watches the -r: set of each project it computes options for, diffed on recompute so an unchanged set touches nothing, and exposes the stamps as FSharpProjectOptionsManager.ReferenceStamps. It does not invalidate anything on a change — see below.
  • The ReferencesOnDisk guard in createProjectSnapshot reads the stamps. On a mismatch with the snapshot's own stamps (FCS stats when it builds one) it drops the project's stamps, so a missed notification costs one re-stat pass rather than a rebuild on every future Project instance.

Why the reactor does not call InvalidateConfiguration

An earlier revision of this PR did. Checked in VS with breakpoints on both paths and a referenced project rebuilt from the command line: for a -r: the workspace holds as a MetadataReference, Roslyn already advises the file (ProjectSystemProjectFactory), swaps the reference when it changes and bumps Project.Version; the reactor sees that through isProjectInvalidated, recomputes and calls InvalidateConfiguration itself — and it gets there first (Roslyn batches over 500 ms; this tracker adds a 2 s debounce on top of the same window). So the watch here exists for the stamps, not for invalidation. c15f47bd52 in the history is that removal; the subscription that returns in 744355228a serves a different purpose.

Not in this PR

  1. FSharpProjectSnapshot.FromOptions stats every -r: when a snapshot is built from scratch (FCS side); Let FSharpProjectSnapshot.FromOptions take reference stamps from the host #20459 adds the getReferenceStamp parameter that lets it read the same cache, and the one-line VS consumer follows once it merges.
  2. IsReferencesInvalidated on the incremental builder (legacy path) stats every reference per request; that needs the reference analogue of useChangeNotifications on the FCS side.
  3. Scripts: #load sources are invisible to the workspace; watching them drops the script's cached options.

docs/ide/file-watching.md records what the workspace already covers, what it does not, and the shape of the watcher.

Checklist

  • Test cases added — FileChangeWatcherTests.fs: WatchedDirectory coverage; tracker ref-counting, debounce and dispose against an in-memory watcher; applyBatch against a recording IVsAsyncFileChangeEx2; and the stamp cache — served while watched until a notification, stat'd when unwatched, dropped by Invalidate, stat'd again once the last watch stops.
  • Performance benchmarks added in case of performance changes — measured in VS instead (transparent compiler + snapshot reuse on, breakpoint in referencesOnDiskChanged): on repeat passes the reuse guard for projects with 187 and 349 -r: read every stamp from the tracker — 0 GetLastWriteTimeUtc, the tracker's stamp count was identical before and after getOnDiskReferences. Touching a referenced dll dropped its stamp; the next guard re-stat'd that one file and rebuilt the snapshot (References on disk changed in the trace), after which passes were cached again.
  • Release notes entry updated — docs/release-notes/.VisualStudio/18.vNext.md; docs/release-notes/.FSharp.Compiler.Service/11.0.100.md for the illib helper.

xperiandri and others added 4 commits September 5, 2026 20:51
The FileChangeWatcher worktree is pull-model I/O (OpenFileForReadShimAsync
plus last-write timestamps on FSharpFileSnapshot), not a push watcher.
Roslyn's FileChangeWatcher is the reference: IVsAsyncFileChangeEx2,
directory subscriptions, 500 ms AsyncBatchingWorkQueue, free-threaded
sinks, coalesced metadata-reference invalidation.

This repo already has two IVsFileChangeEx clients (legacy FileChangeManager
and deprecated FSharpSource.SetDependencyFiles). The intended FSharp.Editor
replacement lives only in stash@{7}
(54465595717b8bb746cb2633d5a4aa834888a481): FileChangeWatcher.fs plus
FileChangeWatcherHub, wired to FSharpProjectOptionsReactor for -r:
assemblies. It is IVsFileChangeEx + JTF.Run, not IVsAsyncFileChangeEx2.
No commit, branch, or GitHub hit implements IVsAsyncFileChangeEx2.

Recommended split: ship the async read shim on its own; restore the stash
watcher or jump straight to IVsAsyncFileChangeEx2 with directory batching;
invalidate FCS via NotifyFileChanged instead of O(N) timestamp polling.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roslyn-shaped push file watching for FSharp.Editor:
- FSharpFileChangeWatcher: batched advise/unadvise (500ms window,
  coalesced same-kind ops), service obtained via Task without
  blocking on UI thread
- FileChangeContext: free-threaded sink (IVsFreeThreadedFileChangeEvents2),
  directory subscriptions with extension filters, per-file watches
  covered by watched directories become no-op tokens
- FSharpReferenceChangeTracker: ref-counted reference watching with
  2s debounce; default directory watches for DOTNET_ROOT\packs,
  dotnet\packs, Reference Assemblies, NuGet cache (.dll filter)

Modeled on Roslyn FileChangeWatcher/ReferenceFileChangeTracker
(all internal there, not reusable from F#).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Subscribe each project's on-disk '-r:' reference assemblies via
FSharpReferenceChangeTracker when options are computed; on a watched
dll change, drop that project's cached options and invalidate the
checker configuration. Subscriptions are ref-counted, cleared on
project removal and reactor disposal.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Cover WatchedDirectory path matching, tracker ref-counting,
debounce of burst notifications, and dispose. Tests use an
in-memory IFSharpFileChangeWatcher mock so they do not need
a live IVsAsyncFileChangeEx2 service.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

❗ Release notes required

You can open this PR in browser to add release notes: open in github.dev


✅ Found changes and release notes in following paths:

Change path Release notes path Description
`src/Compiler` docs/release-notes/.FSharp.Compiler.Service/11.0.100.md
`vsintegration/src` docs/release-notes/.VisualStudio/18.vNext.md

for KeyValue(projectId, paths) in referenceWatches do
if
paths
|> Array.exists (fun p -> String.Equals(p, path, StringComparison.OrdinalIgnoreCase))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a helper extension method that can be used here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no equality helper in illib (only StartsWithOrdinal/EndsWithOrdinal/EndsWithOrdinalIgnoreCase/IndexOfOrdinal), so instead of comparing I changed the data: per-project watches are now a HashSet<string>(StringComparer.OrdinalIgnoreCase) and the notification does paths.Contains path. Done in 57f2bb0.

member _.Dispose() = watched.Clear()

type private MockFileChangeWatcher() =
let mutable context: MockFileChangeContext option = None

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not voption in new tests code?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to voption in 57f2bb0.


/// Manages mappings of Roslyn workspace Projects/Documents to FCS.
type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace) =
type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace, ?fileChangeWatcher: IFSharpFileChangeWatcher) =

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can it ever be none? I would make it mandatory

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made mandatory in 57f2bb0; LanguageService.fs is the only caller and always passes one.


[<Sealed>]
type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
type private FSharpProjectOptionsReactor(checker: FSharpChecker, fileChangeWatcher: IFSharpFileChangeWatcher option) =

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think fileChangeWatcher must be mandatory.
If we need to pass it as null in tests, let's do

let fileChangeWatcher =
	withNull fileChangeWatcher |> ValueOption.ofObj

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 57f2bb0IFSharpFileChangeWatcher is a plain required parameter now on both the reactor and the manager, and the Option.map/Option.iter plumbing around the tracker is gone. No null path is needed: the tests exercise FSharpReferenceChangeTracker with an in-memory mock watcher and never construct the manager.

if path.EndsWith(string IO.Path.DirectorySeparatorChar) then
path
else
path + string IO.Path.DirectorySeparatorChar

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use string intepolation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 57f2bb0.


[<Sealed>]
type internal FSharpWatchedFileToken() =
member val Cookie: uint32 option = None with get, set

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not?

Suggested change
member val Cookie: uint32 option = None with get, set
member val Cookie: uint32 voption = ValueNone with get, set

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 57f2bb0, plus the two use sites (ValueSome cookie on advise, a match on unadvise).

Comment on lines +324 to +340
[
if not (String.IsNullOrEmpty dotnetRoot) then
IO.Path.Combine(dotnetRoot, "packs")

IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs")

IO.Path.Combine(
Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86,
"Reference Assemblies",
"Microsoft",
"Framework"
)

IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages")
]
|> List.distinct
|> List.map (fun d -> WatchedDirectory(d, [ ".dll" ]))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
[
if not (String.IsNullOrEmpty dotnetRoot) then
IO.Path.Combine(dotnetRoot, "packs")
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs")
IO.Path.Combine(
Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86,
"Reference Assemblies",
"Microsoft",
"Framework"
)
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages")
]
|> List.distinct
|> List.map (fun d -> WatchedDirectory(d, [ ".dll" ]))
seq {
if not (String.IsNullOrEmpty dotnetRoot) then
IO.Path.Combine(dotnetRoot, "packs")
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs")
IO.Path.Combine(
Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86,
"Reference Assemblies",
"Microsoft",
"Framework"
)
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages")
}
|> Seq.distinct
|> Seq.map (fun d -> WatchedDirectory(d, [ ".dll" ]))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 57f2bb0 with one addition: |> List.ofSeq at the end. FileChangeContext iterates the watched directories on every EnqueueWatchingFile, so a lazy seq would re-run Environment.GetFolderPath and Seq.distinct per watched file; materializing once here keeps the single Seq chain but evaluates it one time.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

🔍 Tooling Safety Check — Affects-Design-Time
Affects-Design-Time: adds FileChangeWatcher + FSharpReferenceChangeTracker to VS integration that execute at design time

Generated by PR Tooling Safety Check · opus46 4.5M ·

@github-actions github-actions Bot added the ⚠️ Affects-Design-Time Tooling check: PR touches type providers or dependency manager label Sep 5, 2026
* FSharpProjectOptionsReactor/Manager take the IFSharpFileChangeWatcher
  outright; the only caller always has one, so the option wrappers and
  the Option.iter/map plumbing around the tracker go away.
* Reference watches per project are an OrdinalIgnoreCase HashSet, so a
  change notification is a Contains instead of an Array.exists with an
  explicit comparison.
* FSharpWatchedFileToken.Cookie and the test mock's context are voption.
* applyBatch indexes the drained ResizeArray directly instead of
  converting it to a list and re-slicing it with takeWhile/skip/collect.
* StartsWithOrdinal / EndsWithOrdinal / EndsWithOrdinalIgnoreCase from
  Internal.Utilities.Library at the ordinal call sites, interpolation
  for the trailing separator, and the default watched directories go
  through one Seq chain materialized once.
}
|> Seq.distinct
|> Seq.map (fun d -> WatchedDirectory(d, [ ".dll" ]))
|> List.ofSeq

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
|> List.ofSeq
|> Seq.toList

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 4b20482.

Comment thread vsintegration/src/FSharp.Editor/LanguageService/FileChangeWatcher.fs Outdated
The list-typed applyBatch reads better than the index walk; only the
voption use sites differ from the original body.
pending <- rest

if cookies.Count > 0 then
let! _ = service.UnadviseDirChangesAsync(cookies.ToArray(), CancellationToken.None)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why no cancellation? why not cancellableTask CE for enclosing function?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 439e7ce: applyBatch is a cancellableTask, takes the token with CancellableTask.getCancellationToken () and passes it to every IVsAsyncFileChangeEx2 call.

| None -> draining <- false

let! service = fileChangeService |> Async.AwaitTask
do! applyBatch service (List.ofSeq ops) |> Async.AwaitTask

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and then propagate cancellation to CancellableTask here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 439e7ce. The agent is started with a token from a CancellationTokenSource the watcher owns (FSharpFileChangeWatcher is IDisposable now: cancel, dispose, dispose the mailbox), the loop reads it via Async.CancellationToken and runs the batch with CancellableTask.startAsTask ct. The catch-all that kept the loop alive on failed advises became with ex when not (ex :? OperationCanceledException), so cancellation is no longer swallowed. LanguageService.fs constructs the watcher with new accordingly; nothing disposes it there on purpose — it lives as long as the workspace service, same as the checker.

* IFSharpFileChangeWatcher.CreateContext and WatchedDirectory take
  ImmutableArray, the same contract as Roslyn's FileChangeWatcher; the
  set is built once and scanned on every EnqueueWatchingFile.
* applyBatch is a cancellableTask and passes its token to every
  IVsAsyncFileChangeEx2 call. The agent runs under a token owned by the
  watcher, which is now IDisposable; cancellation is no longer swallowed
  by the loop's catch-all.
* Batches are sliced and collected as arrays, so the cookie and path
  arrays go to the service without a List.toArray copy.
The ignore-case StartsWith was the one ordinal comparison in
FileChangeWatcher.fs without an illib helper; the sibling of the
existing EndsWithOrdinalIgnoreCase closes that gap and the watched-
directory check uses it.

type internal IFSharpFileChangeWatcher =
abstract CreateContext: watchedDirectories: ImmutableArray<WatchedDirectory> -> IFSharpFileChangeContext

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@T-Gro what do we prefer? Interface or function type?

Suggested change
type internal CreateIFSharpFileChangeContext = ImmutableArray<WatchedDirectory> -> IFSharpFileChangeContext

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, since you started with interface, unless there is a big win with function (which I don't see), I'd leave it as is.

In general, for a single method, and no design foresight on what it could evolve into later on, function is fine.

The bonus of interface, even for one method is the named arguments.

@xperiandri xperiandri left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh-eyes pass over the whole PR against the current Roslyn FileChangeWatcher / ReferenceFileChangeTracker and against what FCS actually does with references. The shape is right: batched advise/unadvise with the 500 ms window, directory watches for the reference roots, ref-counted per-file watches, free-threaded sink. Three things need fixing, and one framing point matters for the description.

Must fix (inline, with suggestions):

  1. WatchFiles runs are coalesced across sinks and advised with the first one. Latent today, breaks as soon as a second context exists.
  2. FSharpReferenceChangeTracker races Timer.Change against Timer.Dispose and can throw inside the VS file change callback; it also allocates a timer per unwatched path before checking whether anyone watches it.
  3. cache.TryRemove in onWatchedReferenceChanged never reaches consumers (they go through ProjectCache.Projects) and causes a second InvalidateConfiguration at the next recompute, discarding a builder that may already be rebuilt. InvalidateConfiguration alone is the right call.

Framing. As it stands the PR does not change observable latency. On the legacy path getOrCreateBuilder (BackgroundCompiler.fs L492) evaluates IsReferencesInvalidated on every request, and that stats every reference with a fresh TimeStampCache (IncrementalBuild.fs L1245). On the transparent-compiler path the snapshot reuse compares ReferencesOnDisk by stat (WorkspaceExtensions.fs L250) and FromOptions stats on creation. Both notice a rebuilt dll at the next request anyway, and InvalidateConfiguration only swaps in a lazy builder node without computing anything. So "keeps serving options and snapshots computed against the old assembly" is not what happens today. The value of the watcher arrives when it replaces those stat loops: a reference-change notification for the incremental builder on the FCS side (the analogue of useChangeNotifications for sources) and a watcher-invalidated stamp cache for snapshots. Worth presenting this PR as that infrastructure and listing both follow-ups explicitly.

Also inline: diff-based watch updates instead of stop-all/start-all on every options recompute, NUGET_PACKAGES and whether the NuGet cache should be a directory watch at all (Roslyn does not do that), a shared no-op token, the drain loop, logging of swallowed exceptions, the design-review doc, and applyBatch test coverage. The Fixes # (issue, if applicable) placeholder is still in the description.

All suggestions were applied together on the branch and pass dotnet fantomas --check; the changed regions were type-checked and smoke-tested with stand-ins for the VS interop types (sink grouping, drain, debounce and the timer race under 8 concurrent producers, watch diffing).

Comment on lines +117 to +123
| WatchFiles(_, _, sink) :: _ ->
let batch =
pending
|> Seq.takeWhile (function
| WatchFiles _ -> true
| _ -> false)
|> Seq.toArray

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (latent): WatchFiles runs are coalesced across sinks. The whole run of consecutive WatchFiles operations is advised with the sink of the first one. With a single context this cannot bite, but CreateContext allows several, and the moment a second one exists (the script #r/#load watcher from the follow-ups) its files get subscribed with the wrong sink and its notifications are silently lost. Roslyn's CanCombineWith only merges operations with the same sink.

Suggested change
| WatchFiles(_, _, sink) :: _ ->
let batch =
pending
|> Seq.takeWhile (function
| WatchFiles _ -> true
| _ -> false)
|> Seq.toArray
| WatchFiles(_, _, sink) :: _ ->
let batch =
pending
|> Seq.takeWhile (function
| WatchFiles(_, _, s) -> obj.ReferenceEquals(s, sink)
| _ -> false)
|> Seq.toArray

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in d8dce7a: the run stops at the first WatchFiles whose sink is not ReferenceEquals to the head one. Covered by A run of file watches is split when the sink changes.

Comment on lines +378 to +395
ctx.FileChanged.Add(fun path ->
let fire (_: obj) =
pendingTimers.TryRemove path
|> function
| true, timer -> timer.Dispose()
| _ -> ()

// Only notify for paths someone is actually watching; directory watches
// cover whole trees.
let isWatched = lock gate (fun () -> watchedFiles.ContainsKey path)

if isWatched then
onChanged path

let timer =
pendingTimers.GetOrAdd(path, fun _ -> new Timer(fire, null, Timeout.Infinite, Timeout.Infinite))

timer.Change(notificationDelay, Timeout.InfiniteTimeSpan) |> ignore)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Timer.Change races Timer.Dispose. The handler takes the timer out of pendingTimers without the lock; fire (and the tracker's Dispose) remove and dispose it. If an event lands between TryRemove and Change, Change throws ObjectDisposedException straight out of the free-threaded FilesChanged/DirectoryChangedEx2 callback, i.e. inside the VS file change service. Narrow window, but a rebuild is exactly the burst that hits it.

Second problem in the same block: the watchedFiles.ContainsKey check only runs in fire, so every .dll event under the watched directories allocates a Timer first. With ~/.nuget/packages watched recursively, any restore on the machine creates thousands of timers for files nobody watches.

Doing the bookkeeping under gate and checking first fixes both. With everything under the lock pendingTimers can become a plain Dictionary; left out here so the suggestion applies on its own. (Roslyn's current ReferenceFileChangeTracker has no per-path timers at all: one 5 s batching queue with string dedupe.)

Suggested change
ctx.FileChanged.Add(fun path ->
let fire (_: obj) =
pendingTimers.TryRemove path
|> function
| true, timer -> timer.Dispose()
| _ -> ()
// Only notify for paths someone is actually watching; directory watches
// cover whole trees.
let isWatched = lock gate (fun () -> watchedFiles.ContainsKey path)
if isWatched then
onChanged path
let timer =
pendingTimers.GetOrAdd(path, fun _ -> new Timer(fire, null, Timeout.Infinite, Timeout.Infinite))
timer.Change(notificationDelay, Timeout.InfiniteTimeSpan) |> ignore)
ctx.FileChanged.Add(fun path ->
let fire (_: obj) =
let isWatched =
lock gate (fun () ->
match pendingTimers.TryRemove path with
| true, timer -> timer.Dispose()
| _ -> ()
watchedFiles.ContainsKey path)
if isWatched then
onChanged path
lock gate (fun () ->
// Directory watches cover whole trees; only debounce paths someone watches.
if not disposed && watchedFiles.ContainsKey path then
let timer =
pendingTimers.GetOrAdd(path, fun _ -> new Timer(fire, null, Timeout.Infinite, Timeout.Infinite))
timer.Change(notificationDelay, Timeout.InfiniteTimeSpan) |> ignore))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in d8dce7a, including the step you left out: with all bookkeeping under gate, pendingTimers is a plain Dictionary now. The handler checks watchedFiles before it allocates or arms a timer, and fire removes and disposes under the same lock, so Change can no longer meet a disposed timer inside the VS callback.

Comment on lines +128 to +138
// Push invalidation for on-disk '-r:' reference assemblies (not tracked by the Roslyn
// workspace): when one changes after an external rebuild, drop the cached options of every
// project referencing it instead of waiting for a timestamp poll to notice.
let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>()

let onWatchedReferenceChanged (path: string) =
for KeyValue(projectId, paths) in referenceWatches do
if paths.Contains path then
match cache.TryRemove projectId with
| true, (_, _, projectOptions) -> checker.InvalidateConfiguration(projectOptions, userOpName = "onWatchedReferenceChanged")
| _ -> ()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cache.TryRemove here does more harm than good. Consumers resolve options through ProjectCache.Projects (WorkspaceExtensions.fs L177), a ConditionalWeakTable keyed by the Roslyn Project instance, so removing the reactor entry does not reach them. When the reactor is next consulted (any new Project instance) the miss forces a recompute, whose success path calls InvalidateConfiguration again (L464) and throws away a builder that may already have been rebuilt against the new dll. Meanwhile GetCompilationDefinesAndLangVersionForEditingDocument and TryGetQuickParsingOptionsForEditingDocumentOrProject fall back to default defines until that recompute.

The options themselves are still correct (same paths); only the FCS build behind them is stale, and InvalidateConfiguration on the cached options is all that is needed.

Suggested change
// Push invalidation for on-disk '-r:' reference assemblies (not tracked by the Roslyn
// workspace): when one changes after an external rebuild, drop the cached options of every
// project referencing it instead of waiting for a timestamp poll to notice.
let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>()
let onWatchedReferenceChanged (path: string) =
for KeyValue(projectId, paths) in referenceWatches do
if paths.Contains path then
match cache.TryRemove projectId with
| true, (_, _, projectOptions) -> checker.InvalidateConfiguration(projectOptions, userOpName = "onWatchedReferenceChanged")
| _ -> ()
// Push invalidation for on-disk '-r:' reference assemblies, which the Roslyn workspace does not
// track. The cached options stay valid (same paths); only the FCS build behind them goes stale.
let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>()
let onWatchedReferenceChanged (path: string) =
for KeyValue(projectId, paths) in referenceWatches do
if paths.Contains path then
match cache.TryGetValue projectId with
| true, (_, _, projectOptions) -> checker.InvalidateConfiguration(projectOptions, userOpName = "onWatchedReferenceChanged")
| _ -> ()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d8dce7a: cache.TryGetValue + InvalidateConfiguration, with the comment rewritten to say what actually goes stale.

Comment on lines +150 to +163
let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) =
clearReferenceWatches projectId

let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase)

for option in projectOptions.OtherOptions do
if option.StartsWithOrdinal "-r:" then
paths.Add(option.Substring "-r:".Length) |> ignore

if paths.Count > 0 then
for path in paths do
referenceChangeTracker.StartWatchingReference path

referenceWatches[projectId] <- paths

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stop-all/start-all on every recompute defeats the ref-counting. tryComputeOptions reruns on every structural change and, through hasDependentVersionChanged, on every edit of a C# project the F# project references. Each time this drops every -r: to refcount 0 (disposing the token) and re-adds it: for paths outside the default directories that is an Unadvise plus an Advise on the service per reference per recompute, for an unchanged set. Roslyn only touches watches when a reference is actually added or removed. Diffing against the previous set keeps the common case free.

Suggested change
let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) =
clearReferenceWatches projectId
let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase)
for option in projectOptions.OtherOptions do
if option.StartsWithOrdinal "-r:" then
paths.Add(option.Substring "-r:".Length) |> ignore
if paths.Count > 0 then
for path in paths do
referenceChangeTracker.StartWatchingReference path
referenceWatches[projectId] <- paths
let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) =
let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase)
for option in projectOptions.OtherOptions do
if option.StartsWithOrdinal "-r:" then
paths.Add(option.Substring "-r:".Length) |> ignore
match referenceWatches.TryGetValue projectId with
| true, previous ->
for path in previous do
if not (paths.Contains path) then
referenceChangeTracker.StopWatchingReference path
for path in paths do
if not (previous.Contains path) then
referenceChangeTracker.StartWatchingReference path
| _ ->
for path in paths do
referenceChangeTracker.StartWatchingReference path
if paths.Count > 0 then
referenceWatches[projectId] <- paths
else
referenceWatches.TryRemove projectId |> ignore

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied verbatim in d8dce7a. clearReferenceWatches stays for ClearOptions and ClearAllCaches, which are the two places a project really goes away.

Comment on lines +351 to +367
let dotnetRoot = Environment.GetEnvironmentVariable "DOTNET_ROOT"

let directories =
seq {
if not (String.IsNullOrEmpty dotnetRoot) then
IO.Path.Combine(dotnetRoot, "packs")

IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs")

IO.Path.Combine(
Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86,
"Reference Assemblies",
"Microsoft",
"Framework"
)

IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NuGet cache. Two things to decide consciously here. Roslyn's ReferenceFileChangeTracker deliberately does not watch the NuGet cache as a directory (only packs and Reference Assemblies) and lets package dlls get per-file advises; a recursive watch over ~/.nuget/packages sees every restore on the machine. If it stays, it should at least honour NUGET_PACKAGES, otherwise a relocated cache silently degrades to per-file watches for everything.

Suggested change
let dotnetRoot = Environment.GetEnvironmentVariable "DOTNET_ROOT"
let directories =
seq {
if not (String.IsNullOrEmpty dotnetRoot) then
IO.Path.Combine(dotnetRoot, "packs")
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs")
IO.Path.Combine(
Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86,
"Reference Assemblies",
"Microsoft",
"Framework"
)
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages")
let dotnetRoot = Environment.GetEnvironmentVariable "DOTNET_ROOT"
let nugetPackages = Environment.GetEnvironmentVariable "NUGET_PACKAGES"
let directories =
seq {
if not (String.IsNullOrEmpty dotnetRoot) then
IO.Path.Combine(dotnetRoot, "packs")
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs")
IO.Path.Combine(
Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86,
"Reference Assemblies",
"Microsoft",
"Framework"
)
if String.IsNullOrEmpty nugetPackages then
IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages")
else
nugetPackages

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d8dce7a honours NUGET_PACKAGES. I kept the directory watch on purpose: here every -r: is watched uniformly and package assemblies are the bulk of them, so dropping it means one advise per package dll for every project. The cost of an unrelated restore is now one lock plus one dictionary lookup per event, since the tracker checks watchedFiles before it allocates a timer (your other comment). The reasoning is in docs/ide/file-watching.md so the next reader does not have to rediscover the Roslyn difference.

Comment on lines +213 to +216
with ex when not (ex :? OperationCanceledException) ->
// Never let a failed advise/unadvise (e.g. non-existent path) kill the
// subscription loop; we simply won't get events for that path.
()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Swallowing silently hides the one failure mode that matters: if CreateDefaultServiceTask faults (the :?> IVsAsyncFileChangeEx2 cast on an unexpected service object), every batch throws here and file watching never works, with nothing in the output pane. The existing helper is enough.

Suggested change
with ex when not (ex :? OperationCanceledException) ->
// Never let a failed advise/unadvise (e.g. non-existent path) kill the
// subscription loop; we simply won't get events for that path.
()
with ex when not (ex :? OperationCanceledException) ->
// Never let a failed advise/unadvise (e.g. non-existent path) kill the
// subscription loop; we simply won't get events for that path.
DebugHelpers.FSharpOutputPane.logExceptionWithContext (ex, nameof FSharpFileChangeWatcher)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d8dce7aFSharpOutputPane.logExceptionWithContext (ex, nameof FSharpFileChangeWatcher), with the DebugHelpers namespace opened the way RoslynHelpers.fs does.

Comment on lines +279 to +283
if WatchedDirectory.FilePathCoveredByWatchedDirectories(watchedDirectories, filePath) then
// Covered by a directory watch; nothing extra to subscribe.
{ new IFSharpWatchedFile with
member _.Dispose() = ()
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this is the hot path (most references are covered by the directory watches) and every call allocates a fresh object expression. One shared instance, like Roslyn's NoOpWatchedFile.Instance. Apply together with the suggestion at line 82 that declares noOpWatchedFile.

Suggested change
if WatchedDirectory.FilePathCoveredByWatchedDirectories(watchedDirectories, filePath) then
// Covered by a directory watch; nothing extra to subscribe.
{ new IFSharpWatchedFile with
member _.Dispose() = ()
}
if WatchedDirectory.FilePathCoveredByWatchedDirectories(watchedDirectories, filePath) then
noOpWatchedFile

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d8dce7a, together with the declaration in FileChangeWatcherImpl.


/// Empirically strong batching window during high activity (solution open/close); see
/// Roslyn's FileChangeWatcher.
let batchingDelay = TimeSpan.FromMilliseconds 500.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declaration for the shared no-op token used at line 279 (apply together with that suggestion).

Suggested change
let batchingDelay = TimeSpan.FromMilliseconds 500.
let batchingDelay = TimeSpan.FromMilliseconds 500.
let noOpWatchedFile =
{ new IFSharpWatchedFile with
member _.Dispose() = ()
}

Comment thread docs/ide/file-watching-design-review.md Outdated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads as a working note rather than repository documentation: it refers to stash@{7}, a local branch name, commit ids that exist only on one machine, and a suggested PR split. Reviewers will ask for it to go. Either a short design note (what is watched and why exactly those directories, the debounce, and the planned follow-up that replaces the stat polling) or move the rationale into the PR description.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced in d8dce7a by docs/ide/file-watching.md: why a watcher, what is watched and why those directories (including where this deliberately differs from Roslyn on the NuGet cache), batching and debounce, the consumer, and the three follow-ups. No stash, branch or commit references.

let ctx = new MockFileChangeContext()
context <- ValueSome ctx
ctx :> IFSharpFileChangeContext

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applyBatch is the most intricate piece of the PR and has no coverage. FSharpFileChangeWatcher takes a Task<IVsAsyncFileChangeEx2>, so Task.FromResult of a recording fake that hands out sequential cookies is enough to check: consecutive WatchFiles land in one AdviseFileChangesAsync call; a run is split when the sink changes; a watch followed by an unwatch in the same batch unadvises the cookie the watch received; unwatching a token that was never advised is a no-op; Dispose of a context unadvises its directory cookies and its remaining file cookies.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in d8dce7a against a RecordingFileChangeService (Task.FromResult, sequential cookies): consecutive WatchFiles land in one AdviseFileChangesAsync; a run is split when the sink changes; watch + unwatch in one batch unadvises the cookie the watch received; unwatching a token without a cookie is a no-op (unwatching now also clears the token cookie, so a second dispose does not unadvise twice); Dispose of a context unadvises its directory cookies and its remaining file cookies. The batching window became a constructor argument — 704a0ad keeps a second constructor with the 500 ms default for production — so the tests run at 100 ms.

* WatchFiles runs are coalesced only while the sink is the same one, so a
  second context's files are never advised with the first context's sink.
* FSharpReferenceChangeTracker keeps its timers in a Dictionary under the
  gate, checks that a path is watched before allocating a timer, and no
  longer races Timer.Change against Timer.Dispose inside the VS callback.
* onWatchedReferenceChanged only invalidates the FCS configuration; the
  cached options are still correct and dropping them forced a second
  InvalidateConfiguration on the next recompute.
* watchReferenceFiles diffs the new '-r:' set against the previous one, so
  an unchanged reference list touches no watches.
* Unwatching clears the token's cookie, so a token without a cookie is a
  no-op instead of a second unadvise.
* NUGET_PACKAGES is honoured for the NuGet cache directory watch, the
  drain loop uses CurrentQueueLength, swallowed batch failures go to the
  F# output pane, covered paths share one no-op token, and the batching
  window is a constructor parameter so tests can shorten it.
* The design-review working note is replaced by a short design note.
* Tests cover applyBatch against a recording IVsAsyncFileChangeEx2.
An F# optional parameter is an option cell per call; the production
callers never pass the delay, so give them a constructor without it and
keep the explicit-delay one for tests.
@xperiandri

Copy link
Copy Markdown
Contributor Author

Addressed the fresh-eyes review in d8dce7a / 704a0ad; each inline thread has a reply. The description is reframed as you suggested: the watcher is infrastructure, the current stat loops (IsReferencesInvalidated, ReferencesOnDisk) still notice a rebuilt dll on the next request, and the latency win arrives with the two follow-ups now listed explicitly (an FCS-side reference-change notification for the incremental builder, and a watcher-invalidated stamp cache for snapshots), plus script #r/#load watching. The Fixes # placeholder is gone.

Checked in VS with breakpoints on both paths: for a `-r:` the workspace
holds as a MetadataReference, Roslyn advises the file itself, swaps the
reference when it changes and bumps Project.Version, so the reactor
recomputes and calls InvalidateConfiguration on its own. That path hits
first — Roslyn batches over 500 ms where the tracker adds a 2 s debounce
on top — and a second subscription only invalidates the same
configuration again, later.

So FSharpProjectOptionsReactor goes back to what it was, and this PR
ships the transport alone. The consumers that the workspace does not
already cover — script `#load` sources, the snapshot stamp cache,
an FCS-side reference notification — follow separately.
@xperiandri xperiandri changed the title Invalidate F# project options when a referenced assembly is rebuilt Add an IVsAsyncFileChangeEx2 file change watcher to FSharp.Editor Sep 6, 2026
@xperiandri

Copy link
Copy Markdown
Contributor Author

Reworked in c15f47b after checking the premise in VS instead of assuming it.

Breakpoints on both invalidation paths — onWatchedReferenceChanged in the reactor and the isProjectInvalidated branch of tryComputeOptions — with a referenced project rebuilt from the command line: Roslyn's path fires first. ProjectSystemProjectFactory already advises every MetadataReference path, swaps the reference when the file changes and bumps Project.Version, and the reactor recomputes and calls InvalidateConfiguration off the back of that. The tracker gets there later by construction: Roslyn batches over 500 ms, this adds a 2 s debounce on top of the same window.

So the reference subscription is gone and FSharpProjectOptionsReactor is untouched by this PR. What remains is the transport plus its tests, and the description now says plainly that it has no consumer yet and lists the three the workspace genuinely does not cover: script #load sources, the ReferencesOnDisk stamp cache, and an FCS-side notification replacing IsReferencesInvalidated.

Happy to fold the first of those into this PR instead, if you would rather not take a transport on its own.

@xperiandri xperiandri changed the title Add an IVsAsyncFileChangeEx2 file change watcher to FSharp.Editor Add an IVsAsyncFileChangeEx2 file change watcher to FSharp.Editor Sep 6, 2026
FSharpReferenceChangeTracker now records each watched path's last-write
stamp in the same entry as its ref-count and token, and drops it on the
raw change notification. IReferenceStamps serves a cached stamp only
while the path is watched — a notification can still reach it — and
stats unwatched paths directly.
The reactor registers the '-r:' paths of every project it computes
options for, diffing against the previous set so an unchanged list
touches no watches, and exposes the tracker's stamps. It passes no
change handler: invalidating the FCS build is Roslyn's job, which swaps
the MetadataReference and bumps Project.Version.
The snapshot-reuse guard compared ReferencesOnDisk by stat'ing every
'-r:' on each new Project instance, before the same-version fast path.
It now reads the tracker's stamps, and on a mismatch drops the project's
stamps so a missed notification costs one re-stat rather than a rebuild
per Project instance.
@xperiandri xperiandri changed the title Add an IVsAsyncFileChangeEx2 file change watcher to FSharp.Editor Cache reference timestamps behind an IVsAsyncFileChangeEx2 watcher for snapshot reuse Sep 6, 2026
@xperiandri

Copy link
Copy Markdown
Contributor Author

The transport now has its consumer, and it is the one that actually removes a stat loop: the snapshot-reuse guard in createProjectSnapshot no longer stats every -r: per new Project instance; it reads stamps that the watcher keeps inside its watch entries (dadf230f5a, 744355228a, 7153224f14, 15bd8ce03e).

The reactor subscription is back, but not for the reason it was removed in c15f47bd52: it registers the -r: set so the stamps exist, and passes no change handler. Invalidating the FCS build stays Roslyn's job, as measured earlier.

Soundness rule, since it is the one thing worth reading closely: a cached stamp lives in the watch entry and is dropped on the raw notification, so it can only be served while a notification can still reach it; anything unwatched is stat'd directly; a mismatch against the snapshot's own stamps drops the project's stamps, so a missed notification costs one re-stat pass, not a rebuild per Project instance.

Description and docs/ide/file-watching.md are updated accordingly; the three remaining stat sites (FromOptions, IsReferencesInvalidated, script #load) are listed as follow-ups.

FSharpReferenceChangeTracker gets a public Dispose with the interface
forwarding to it, the MailboxProcessor pattern, so the reactor disposes
it and the agent without casts. The reactor's opens follow the
System / FSharp.Compiler / Microsoft / Internal.Utilities grouping.
@xperiandri

xperiandri commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

CI: every Windows leg failed with FS1116/FS1118 at FileChangeWatcher.fs(31,12) (EndsWithOrdinal), and CheckCodeFormatting flagged WorkspaceExtensions.fs.

Cause: inline members of the internal module Internal.Utilities.Library cannot be inlined into another assembly, InternalsVisibleTo or not — the optimizer drops their optimization data at the assembly boundary, so a consumer compiled with --optimize+ (Release) fails with FS1118, while --optimize- (Debug) compiles and calls the compiled method instead. Reproduced with a two-file library: module internal Lib + [<InternalsVisibleTo>] + let inline f … gives FS1116/FS1118 in the consumer under --optimize+ with both the SDK and the bootstrap compiler; a public module works; --realsig and a signature file make no difference. FSharp.Editor on main uses none of those helpers, which is why it only surfaced here.

Fix: call String.StartsWith/EndsWith with an explicit StringComparison directly in FSharp.Editor (plus the fantomas fix); the illib StartsWithOrdinalIgnoreCase helper stays. Verified locally: FSharp.Editor builds in Release with that change.

Inline members of the internal Internal.Utilities.Library module cannot be inlined into another assembly, InternalsVisibleTo or not: the optimizer drops their optimization data at the assembly boundary, so FSharp.Editor fails with FS1116/FS1118 under --optimize+ (every Windows Release leg of the CI). Debug compiled only because --optimize- never tries to inline them. Also formats WorkspaceExtensions.fs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@xperiandri
xperiandri force-pushed the vs/file-change-watcher branch 2 times, most recently from 6f7e4be to bc53ca8 Compare September 6, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

⚠️ Affects-Design-Time Tooling check: PR touches type providers or dependency manager

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

2 participants