Cache reference timestamps behind an IVsAsyncFileChangeEx2 watcher for snapshot reuse - #20457
Cache reference timestamps behind an IVsAsyncFileChangeEx2 watcher for snapshot reuse#20457xperiandri wants to merge 20 commits into
Conversation
The FileChangeWatcher worktree is pull-model I/O (OpenFileForReadShimAsync
plus last-write timestamps on FSharpFileSnapshot), not a push watcher.
Roslyn's FileChangeWatcher is the reference: IVsAsyncFileChangeEx2,
directory subscriptions, 500 ms AsyncBatchingWorkQueue, free-threaded
sinks, coalesced metadata-reference invalidation.
This repo already has two IVsFileChangeEx clients (legacy FileChangeManager
and deprecated FSharpSource.SetDependencyFiles). The intended FSharp.Editor
replacement lives only in stash@{7}
(54465595717b8bb746cb2633d5a4aa834888a481): FileChangeWatcher.fs plus
FileChangeWatcherHub, wired to FSharpProjectOptionsReactor for -r:
assemblies. It is IVsFileChangeEx + JTF.Run, not IVsAsyncFileChangeEx2.
No commit, branch, or GitHub hit implements IVsAsyncFileChangeEx2.
Recommended split: ship the async read shim on its own; restore the stash
watcher or jump straight to IVsAsyncFileChangeEx2 with directory batching;
invalidate FCS via NotifyFileChanged instead of O(N) timestamp polling.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Roslyn-shaped push file watching for FSharp.Editor: - FSharpFileChangeWatcher: batched advise/unadvise (500ms window, coalesced same-kind ops), service obtained via Task without blocking on UI thread - FileChangeContext: free-threaded sink (IVsFreeThreadedFileChangeEvents2), directory subscriptions with extension filters, per-file watches covered by watched directories become no-op tokens - FSharpReferenceChangeTracker: ref-counted reference watching with 2s debounce; default directory watches for DOTNET_ROOT\packs, dotnet\packs, Reference Assemblies, NuGet cache (.dll filter) Modeled on Roslyn FileChangeWatcher/ReferenceFileChangeTracker (all internal there, not reusable from F#). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Subscribe each project's on-disk '-r:' reference assemblies via FSharpReferenceChangeTracker when options are computed; on a watched dll change, drop that project's cached options and invalidate the checker configuration. Subscriptions are ref-counted, cleared on project removal and reactor disposal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Cover WatchedDirectory path matching, tracker ref-counting, debounce of burst notifications, and dispose. Tests use an in-memory IFSharpFileChangeWatcher mock so they do not need a live IVsAsyncFileChangeEx2 service. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
❗ Release notes requiredYou can open this PR in browser to add release notes: open in github.dev
|
| for KeyValue(projectId, paths) in referenceWatches do | ||
| if | ||
| paths | ||
| |> Array.exists (fun p -> String.Equals(p, path, StringComparison.OrdinalIgnoreCase)) |
There was a problem hiding this comment.
We have a helper extension method that can be used here
There was a problem hiding this comment.
There is no equality helper in illib (only StartsWithOrdinal/EndsWithOrdinal/EndsWithOrdinalIgnoreCase/IndexOfOrdinal), so instead of comparing I changed the data: per-project watches are now a HashSet<string>(StringComparer.OrdinalIgnoreCase) and the notification does paths.Contains path. Done in 57f2bb0.
| member _.Dispose() = watched.Clear() | ||
|
|
||
| type private MockFileChangeWatcher() = | ||
| let mutable context: MockFileChangeContext option = None |
There was a problem hiding this comment.
Why not voption in new tests code?
|
|
||
| /// Manages mappings of Roslyn workspace Projects/Documents to FCS. | ||
| type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace) = | ||
| type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace, ?fileChangeWatcher: IFSharpFileChangeWatcher) = |
There was a problem hiding this comment.
Can it ever be none? I would make it mandatory
There was a problem hiding this comment.
Made mandatory in 57f2bb0; LanguageService.fs is the only caller and always passes one.
|
|
||
| [<Sealed>] | ||
| type private FSharpProjectOptionsReactor(checker: FSharpChecker) = | ||
| type private FSharpProjectOptionsReactor(checker: FSharpChecker, fileChangeWatcher: IFSharpFileChangeWatcher option) = |
There was a problem hiding this comment.
I think fileChangeWatcher must be mandatory.
If we need to pass it as null in tests, let's do
let fileChangeWatcher =
withNull fileChangeWatcher |> ValueOption.ofObjThere was a problem hiding this comment.
Done in 57f2bb0 — IFSharpFileChangeWatcher is a plain required parameter now on both the reactor and the manager, and the Option.map/Option.iter plumbing around the tracker is gone. No null path is needed: the tests exercise FSharpReferenceChangeTracker with an in-memory mock watcher and never construct the manager.
| if path.EndsWith(string IO.Path.DirectorySeparatorChar) then | ||
| path | ||
| else | ||
| path + string IO.Path.DirectorySeparatorChar |
There was a problem hiding this comment.
use string intepolation
|
|
||
| [<Sealed>] | ||
| type internal FSharpWatchedFileToken() = | ||
| member val Cookie: uint32 option = None with get, set |
There was a problem hiding this comment.
why not?
| member val Cookie: uint32 option = None with get, set | |
| member val Cookie: uint32 voption = ValueNone with get, set |
There was a problem hiding this comment.
Done in 57f2bb0, plus the two use sites (ValueSome cookie on advise, a match on unadvise).
| [ | ||
| if not (String.IsNullOrEmpty dotnetRoot) then | ||
| IO.Path.Combine(dotnetRoot, "packs") | ||
|
|
||
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs") | ||
|
|
||
| IO.Path.Combine( | ||
| Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86, | ||
| "Reference Assemblies", | ||
| "Microsoft", | ||
| "Framework" | ||
| ) | ||
|
|
||
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages") | ||
| ] | ||
| |> List.distinct | ||
| |> List.map (fun d -> WatchedDirectory(d, [ ".dll" ])) |
There was a problem hiding this comment.
| [ | |
| if not (String.IsNullOrEmpty dotnetRoot) then | |
| IO.Path.Combine(dotnetRoot, "packs") | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs") | |
| IO.Path.Combine( | |
| Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86, | |
| "Reference Assemblies", | |
| "Microsoft", | |
| "Framework" | |
| ) | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages") | |
| ] | |
| |> List.distinct | |
| |> List.map (fun d -> WatchedDirectory(d, [ ".dll" ])) | |
| seq { | |
| if not (String.IsNullOrEmpty dotnetRoot) then | |
| IO.Path.Combine(dotnetRoot, "packs") | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs") | |
| IO.Path.Combine( | |
| Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86, | |
| "Reference Assemblies", | |
| "Microsoft", | |
| "Framework" | |
| ) | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages") | |
| } | |
| |> Seq.distinct | |
| |> Seq.map (fun d -> WatchedDirectory(d, [ ".dll" ])) |
There was a problem hiding this comment.
Applied in 57f2bb0 with one addition: |> List.ofSeq at the end. FileChangeContext iterates the watched directories on every EnqueueWatchingFile, so a lazy seq would re-run Environment.GetFolderPath and Seq.distinct per watched file; materializing once here keeps the single Seq chain but evaluates it one time.
|
🔍 Tooling Safety Check — Affects-Design-Time
|
* FSharpProjectOptionsReactor/Manager take the IFSharpFileChangeWatcher outright; the only caller always has one, so the option wrappers and the Option.iter/map plumbing around the tracker go away. * Reference watches per project are an OrdinalIgnoreCase HashSet, so a change notification is a Contains instead of an Array.exists with an explicit comparison. * FSharpWatchedFileToken.Cookie and the test mock's context are voption. * applyBatch indexes the drained ResizeArray directly instead of converting it to a list and re-slicing it with takeWhile/skip/collect. * StartsWithOrdinal / EndsWithOrdinal / EndsWithOrdinalIgnoreCase from Internal.Utilities.Library at the ordinal call sites, interpolation for the trailing separator, and the default watched directories go through one Seq chain materialized once.
| } | ||
| |> Seq.distinct | ||
| |> Seq.map (fun d -> WatchedDirectory(d, [ ".dll" ])) | ||
| |> List.ofSeq |
There was a problem hiding this comment.
| |> List.ofSeq | |
| |> Seq.toList |
The list-typed applyBatch reads better than the index walk; only the voption use sites differ from the original body.
| pending <- rest | ||
|
|
||
| if cookies.Count > 0 then | ||
| let! _ = service.UnadviseDirChangesAsync(cookies.ToArray(), CancellationToken.None) |
There was a problem hiding this comment.
Why no cancellation? why not cancellableTask CE for enclosing function?
There was a problem hiding this comment.
Done in 439e7ce: applyBatch is a cancellableTask, takes the token with CancellableTask.getCancellationToken () and passes it to every IVsAsyncFileChangeEx2 call.
| | None -> draining <- false | ||
|
|
||
| let! service = fileChangeService |> Async.AwaitTask | ||
| do! applyBatch service (List.ofSeq ops) |> Async.AwaitTask |
There was a problem hiding this comment.
and then propagate cancellation to CancellableTask here
There was a problem hiding this comment.
Done in 439e7ce. The agent is started with a token from a CancellationTokenSource the watcher owns (FSharpFileChangeWatcher is IDisposable now: cancel, dispose, dispose the mailbox), the loop reads it via Async.CancellationToken and runs the batch with CancellableTask.startAsTask ct. The catch-all that kept the loop alive on failed advises became with ex when not (ex :? OperationCanceledException), so cancellation is no longer swallowed. LanguageService.fs constructs the watcher with new accordingly; nothing disposes it there on purpose — it lives as long as the workspace service, same as the checker.
* IFSharpFileChangeWatcher.CreateContext and WatchedDirectory take ImmutableArray, the same contract as Roslyn's FileChangeWatcher; the set is built once and scanned on every EnqueueWatchingFile. * applyBatch is a cancellableTask and passes its token to every IVsAsyncFileChangeEx2 call. The agent runs under a token owned by the watcher, which is now IDisposable; cancellation is no longer swallowed by the loop's catch-all. * Batches are sliced and collected as arrays, so the cookie and path arrays go to the service without a List.toArray copy.
The ignore-case StartsWith was the one ordinal comparison in FileChangeWatcher.fs without an illib helper; the sibling of the existing EndsWithOrdinalIgnoreCase closes that gap and the watched- directory check uses it.
|
|
||
| type internal IFSharpFileChangeWatcher = | ||
| abstract CreateContext: watchedDirectories: ImmutableArray<WatchedDirectory> -> IFSharpFileChangeContext | ||
|
|
There was a problem hiding this comment.
@T-Gro what do we prefer? Interface or function type?
| type internal CreateIFSharpFileChangeContext = ImmutableArray<WatchedDirectory> -> IFSharpFileChangeContext |
There was a problem hiding this comment.
IMO, since you started with interface, unless there is a big win with function (which I don't see), I'd leave it as is.
In general, for a single method, and no design foresight on what it could evolve into later on, function is fine.
The bonus of interface, even for one method is the named arguments.
xperiandri
left a comment
There was a problem hiding this comment.
Fresh-eyes pass over the whole PR against the current Roslyn FileChangeWatcher / ReferenceFileChangeTracker and against what FCS actually does with references. The shape is right: batched advise/unadvise with the 500 ms window, directory watches for the reference roots, ref-counted per-file watches, free-threaded sink. Three things need fixing, and one framing point matters for the description.
Must fix (inline, with suggestions):
WatchFilesruns are coalesced across sinks and advised with the first one. Latent today, breaks as soon as a second context exists.FSharpReferenceChangeTrackerracesTimer.ChangeagainstTimer.Disposeand can throw inside the VS file change callback; it also allocates a timer per unwatched path before checking whether anyone watches it.cache.TryRemoveinonWatchedReferenceChangednever reaches consumers (they go throughProjectCache.Projects) and causes a secondInvalidateConfigurationat the next recompute, discarding a builder that may already be rebuilt.InvalidateConfigurationalone is the right call.
Framing. As it stands the PR does not change observable latency. On the legacy path getOrCreateBuilder (BackgroundCompiler.fs L492) evaluates IsReferencesInvalidated on every request, and that stats every reference with a fresh TimeStampCache (IncrementalBuild.fs L1245). On the transparent-compiler path the snapshot reuse compares ReferencesOnDisk by stat (WorkspaceExtensions.fs L250) and FromOptions stats on creation. Both notice a rebuilt dll at the next request anyway, and InvalidateConfiguration only swaps in a lazy builder node without computing anything. So "keeps serving options and snapshots computed against the old assembly" is not what happens today. The value of the watcher arrives when it replaces those stat loops: a reference-change notification for the incremental builder on the FCS side (the analogue of useChangeNotifications for sources) and a watcher-invalidated stamp cache for snapshots. Worth presenting this PR as that infrastructure and listing both follow-ups explicitly.
Also inline: diff-based watch updates instead of stop-all/start-all on every options recompute, NUGET_PACKAGES and whether the NuGet cache should be a directory watch at all (Roslyn does not do that), a shared no-op token, the drain loop, logging of swallowed exceptions, the design-review doc, and applyBatch test coverage. The Fixes # (issue, if applicable) placeholder is still in the description.
All suggestions were applied together on the branch and pass dotnet fantomas --check; the changed regions were type-checked and smoke-tested with stand-ins for the VS interop types (sink grouping, drain, debounce and the timer race under 8 concurrent producers, watch diffing).
| | WatchFiles(_, _, sink) :: _ -> | ||
| let batch = | ||
| pending | ||
| |> Seq.takeWhile (function | ||
| | WatchFiles _ -> true | ||
| | _ -> false) | ||
| |> Seq.toArray |
There was a problem hiding this comment.
Bug (latent): WatchFiles runs are coalesced across sinks. The whole run of consecutive WatchFiles operations is advised with the sink of the first one. With a single context this cannot bite, but CreateContext allows several, and the moment a second one exists (the script #r/#load watcher from the follow-ups) its files get subscribed with the wrong sink and its notifications are silently lost. Roslyn's CanCombineWith only merges operations with the same sink.
| | WatchFiles(_, _, sink) :: _ -> | |
| let batch = | |
| pending | |
| |> Seq.takeWhile (function | |
| | WatchFiles _ -> true | |
| | _ -> false) | |
| |> Seq.toArray | |
| | WatchFiles(_, _, sink) :: _ -> | |
| let batch = | |
| pending | |
| |> Seq.takeWhile (function | |
| | WatchFiles(_, _, s) -> obj.ReferenceEquals(s, sink) | |
| | _ -> false) | |
| |> Seq.toArray |
There was a problem hiding this comment.
Applied in d8dce7a: the run stops at the first WatchFiles whose sink is not ReferenceEquals to the head one. Covered by A run of file watches is split when the sink changes.
| ctx.FileChanged.Add(fun path -> | ||
| let fire (_: obj) = | ||
| pendingTimers.TryRemove path | ||
| |> function | ||
| | true, timer -> timer.Dispose() | ||
| | _ -> () | ||
|
|
||
| // Only notify for paths someone is actually watching; directory watches | ||
| // cover whole trees. | ||
| let isWatched = lock gate (fun () -> watchedFiles.ContainsKey path) | ||
|
|
||
| if isWatched then | ||
| onChanged path | ||
|
|
||
| let timer = | ||
| pendingTimers.GetOrAdd(path, fun _ -> new Timer(fire, null, Timeout.Infinite, Timeout.Infinite)) | ||
|
|
||
| timer.Change(notificationDelay, Timeout.InfiniteTimeSpan) |> ignore) |
There was a problem hiding this comment.
Bug: Timer.Change races Timer.Dispose. The handler takes the timer out of pendingTimers without the lock; fire (and the tracker's Dispose) remove and dispose it. If an event lands between TryRemove and Change, Change throws ObjectDisposedException straight out of the free-threaded FilesChanged/DirectoryChangedEx2 callback, i.e. inside the VS file change service. Narrow window, but a rebuild is exactly the burst that hits it.
Second problem in the same block: the watchedFiles.ContainsKey check only runs in fire, so every .dll event under the watched directories allocates a Timer first. With ~/.nuget/packages watched recursively, any restore on the machine creates thousands of timers for files nobody watches.
Doing the bookkeeping under gate and checking first fixes both. With everything under the lock pendingTimers can become a plain Dictionary; left out here so the suggestion applies on its own. (Roslyn's current ReferenceFileChangeTracker has no per-path timers at all: one 5 s batching queue with string dedupe.)
| ctx.FileChanged.Add(fun path -> | |
| let fire (_: obj) = | |
| pendingTimers.TryRemove path | |
| |> function | |
| | true, timer -> timer.Dispose() | |
| | _ -> () | |
| // Only notify for paths someone is actually watching; directory watches | |
| // cover whole trees. | |
| let isWatched = lock gate (fun () -> watchedFiles.ContainsKey path) | |
| if isWatched then | |
| onChanged path | |
| let timer = | |
| pendingTimers.GetOrAdd(path, fun _ -> new Timer(fire, null, Timeout.Infinite, Timeout.Infinite)) | |
| timer.Change(notificationDelay, Timeout.InfiniteTimeSpan) |> ignore) | |
| ctx.FileChanged.Add(fun path -> | |
| let fire (_: obj) = | |
| let isWatched = | |
| lock gate (fun () -> | |
| match pendingTimers.TryRemove path with | |
| | true, timer -> timer.Dispose() | |
| | _ -> () | |
| watchedFiles.ContainsKey path) | |
| if isWatched then | |
| onChanged path | |
| lock gate (fun () -> | |
| // Directory watches cover whole trees; only debounce paths someone watches. | |
| if not disposed && watchedFiles.ContainsKey path then | |
| let timer = | |
| pendingTimers.GetOrAdd(path, fun _ -> new Timer(fire, null, Timeout.Infinite, Timeout.Infinite)) | |
| timer.Change(notificationDelay, Timeout.InfiniteTimeSpan) |> ignore)) |
There was a problem hiding this comment.
Applied in d8dce7a, including the step you left out: with all bookkeeping under gate, pendingTimers is a plain Dictionary now. The handler checks watchedFiles before it allocates or arms a timer, and fire removes and disposes under the same lock, so Change can no longer meet a disposed timer inside the VS callback.
| // Push invalidation for on-disk '-r:' reference assemblies (not tracked by the Roslyn | ||
| // workspace): when one changes after an external rebuild, drop the cached options of every | ||
| // project referencing it instead of waiting for a timestamp poll to notice. | ||
| let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>() | ||
|
|
||
| let onWatchedReferenceChanged (path: string) = | ||
| for KeyValue(projectId, paths) in referenceWatches do | ||
| if paths.Contains path then | ||
| match cache.TryRemove projectId with | ||
| | true, (_, _, projectOptions) -> checker.InvalidateConfiguration(projectOptions, userOpName = "onWatchedReferenceChanged") | ||
| | _ -> () |
There was a problem hiding this comment.
cache.TryRemove here does more harm than good. Consumers resolve options through ProjectCache.Projects (WorkspaceExtensions.fs L177), a ConditionalWeakTable keyed by the Roslyn Project instance, so removing the reactor entry does not reach them. When the reactor is next consulted (any new Project instance) the miss forces a recompute, whose success path calls InvalidateConfiguration again (L464) and throws away a builder that may already have been rebuilt against the new dll. Meanwhile GetCompilationDefinesAndLangVersionForEditingDocument and TryGetQuickParsingOptionsForEditingDocumentOrProject fall back to default defines until that recompute.
The options themselves are still correct (same paths); only the FCS build behind them is stale, and InvalidateConfiguration on the cached options is all that is needed.
| // Push invalidation for on-disk '-r:' reference assemblies (not tracked by the Roslyn | |
| // workspace): when one changes after an external rebuild, drop the cached options of every | |
| // project referencing it instead of waiting for a timestamp poll to notice. | |
| let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>() | |
| let onWatchedReferenceChanged (path: string) = | |
| for KeyValue(projectId, paths) in referenceWatches do | |
| if paths.Contains path then | |
| match cache.TryRemove projectId with | |
| | true, (_, _, projectOptions) -> checker.InvalidateConfiguration(projectOptions, userOpName = "onWatchedReferenceChanged") | |
| | _ -> () | |
| // Push invalidation for on-disk '-r:' reference assemblies, which the Roslyn workspace does not | |
| // track. The cached options stay valid (same paths); only the FCS build behind them goes stale. | |
| let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>() | |
| let onWatchedReferenceChanged (path: string) = | |
| for KeyValue(projectId, paths) in referenceWatches do | |
| if paths.Contains path then | |
| match cache.TryGetValue projectId with | |
| | true, (_, _, projectOptions) -> checker.InvalidateConfiguration(projectOptions, userOpName = "onWatchedReferenceChanged") | |
| | _ -> () |
There was a problem hiding this comment.
Done in d8dce7a: cache.TryGetValue + InvalidateConfiguration, with the comment rewritten to say what actually goes stale.
| let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) = | ||
| clearReferenceWatches projectId | ||
|
|
||
| let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase) | ||
|
|
||
| for option in projectOptions.OtherOptions do | ||
| if option.StartsWithOrdinal "-r:" then | ||
| paths.Add(option.Substring "-r:".Length) |> ignore | ||
|
|
||
| if paths.Count > 0 then | ||
| for path in paths do | ||
| referenceChangeTracker.StartWatchingReference path | ||
|
|
||
| referenceWatches[projectId] <- paths |
There was a problem hiding this comment.
Stop-all/start-all on every recompute defeats the ref-counting. tryComputeOptions reruns on every structural change and, through hasDependentVersionChanged, on every edit of a C# project the F# project references. Each time this drops every -r: to refcount 0 (disposing the token) and re-adds it: for paths outside the default directories that is an Unadvise plus an Advise on the service per reference per recompute, for an unchanged set. Roslyn only touches watches when a reference is actually added or removed. Diffing against the previous set keeps the common case free.
| let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) = | |
| clearReferenceWatches projectId | |
| let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase) | |
| for option in projectOptions.OtherOptions do | |
| if option.StartsWithOrdinal "-r:" then | |
| paths.Add(option.Substring "-r:".Length) |> ignore | |
| if paths.Count > 0 then | |
| for path in paths do | |
| referenceChangeTracker.StartWatchingReference path | |
| referenceWatches[projectId] <- paths | |
| let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) = | |
| let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase) | |
| for option in projectOptions.OtherOptions do | |
| if option.StartsWithOrdinal "-r:" then | |
| paths.Add(option.Substring "-r:".Length) |> ignore | |
| match referenceWatches.TryGetValue projectId with | |
| | true, previous -> | |
| for path in previous do | |
| if not (paths.Contains path) then | |
| referenceChangeTracker.StopWatchingReference path | |
| for path in paths do | |
| if not (previous.Contains path) then | |
| referenceChangeTracker.StartWatchingReference path | |
| | _ -> | |
| for path in paths do | |
| referenceChangeTracker.StartWatchingReference path | |
| if paths.Count > 0 then | |
| referenceWatches[projectId] <- paths | |
| else | |
| referenceWatches.TryRemove projectId |> ignore |
There was a problem hiding this comment.
Applied verbatim in d8dce7a. clearReferenceWatches stays for ClearOptions and ClearAllCaches, which are the two places a project really goes away.
| let dotnetRoot = Environment.GetEnvironmentVariable "DOTNET_ROOT" | ||
|
|
||
| let directories = | ||
| seq { | ||
| if not (String.IsNullOrEmpty dotnetRoot) then | ||
| IO.Path.Combine(dotnetRoot, "packs") | ||
|
|
||
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs") | ||
|
|
||
| IO.Path.Combine( | ||
| Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86, | ||
| "Reference Assemblies", | ||
| "Microsoft", | ||
| "Framework" | ||
| ) | ||
|
|
||
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages") |
There was a problem hiding this comment.
NuGet cache. Two things to decide consciously here. Roslyn's ReferenceFileChangeTracker deliberately does not watch the NuGet cache as a directory (only packs and Reference Assemblies) and lets package dlls get per-file advises; a recursive watch over ~/.nuget/packages sees every restore on the machine. If it stays, it should at least honour NUGET_PACKAGES, otherwise a relocated cache silently degrades to per-file watches for everything.
| let dotnetRoot = Environment.GetEnvironmentVariable "DOTNET_ROOT" | |
| let directories = | |
| seq { | |
| if not (String.IsNullOrEmpty dotnetRoot) then | |
| IO.Path.Combine(dotnetRoot, "packs") | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs") | |
| IO.Path.Combine( | |
| Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86, | |
| "Reference Assemblies", | |
| "Microsoft", | |
| "Framework" | |
| ) | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages") | |
| let dotnetRoot = Environment.GetEnvironmentVariable "DOTNET_ROOT" | |
| let nugetPackages = Environment.GetEnvironmentVariable "NUGET_PACKAGES" | |
| let directories = | |
| seq { | |
| if not (String.IsNullOrEmpty dotnetRoot) then | |
| IO.Path.Combine(dotnetRoot, "packs") | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.ProgramFiles, "dotnet", "packs") | |
| IO.Path.Combine( | |
| Environment.GetFolderPath Environment.SpecialFolder.ProgramFilesX86, | |
| "Reference Assemblies", | |
| "Microsoft", | |
| "Framework" | |
| ) | |
| if String.IsNullOrEmpty nugetPackages then | |
| IO.Path.Combine(Environment.GetFolderPath Environment.SpecialFolder.UserProfile, ".nuget", "packages") | |
| else | |
| nugetPackages |
There was a problem hiding this comment.
d8dce7a honours NUGET_PACKAGES. I kept the directory watch on purpose: here every -r: is watched uniformly and package assemblies are the bulk of them, so dropping it means one advise per package dll for every project. The cost of an unrelated restore is now one lock plus one dictionary lookup per event, since the tracker checks watchedFiles before it allocates a timer (your other comment). The reasoning is in docs/ide/file-watching.md so the next reader does not have to rediscover the Roslyn difference.
| with ex when not (ex :? OperationCanceledException) -> | ||
| // Never let a failed advise/unadvise (e.g. non-existent path) kill the | ||
| // subscription loop; we simply won't get events for that path. | ||
| () |
There was a problem hiding this comment.
Swallowing silently hides the one failure mode that matters: if CreateDefaultServiceTask faults (the :?> IVsAsyncFileChangeEx2 cast on an unexpected service object), every batch throws here and file watching never works, with nothing in the output pane. The existing helper is enough.
| with ex when not (ex :? OperationCanceledException) -> | |
| // Never let a failed advise/unadvise (e.g. non-existent path) kill the | |
| // subscription loop; we simply won't get events for that path. | |
| () | |
| with ex when not (ex :? OperationCanceledException) -> | |
| // Never let a failed advise/unadvise (e.g. non-existent path) kill the | |
| // subscription loop; we simply won't get events for that path. | |
| DebugHelpers.FSharpOutputPane.logExceptionWithContext (ex, nameof FSharpFileChangeWatcher) |
There was a problem hiding this comment.
Done in d8dce7a — FSharpOutputPane.logExceptionWithContext (ex, nameof FSharpFileChangeWatcher), with the DebugHelpers namespace opened the way RoslynHelpers.fs does.
| if WatchedDirectory.FilePathCoveredByWatchedDirectories(watchedDirectories, filePath) then | ||
| // Covered by a directory watch; nothing extra to subscribe. | ||
| { new IFSharpWatchedFile with | ||
| member _.Dispose() = () | ||
| } |
There was a problem hiding this comment.
Nit: this is the hot path (most references are covered by the directory watches) and every call allocates a fresh object expression. One shared instance, like Roslyn's NoOpWatchedFile.Instance. Apply together with the suggestion at line 82 that declares noOpWatchedFile.
| if WatchedDirectory.FilePathCoveredByWatchedDirectories(watchedDirectories, filePath) then | |
| // Covered by a directory watch; nothing extra to subscribe. | |
| { new IFSharpWatchedFile with | |
| member _.Dispose() = () | |
| } | |
| if WatchedDirectory.FilePathCoveredByWatchedDirectories(watchedDirectories, filePath) then | |
| noOpWatchedFile |
There was a problem hiding this comment.
Done in d8dce7a, together with the declaration in FileChangeWatcherImpl.
|
|
||
| /// Empirically strong batching window during high activity (solution open/close); see | ||
| /// Roslyn's FileChangeWatcher. | ||
| let batchingDelay = TimeSpan.FromMilliseconds 500. |
There was a problem hiding this comment.
Declaration for the shared no-op token used at line 279 (apply together with that suggestion).
| let batchingDelay = TimeSpan.FromMilliseconds 500. | |
| let batchingDelay = TimeSpan.FromMilliseconds 500. | |
| let noOpWatchedFile = | |
| { new IFSharpWatchedFile with | |
| member _.Dispose() = () | |
| } |
There was a problem hiding this comment.
This reads as a working note rather than repository documentation: it refers to stash@{7}, a local branch name, commit ids that exist only on one machine, and a suggested PR split. Reviewers will ask for it to go. Either a short design note (what is watched and why exactly those directories, the debounce, and the planned follow-up that replaces the stat polling) or move the rationale into the PR description.
There was a problem hiding this comment.
Replaced in d8dce7a by docs/ide/file-watching.md: why a watcher, what is watched and why those directories (including where this deliberately differs from Roslyn on the NuGet cache), batching and debounce, the consumer, and the three follow-ups. No stash, branch or commit references.
| let ctx = new MockFileChangeContext() | ||
| context <- ValueSome ctx | ||
| ctx :> IFSharpFileChangeContext | ||
|
|
There was a problem hiding this comment.
applyBatch is the most intricate piece of the PR and has no coverage. FSharpFileChangeWatcher takes a Task<IVsAsyncFileChangeEx2>, so Task.FromResult of a recording fake that hands out sequential cookies is enough to check: consecutive WatchFiles land in one AdviseFileChangesAsync call; a run is split when the sink changes; a watch followed by an unwatch in the same batch unadvises the cookie the watch received; unwatching a token that was never advised is a no-op; Dispose of a context unadvises its directory cookies and its remaining file cookies.
There was a problem hiding this comment.
Added in d8dce7a against a RecordingFileChangeService (Task.FromResult, sequential cookies): consecutive WatchFiles land in one AdviseFileChangesAsync; a run is split when the sink changes; watch + unwatch in one batch unadvises the cookie the watch received; unwatching a token without a cookie is a no-op (unwatching now also clears the token cookie, so a second dispose does not unadvise twice); Dispose of a context unadvises its directory cookies and its remaining file cookies. The batching window became a constructor argument — 704a0ad keeps a second constructor with the 500 ms default for production — so the tests run at 100 ms.
* WatchFiles runs are coalesced only while the sink is the same one, so a second context's files are never advised with the first context's sink. * FSharpReferenceChangeTracker keeps its timers in a Dictionary under the gate, checks that a path is watched before allocating a timer, and no longer races Timer.Change against Timer.Dispose inside the VS callback. * onWatchedReferenceChanged only invalidates the FCS configuration; the cached options are still correct and dropping them forced a second InvalidateConfiguration on the next recompute. * watchReferenceFiles diffs the new '-r:' set against the previous one, so an unchanged reference list touches no watches. * Unwatching clears the token's cookie, so a token without a cookie is a no-op instead of a second unadvise. * NUGET_PACKAGES is honoured for the NuGet cache directory watch, the drain loop uses CurrentQueueLength, swallowed batch failures go to the F# output pane, covered paths share one no-op token, and the batching window is a constructor parameter so tests can shorten it. * The design-review working note is replaced by a short design note. * Tests cover applyBatch against a recording IVsAsyncFileChangeEx2.
An F# optional parameter is an option cell per call; the production callers never pass the delay, so give them a constructor without it and keep the explicit-delay one for tests.
|
Addressed the fresh-eyes review in d8dce7a / 704a0ad; each inline thread has a reply. The description is reframed as you suggested: the watcher is infrastructure, the current stat loops ( |
Checked in VS with breakpoints on both paths: for a `-r:` the workspace holds as a MetadataReference, Roslyn advises the file itself, swaps the reference when it changes and bumps Project.Version, so the reactor recomputes and calls InvalidateConfiguration on its own. That path hits first — Roslyn batches over 500 ms where the tracker adds a 2 s debounce on top — and a second subscription only invalidates the same configuration again, later. So FSharpProjectOptionsReactor goes back to what it was, and this PR ships the transport alone. The consumers that the workspace does not already cover — script `#load` sources, the snapshot stamp cache, an FCS-side reference notification — follow separately.
|
Reworked in c15f47b after checking the premise in VS instead of assuming it. Breakpoints on both invalidation paths — So the reference subscription is gone and Happy to fold the first of those into this PR instead, if you would rather not take a transport on its own. |
IVsAsyncFileChangeEx2 file change watcher to FSharp.Editor
FSharpReferenceChangeTracker now records each watched path's last-write stamp in the same entry as its ref-count and token, and drops it on the raw change notification. IReferenceStamps serves a cached stamp only while the path is watched — a notification can still reach it — and stats unwatched paths directly.
The reactor registers the '-r:' paths of every project it computes options for, diffing against the previous set so an unchanged list touches no watches, and exposes the tracker's stamps. It passes no change handler: invalidating the FCS build is Roslyn's job, which swaps the MetadataReference and bumps Project.Version.
The snapshot-reuse guard compared ReferencesOnDisk by stat'ing every '-r:' on each new Project instance, before the same-version fast path. It now reads the tracker's stamps, and on a mismatch drops the project's stamps so a missed notification costs one re-stat rather than a rebuild per Project instance.
IVsAsyncFileChangeEx2 file change watcher to FSharp.Editor|
The transport now has its consumer, and it is the one that actually removes a stat loop: the snapshot-reuse guard in The reactor subscription is back, but not for the reason it was removed in Soundness rule, since it is the one thing worth reading closely: a cached stamp lives in the watch entry and is dropped on the raw notification, so it can only be served while a notification can still reach it; anything unwatched is stat'd directly; a mismatch against the snapshot's own stamps drops the project's stamps, so a missed notification costs one re-stat pass, not a rebuild per Description and |
FSharpReferenceChangeTracker gets a public Dispose with the interface forwarding to it, the MailboxProcessor pattern, so the reactor disposes it and the agent without casts. The reactor's opens follow the System / FSharp.Compiler / Microsoft / Internal.Utilities grouping.
|
CI: every Windows leg failed with FS1116/FS1118 at Cause: Fix: call |
Inline members of the internal Internal.Utilities.Library module cannot be inlined into another assembly, InternalsVisibleTo or not: the optimizer drops their optimization data at the assembly boundary, so FSharp.Editor fails with FS1116/FS1118 under --optimize+ (every Windows Release leg of the CI). Debug compiled only because --optimize- never tries to inline them. Also formats WorkspaceExtensions.fs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
6f7e4be to
bc53ca8
Compare
Description
The transparent compiler's snapshot-reuse guard (
WorkspaceExtensions.createProjectSnapshot) decided whether an old snapshot could be reused by stat'ing every-r:reference of the project (getOnDiskReferences) — on every new RoslynProjectinstance, i.e. on every workspace fork, and before the cheap same-version fast path. For a project with a few hundred references that is a few hundredGetLastWriteTimeUtccalls per keystroke-driven typecheck.This PR replaces that stat with a cached stamp that an
IVsAsyncFileChangeEx2watcher keeps fresh.FileChangeWatcher.fs— the watcher, the F# counterpart of Roslyn'sFileChangeWatcher/ReferenceFileChangeTracker(internal toMicrosoft.VisualStudio.LanguageServices, not reachable throughExternalAccess.FSharp): advise/unadvise batched on a single-consumer queue with a 500 ms window, coalesced per kind and per sink; recursive.dlldirectory watches over the reference roots (DOTNET_ROOT/packs, machinedotnet/packs, .NET Framework reference assemblies, the NuGet cache honouringNUGET_PACKAGES); ref-counted per-file watches for everything else; free-threaded sink; batch application as acancellableTaskunder a token the watcher owns.FSharpReferenceChangeTrackerkeeps each watched path's last-write stamp inside its watch entry and drops it on the raw change notification.IReferenceStamps.GetLastWriteTimeUtcserves the cached value only while the path is watched — a stamp is only ever served while a notification can still reach it — and stats unwatched paths directly. The 2 s debounce stays for consumers that do expensive work per change.FSharpProjectOptionsReactorwatches the-r:set of each project it computes options for, diffed on recompute so an unchanged set touches nothing, and exposes the stamps asFSharpProjectOptionsManager.ReferenceStamps. It does not invalidate anything on a change — see below.ReferencesOnDiskguard increateProjectSnapshotreads the stamps. On a mismatch with the snapshot's own stamps (FCS stats when it builds one) it drops the project's stamps, so a missed notification costs one re-stat pass rather than a rebuild on every futureProjectinstance.Why the reactor does not call
InvalidateConfigurationAn earlier revision of this PR did. Checked in VS with breakpoints on both paths and a referenced project rebuilt from the command line: for a
-r:the workspace holds as aMetadataReference, Roslyn already advises the file (ProjectSystemProjectFactory), swaps the reference when it changes and bumpsProject.Version; the reactor sees that throughisProjectInvalidated, recomputes and callsInvalidateConfigurationitself — and it gets there first (Roslyn batches over 500 ms; this tracker adds a 2 s debounce on top of the same window). So the watch here exists for the stamps, not for invalidation.c15f47bd52in the history is that removal; the subscription that returns in744355228aserves a different purpose.Not in this PR
FSharpProjectSnapshot.FromOptionsstats every-r:when a snapshot is built from scratch (FCS side); LetFSharpProjectSnapshot.FromOptionstake reference stamps from the host #20459 adds thegetReferenceStampparameter that lets it read the same cache, and the one-line VS consumer follows once it merges.IsReferencesInvalidatedon the incremental builder (legacy path) stats every reference per request; that needs the reference analogue ofuseChangeNotificationson the FCS side.#loadsources are invisible to the workspace; watching them drops the script's cached options.docs/ide/file-watching.mdrecords what the workspace already covers, what it does not, and the shape of the watcher.Checklist
FileChangeWatcherTests.fs:WatchedDirectorycoverage; tracker ref-counting, debounce and dispose against an in-memory watcher;applyBatchagainst a recordingIVsAsyncFileChangeEx2; and the stamp cache — served while watched until a notification, stat'd when unwatched, dropped byInvalidate, stat'd again once the last watch stops.referencesOnDiskChanged): on repeat passes the reuse guard for projects with 187 and 349-r:read every stamp from the tracker — 0GetLastWriteTimeUtc, the tracker's stamp count was identical before and aftergetOnDiskReferences. Touching a referenced dll dropped its stamp; the next guard re-stat'd that one file and rebuilt the snapshot (References on disk changedin the trace), after which passes were cached again.docs/release-notes/.VisualStudio/18.vNext.md;docs/release-notes/.FSharp.Compiler.Service/11.0.100.mdfor the illib helper.