Switch to offline stack trace symbolization on Windows

[notvictorl]

Previously, stack_symbolizer bypassed symbolization on Windows, assuming Windows Clang ASan would provide provide symbolized stack traces through an online symbolizer. However, Windows Clang ASan's online symbolizer is flaky and could fail to restart the symbolizer.

Chrome archives provide a llvm-symbolizer.exe for use but it's ignored due to favoring online symbolizer on windows on ClusterFuzz and its windows extraction was not supported. ClusterFuzz also includes an llvm-symbolizer.exe but this acts as a fallback if one is not included or USE_DEFAULT_LLVM_SYMBOLIZER = True.

Switches to offline symbolization for windows and updates llvm-symbolizer.exe tooling for Windows to LLVM 24. Adds support for extracting llvm-symbolizer.exe as a common file on ChromeBuildArchives. Adds documentation for updating symbolizer binaries and adds unit testing that symbolization is performed on Windows.

Tested that the changes don't crash on dev by uploading a testcase: https://gosst.development.clusterfuzz.com/testcase-detail/5136503223648256

b/552035861

[notvictorl]

I'll also try testing this on dev

[dylanjew]

@ViniciustCosta Do you know if OSSFuzz does any Windows fuzzing? I'm wondering if we should mark this under a chrome directory?

[notvictorl]

I'll also try testing this on dev

Need to clarify on this PR that Chrome has a primary llvm-symbolizer.exe included for windows so the updated llvm-symbolizer.exe is a fallback. Also, can't upload testcases on windows so I don't think we can easily verify the symbolizer works as intended until it hits a crash to output :/

will make some corrections

[g-ortuno]

A couple of questions:

1. Does ClusterFuzz actually extract the llvm-symbolizer? Does the "llvm-symbolizer" matcher in get_target_dependencies find llvm-symbolizer.exe? Can we add a test?
2. My understanding from this comment was that we disabled the external symbolizer because it didn't work. I guess we think it works now and we want to match linux and mac? If that's the case, we should probably change that code to not enable online symbolization.

However, Windows does not automatically have symbolized stack traces so we should proceed with symbolizing the addresses.

Also this is not quite correct, I think, please correct me if I'm wrong! AFAICT, we do enable symbolization for Windows, see this testcase's original stack trace where it says we failed to restart the symbolizer. This is where my understanding gets a bit iffy, but I think since we're not extracting llvm-symbolzer.exe, we use the old one in the repo which fails and leaves the stack trace unsymbolized. Anyway, I think what you're doing by enabling symbolization is the right fix, but we should make sure we disable online symbolization.

[letitz]

Also, can't upload testcases on windows so I don't think we can easily verify the symbolizer works as intended until it hits a crash to output :/

You should be able to upload testcases to windows jobs, is there a known issue with that path?

[notvictorl]

You should be able to upload testcases to windows jobs, is there a known issue with that path?

What I receive when uploading to clusterfuzz is:

ERROR: Job "windows_libfuzzer_chrome_asan" does not support running untrusted workloads. Untrusted workloads on Chrome are only supported on Linux.

[ViniciustCosta]

@ViniciustCosta Do you know if OSSFuzz does any Windows fuzzing? I'm wondering if we should mark this under a chrome directory?

Yeah I do think that only Chrome is currently fuzzing on Windows

[letitz]

You should be able to upload testcases to windows jobs, is there a known issue with that path?

What I receive when uploading to clusterfuzz is:

ERROR: Job "windows_libfuzzer_chrome_asan" does not support running untrusted workloads. Untrusted workloads on Chrome are only supported on Linux.

You need to type in "This testcase is safe to run." or whatever the exact string is in the text box at the bottom of the upload form.

[notvictorl]

1. Does ClusterFuzz actually extract the llvm-symbolizer?

thanks for pointing this out, ClusterFuzz does extract the llvm-symbolizer for DefaultBuildArchives but not ChomeBuildArchives, so I'll need to modify that.

For the other points, we should use offline symbolization for windows because we can see its brittle and we know its consistently packaged in our archives and in ClusterFuzz as a fallback.

[g-ortuno]

Just one small thing otherwise LGTM!

[g-ortuno]

Hmmm there's a bug in our mac arm64 code for this 🙃 . Unrelated to this CL though.

[notvictorl]

oop I added a TODO to support mac arm64 in the README whenever its added. i think manuel is the contact for this (?)

[dylanjew]

is this captured in a bug? We aren't tracking TODOs in the clusterfuzz repo :)

[dylanjew]

Maybe add a child bug on b/537850811?

[notvictorl]

a bug to track does make more sense. created bug b/559164517 and changed it in the readme

[dylanjew]

Can you update the description with the results from testing in dev?

Otherwise LGTM and approving since Windows is Chrome only.

[dylanjew]

thanks, this is helpful!