[FastDev] Add faster FastDeploy2

[simonrozsival]

Summary

This is the cleaned-up follow-up to the experimentation PR:

• Experiment PR: [WIP] Experiment with alternate FastDeploy strategies #11692

This PR keeps legacy FastDeploy in place, adds a new FastDeploy2 strategy, and makes FastDeploy2 the default for app install fast deployment.

$(_AndroidFastDevStrategy)=FastDeploy|FastDeploy2

FastDeploy remains selectable as a fallback.

Review follow-up

Jon's review feedback is addressed in the latest cleanup commit:

• FastDeploy2 no longer accepts legacy FastDeploy task inputs it does not use.
• FastDeploy2 is now a single concrete task with no FastDeploy2Base inheritance split.
• Development-only FastDeploy2 diagnostic/telemetry JSON was removed, so production builds only keep the functional manifest state.
• FastDeploy2 manifest JSON serialization uses a System.Text.Json source-generated context.
• Repeated remote path, shell quoting, and directory grouping helpers are consolidated.
• Remote staging now uses Android's writable /data/local/tmp/fastdeploy2 instead of /tmp/fastdeploy2, because /tmp is read-only on CI emulators.
• The local manifest now records device/package/user/ABI identity, and device-side staging/override state is validated with manifest hash markers read in a single adb shell call.
• FastDeploy2 diagnostic logging now records exact adb commands, exit codes, stdout, and stderr for device-state sanity checks.

A fresh dotnet new maui Android sample build binlog is available here:

• https://gist.github.com/simonrozsival/c06cacff00c99ee3504c256205e24492

CI on-device test fixes

The first full CI run surfaced regressions in existing fast-deployment device tests now that FastDeploy2 is the default. Three deterministic root causes were found (each reproduced and verified on an API 36 emulator) and fixed:

• Missing sync log markers. IncrementalFastDeployment and SkipFastDevAlreadyInstalledFile assert per-file NotifySync CopyFile / NotifySync SkipCopyFile markers. FastDeploy2 only logged Prepared X => Y, so these tests failed even though deployment succeeded. FastDeploy2 now emits the markers from the manifest's changed-file set, and the abi-skip marker uses the same device-relative identifier for consistency.
• Symlink mode followed subdirectory symlinks into staging. The full rewrite used ln -sf "$s"/* ., which also symlinked staging subdirectories. Processing a child directory then cd'd through that symlink into the shell-owned /data/local/tmp staging area, and rm -rf ./* failed with "Permission denied" under run-as (XA0133 / XA0129). This broke any app with culture/satellite subdirectories (LocalizedAssemblies_ShouldBeFastDeployed, DotNetNewAndroidTest). The full rewrite now clears and symlinks only the regular files in each directory and leaves subdirectories to their own iterations, so it no longer follows or deletes child directories.
• Copy-fallback stat format split by the device shell. find ... -exec stat -c "%n|%s|%Y" {} + reached the device shell unquoted, so | was treated as a pipe (exit 127 → XA0129) and the copy fallback also failed. The command is now built as a single string with the format single-quoted (matching the RunAsShell quoting pattern).
• Device-state probe printf '\n' arrived as a literal \n. The readiness check built its probe with printf '\noverride=' / printf '\n'; the backslash escape does not survive the adb + device-shell quoting layers, so the device emitted remote=<hash>\noverride=<hash>\n on a single line. ParseDeviceManifestState (splitting on real newlines) put the whole tail into RemoteHash and left OverrideHash empty, so remoteReady was always false and every incremental install reset the staging directory and redeployed all files (changed files: 329 in CI), reporting unchanged assemblies as CopyFile. The probe is now built with echo + command substitution (real newlines, no backslash escapes); verified on an API 36 emulator that remoteReady becomes true on an unchanged reinstall.

FastDeploy2 approach

FastDeploy2 uses the final direction from the experiment PR:

1. Build a local manifest of the deployment target and FastDev inputs:

   device id, package, Android user, ABI
   target path -> local source path, size, mtime
   

2. Compare with the last successful deploy manifest, then verify that the device-side staging tree and app override tree both carry the expected manifest hash.

3. Push only changed files to device temp storage, without adb push --sync:

   adb push -z any changed-file-1 changed-file-2 /data/local/tmp/fastdeploy2/<package>/<user>/<abi>/

4. Remove stale staged files for removed inputs.

5. Maintain files/.__override__ as symlinks to staged files with batched run-as sh -c commands. The generated shell scripts use short d/s variables plus cd so long source/target directories are not repeated for every file.

6. After a successful deploy, write the manifest and matching device-side manifest hash markers:

   /data/local/tmp/fastdeploy2/<package>/<user>/.fastdeploy2-manifest-hash
   files/.__override__/.fastdeploy2-manifest-hash
   

Why this shape is viable now

The existing FastDeploy v2.0 design was added when Android 11 broke the older external-storage based fast deployment model. At that time, adb compression and multi-file push/sync behavior were still new: Android SDK Platform-Tools 30.0.0 (April 2020) introduced client-side compression for adb {push,pull,sync} when used with Android 11 devices, and Platform-Tools 30.0.5 (November 2020) later fixed adb push --sync with multiple inputs and improved pushing many files over high-latency links.

Today it is more reasonable to depend on a modern Android SDK Platform-Tools package for developer inner-loop scenarios. adb is supplied by the Android SDK Platform-Tools package, and developers using .NET for Android / MAUI can update it independently of end-user device support. With Platform-Tools 36.0.0, adb push -z any was also tested against API 24 and API 29 emulators that did not advertise sendrecv_v2* compression features; both accepted the command and transferred the file, apparently negotiating down to uncompressed transfer.

This PR also no longer relies on adb push --sync: the local manifest tells us exactly which files changed, so FastDeploy2 pushes only those files and avoids an adb-side scan of the full directory.

What else we tried but abandoned

From the experiment PR:

• Existing FastDeploy is excellent for very small changes but very slow for full payloads.
• adb push --sync over the whole directory is much faster for full payloads, but still scans/probes too much for tiny changes.
• One adb push per file is too slow because every adb invocation has ~40ms median overhead even when skipped.
• A local manifest gives us the changed set without probing all files through adb.
• Symlinks avoid app-private copy after the initial setup.
• A shell-based symlink update was faster than a native helper (maui.link), so this PR does not include a native helper.

How legacy FastDeploy could be improved instead

If we wanted to keep the native FastDeploy tool model, the most promising improvement would be to make xamarin.sync batch-aware. Instead of invoking run-as ... xamarin.sync once per changed file, the host could open a single run-as <package> xamarin.sync-batch stdin stream and write all changed files one after another using a small binary protocol:

magic/version
file-count
repeat file-count times:
  target-path-length
  target-path
  uncompressed-size
  mtime
  compressed-blocks
removed-file-count
repeat removed-file-count times:
  target-path-length
  target-path

The device helper would create directories, write each file to a temporary path, set mtime/permissions, rename into place, and remove stale files. This would keep the existing app-private real-file model and avoid symlink assumptions, while removing most per-file adb/run-as process overhead.

That approach is intentionally not part of this PR because it keeps the custom native tool/protocol complexity. FastDeploy2 tries the simpler path first: use adb's built-in transfer/compression support, keep the changed-file decision local, and leave legacy FastDeploy available as a fallback while we validate the symlink approach across more devices.

Preliminary benchmark from the experiment PR

Device: physical Samsung Galaxy A16 (SM-A165F, R58Y30HZ65V) connected by USB 3 cable
Project: samples/HelloWorld/HelloWorld/HelloWorld.DotNet.csproj

Strategy	Case	Deploy task	Wall	Upload/transfer	Changed
FastDeploy (existing)	first	22.11s	31.61s	19.09s	194
FastDeploy (existing)	one DLL	0.68s	10.10s	0.14s	2
FastDeploy (existing)	app DLLs	0.69s	10.10s	0.20s	3
FastDeploy (existing)	all DLLs	17.72s	27.27s	17.16s	189
FastDeploy2	first	5.41s	14.51s	3.09s	195
FastDeploy2	one DLL	0.56s	9.70s	0.06s	2
FastDeploy2	app DLLs	0.59s	9.99s	0.04s	3
FastDeploy2	all DLLs	3.02s	12.40s	2.42s	189

Validation

Focused task builds pass in this branch:

dotnet build-server shutdown
 dotnet build build-tools/xa-prep-tasks/xa-prep-tasks.csproj --nologo -v:minimal -m:1
 dotnet build src/Xamarin.Android.Build.Debugging.Tasks/Xamarin.Android.Build.Debugging.Tasks.csproj --no-restore --nologo -v:minimal -m:1
 dotnet build build-tools/xa-prep-tasks/xa-prep-tasks.csproj -c Release --nologo -v:minimal -m:1
 dotnet build src/Xamarin.Android.Build.Debugging.Tasks/Xamarin.Android.Build.Debugging.Tasks.csproj -c Release --no-restore --nologo -v:minimal -m:1

Device/emulator validation passed:

Device	Scenario	Result
Physical Samsung Galaxy A16 (SM-A165F, R58Y30HZ65V) over USB 3 cable	clean first install with default FastDeploy2 symlink mode	passed; app launched successfully
Physical Samsung Galaxy A16 (SM-A165F, R58Y30HZ65V) over USB 3 cable	one-DLL incremental install with default FastDeploy2 symlink mode	passed; app launched successfully
Physical Samsung Galaxy A16 (SM-A165F, R58Y30HZ65V) over USB 3 cable	one-DLL incremental install with copy fallback mode	passed; app launched successfully
API 36 emulator	clean first install with shortened d/s shell-variable symlink scripts	passed; symlink tree created successfully
API 36 emulator	Symlink-mode install followed by Copy-mode install	passed; symlinks were replaced by regular files and the symlink marker was removed

Observed validation details:

first default: deploy=15.84s, mode=Symlink, changed=195, pushed=195, symlink update=161ms
one DLL default: deploy=0.89s, mode=Symlink, changed=2, pushed=2, symlink update=4ms
one DLL Copy fallback: deploy=2.80s, mode=Copy, changed=2, pushed=2, copy=977ms

Additional adb compatibility probes with Platform-Tools 36.0.0:

Target	Device adb features	adb push -z any
API 24 emulator	cmd only, no sendrecv_v2*	passed
API 29 emulator	cmd only, no sendrecv_v2*	passed
API 36 emulator	sendrecv_v2, brotli, lz4, zstd	passed

Risks

• Legacy FastDeploy is still present and selectable.
• Symlinks from app-private files/.__override__ to /data/local/tmp/fastdeploy2/... worked on the tested physical Samsung device over USB.
• The new approach has only been tested on a single device from a single vendor so far; it needs broader Android/OEM/API/device testing.
• If symlink creation fails, this code can fallback to copy mode and clears symlink-managed override state before copying.
• If the local manifest is missing, does not match the current device/package/user/ABI, or does not match the device-side manifest hash marker, FastDeploy2 does a full push.
• Developers with very old Android SDK Platform-Tools may need to update adb for adb push -z any; older devices can still fall back to uncompressed transfer when driven by a modern adb.

Future work

• Delete the old FastDeploy implementation once we are confident the new approach is better across a broader device matrix.
• Delete the old native FastDeploy helper tools once FastDeploy no longer needs them.
• If symlink validation across devices is not good enough, consider a batched xamarin.sync protocol as the app-private real-file fallback design.

[Copilot]

Pull request overview

This PR introduces a new manifest-driven fast deployment implementation (FastDeploy2) alongside the legacy FastDeploy, and makes FastDeploy2 the default strategy for app install fast deployment.

Changes:

• Register and invoke the new FastDeploy2 MSBuild task, with _AndroidFastDevStrategy defaulting to FastDeploy2.
• Add FastDeploy2Base (core orchestration + adb/run-as helpers) plus a manifest-based FastDeploy2 implementation that pushes only changed files and updates overrides via symlink (with copy fallback).
• Add MSBuild-time validation for supported strategy/transfer-mode values, and a new adb compression algorithm property default (any).

Show a summary per file

File	Description
src/Xamarin.Android.Build.Debugging.Tasks/Xamarin.Android.Common.Debugging.targets	Registers FastDeploy2, selects strategy, sets defaults for strategy/transfer mode/compression, and wires task invocation into _Upload.
src/Xamarin.Android.Build.Debugging.Tasks/Tasks/FastDeploy2.cs	Adds FastDeploy2Base task implementation: device/package orchestration, adb invocation, file staging/stat/diff/copy paths, diagnostics logging, and error handling.
src/Xamarin.Android.Build.Debugging.Tasks/Tasks/FastDeploy2.Manifest.cs	Implements manifest-driven incremental upload, remote “ready” marker handling, stale cleanup, and shell-symlink override updates with copy fallback.

Copilot's findings

• Files reviewed: 3/3 changed files
• Comments generated: 8

[simonrozsival]

/review

[github-actions]

✅ Android PR Reviewer completed successfully!

[github-actions]

⚠️ Needs Changes

Thorough review of the new FastDeploy2 implementation. The architecture is well thought out — manifest-based incremental diffing, a clean strategy/mode split in the targets with condition-first <Error> validation, careful shell-argument quoting (ShellQuote single-quote escaping) and package-name sanitization (IsSafePackageNameForShell) to prevent shell injection, ADB cancellation handling with process kill + output draining, and reuse of the existing XA012x/XA013x error codes. Nice work.

A few things to address before merge:

Issues by severity

• ⚠️ 2 warnings
  • Order-dependent shell globbing (FastDeploy2.Manifest.cs:132): the full-refresh rm -f ./* / ln -sf "$s"/* . globs also match subdirectories such as the staged {abi}/environment. Because directories is iterated in nondeterministic HashSet order, this can either delete real staged files through a directory symlink or silently fall back to copy. Please handle nested directories explicitly.
  • Dead/duplicated base implementation (FastDeploy2.cs:549): the virtual DeployFastDevFilesWithAdbPush and its exclusive helpers are fully superseded by the override and never run — including a duplicated CreatePushArgsPrefix that still uses --sync. Recommend deleting it (or making the method abstract).
• 💡 2 suggestions
  • MarkRemoteReady ignores the adb exit code (FastDeploy2.Manifest.cs:447).
  • ManifestEntry.LocalPath is persisted but never read (FastDeploy2.Manifest.cs:496).

See the inline comments for mechanisms and concrete fixes.

CI: license/cla passed; the Azure DevOps builds (Linux/Mac/Windows) are still in progress at review time, so green CI is not yet confirmed — please ensure the internal pipeline passes before merge.

The symlink-glob behavior is the main correctness concern; the rest are maintainability cleanups. I couldn't reproduce on-device, so if I've misread the staging layout for the glob case, a quick confirmation of how {abi}/environment is handled during a full refresh would resolve it.

Generated by Android PR Reviewer for issue #11698 · 1.4K AIC · ⌖ 48.9 AIC · ⊞ 37.8K
_{Comment /review to run again}

[jonathanpeppers]

I think I want to see a .binlog of a deploy to understand what it does -- I might have time to manually test some later.

Overall, I like what it does -- introduce a private property by default you can opt out of if something goes wrong.

[jonathanpeppers]

Is there an existing helper somewhere that does this? I can think of maybe things related to Java names, or Android resources.

[simonrozsival]

I looked and couldn't find an existing helper for turning an arbitrary string into a filesystem-safe path segment. The Java-name and Android-resource helpers (identifier mangling / resource-designer naming) solve a different problem with different rules — they're about valid C#/Java identifiers, not reversible, collision-free directory names. Here we just need a unique cache directory per device/package/user/abi, which is what Uri.EscapeDataString gives (reversible, no path separators). Happy to switch to a shared helper if you have one in mind — leaving this thread open for your call.

[simonrozsival]

@jonathanpeppers I captured a binlog from building a fresh dotnet new maui sample app with this branch's local SDK:

https://gist.github.com/simonrozsival/c06cacff00c99ee3504c256205e24492

Command used for the successful binlog:

./dotnet-local.sh build JonMauiBinlog/JonMauiBinlog.csproj \
  -f net11.0-android \
  -p:TargetFrameworks=net11.0-android \
  -p:SupportedOSPlatformVersion=24.0 \
  -bl:JonMauiBinlog-net11.0-android.binlog

The TFM override keeps the template Android-only because I only installed maui-android locally; the supported OS override matches the local Android SDK minimum of 24.

[jonathanpeppers]

@jonathanpeppers I captured a binlog from building a fresh dotnet new maui sample app with this branch's local SDK:

https://gist.github.com/simonrozsival/c06cacff00c99ee3504c256205e24492

Command used for the successful binlog:

./dotnet-local.sh build JonMauiBinlog/JonMauiBinlog.csproj \
  -f net11.0-android \
  -p:TargetFrameworks=net11.0-android \
  -p:SupportedOSPlatformVersion=24.0 \
  -bl:JonMauiBinlog-net11.0-android.binlog

The TFM override keeps the template Android-only because I only installed maui-android locally; the supported OS override matches the local Android SDK minimum of 24.

I think your command needs -t:Install to see the FastDeploy2 task run.

[simonrozsival]

@jonathanpeppers you're right — the earlier binlog was build-only. I updated the same gist with a new -t:Install binlog that includes the FastDeploy2 task:

https://gist.github.com/simonrozsival/c06cacff00c99ee3504c256205e24492

New file:

JonMauiBinlog-net11.0-android-install-fastdeploy2.binlog

Command used:

./dotnet-local.sh build JonMauiBinlog/JonMauiBinlog.csproj \
  -t:Install \
  -f net11.0-android \
  --no-restore \
  -m:1 \
  -nr:false \
  -p:TargetFrameworks=net11.0-android \
  -p:SupportedOSPlatformVersion=24.0 \
  -p:AdbTarget=-s%20R58Y30HZ65V \
  -p:_FastDeploymentDiagnosticLogging=False \
  -bl:JonMauiBinlog-net11.0-android-install-fastdeploy2.binlog \
  -clp:PerformanceSummary \
  -v:minimal

The console log shows FastDeploy2 ran and the build succeeded:

11089 ms  FastDeploy2  1 calls
Build succeeded.

[jonathanpeppers]

There are many log messages that just say rm returned:

I think we should probably skip these messages in general unless _FastDeploymentDiagnosticLogging=true. And maybe it should not log if blank?

But then I probably need to see a log where _FastDeploymentDiagnosticLogging is true.

[simonrozsival]

@jonathanpeppers to clarify the physical-device validation: that was a Samsung Galaxy A16 (SM-A165F, serial R58Y30HZ65V) connected over a USB 3 cable, not an emulator and not Wi-Fi adb.

The emulator validation rows in the PR description are separate API 24/29/36 emulator checks.

[simonrozsival]

/review output (fork PR, I had to run it locally):

✅ LGTM — Re-reviewed after the latest commits. All three suggestions from the prior pass are addressed: dead TryParsePushSummary/AdbPushSummaryRegex removed (and the unused RegularExpressions using), DeployFastDevFilesWithAdbPush's result is now explicitly checked, and the new FastDeploy2/transfer-mode/compression knobs are documented in a release note. Task project builds clean (0 warnings). Confirm the queued CI run goes green before merge.

[simonrozsival]

Superseded by #11795, which is based on a branch in dotnet/android (origin) instead of a fork. Continuing review there.