test-suite-setup

WinUI3 Full Test Suite Setup & Execution

This skill performs a complete end-to-end test run of the WinUI3 repository on the local machine. It covers every step from environment initialization through running all ~8000 tests.

AI Agent Instructions

When something goes wrong, read these first. This skill has accumulated reference material later in this document that will save hours of debugging:

• HRESULT Decoder Table — every crash code we've seen (0xC0000602, 0x8007007E, 0x80000003, etc.) with cause + first fix.
• OSS Package Coherence — symptom: cascading combase crashes / >10% External failures. Root cause is almost always an incoherent OSS-pinned package set, not your code.
• CI Test Invocation Map — what hosting mode each CI work item uses. Critical: Managed tests run with /p:HostingMode=WPF, not AppX. Missing this flag will look like "all 700+ Managed tests blocked" when nothing is actually broken.
• Baseline Failure Catalog — methodology for "is this my bug or pre-existing?" before treating any failure as a regression.
• ADO ↔ GitHub Mirror Reference — which clone to edit when the file exists in both.

When this skill is invoked:

1. First, ensure the environment is ready — check if TestPayload exists and is populated. If not, invoke the build skill to initialize and build the repo (product + tests), then run Steps 3-5 below (create payload, machine setup, install runtimes).

   Platform/configuration: The default flavor is x64 Debug (amd64chk). If the user requests a different architecture (e.g., arm64) or configuration (fre/release), pass the appropriate flags to the build skill and adjust -Platform/-Configuration in all subsequent steps. Supported platforms: x86, x64, arm64, arm64ec. Supported configurations: chk (debug), fre (release).

2. Determine the comparison branch, then analyze changed files to suggest relevant tests.

   Branch detection logic (in order):

   1. If the user explicitly specified a branch (e.g., "compare against release/2.8"), use that.
   2. Try to auto-detect the PR target branch:

      git --no-pager config --get "branch.$(git branch --show-current).merge" 2>$null
      

   3. If the current branch name suggests a release/servicing target (matches release/*, servicing/*, hotfix/*), ask the user via ask_user:

      "Your branch appears to target a release branch. Which branch should I compare against?" Choices: detected release branch, main, "Let me specify"

   4. Default to main.

   Get changed files:

   git --no-pager diff --name-only <comparison-branch>...HEAD
   

   If the comparison branch is not available locally, fetch it first:

   git fetch origin <comparison-branch>
   git --no-pager diff --name-only origin/<comparison-branch>...HEAD
   

   If that also fails, fall back to:

   git --no-pager diff --name-only HEAD~5..HEAD
   

3. Map changed files to test suites using a hybrid approach: known mappings first, then dynamic discovery for anything new, and a user fallback for unmatched files.

   Step 3a: Known mappings (fast path)

   These curated rules cover the most common source → test relationships:

   Changed files match	Recommended test DLLs	Hosting Mode
   dxaml/xcp/core/**, dxaml/xcp/dxaml/**	External.Foundation, External.Framework, External.Controls	WPF
   dxaml/xcp/components/imaging/**	Isolated.Foundation.Imaging	None
   dxaml/xcp/components/text/**	External.Foundation (text tests), Isolated.Text	WPF / None
   dxaml/xcp/components/focus/**	Isolated.Xaml.Focus.*	None
   dxaml/xcp/components/accesskeys/**	Isolated.AccessKeys.*	None
   dxaml/test/native/external/foundation/**	External.Foundation	WPF
   dxaml/test/native/external/framework/**	External.Framework	WPF
   dxaml/test/native/external/controls/**	External.Controls	WPF
   dxaml/test/native/external/automation/**	External.Automation	WPF
   dxaml/test/native/external/enterprise/**	External.Enterprise	WPF
   dxaml/test/native/external/win32/**	External.Win32	WPF
   dxaml/test/native/external/activation/**	External.Activation	WPF
   dxaml/test/native/external/tools/**	External.Tools	WPF
   dxaml/test/resources/** (images, GIFs, masters)	Isolated.Foundation.Imaging, External.Foundation	None / WPF
   dxaml/test/managed/**	Managed.*	WPF
   controls/dev/** (WinUI controls source)	MUXControls.Test.dll, External.Controls	None / WPF
   controls/test/**	MUXControls.Test.dll	None
   controls/idl/**	MUXControls.Test.dll, External.Controls	None / WPF
   src/XamlCompiler/**	Isolated.Tools.XbfGenerator	None
   dxaml/xcp/dxaml/themes/**	External.Controls, External.Framework	WPF
   Samples/**	ScenarioTestSuite (sample app tests, separate payload)	N/A
   build/**, eng/**, packaging/**	No functional tests needed (build/infra only)	N/A
   Only .md, .txt, config files	No tests needed	N/A

   Step 3b: Dynamic discovery (for new/unknown test areas)

   For any changed files that don't match the known mappings above, dynamically discover matching tests:

   1. Scan available test DLLs in the TestPayload:

      Get-ChildItem "<repo-root>\TestPayload\x64chk\Test\Microsoft.UI.Xaml.Tests.*.dll" | Select-Object -ExpandProperty Name
      

   2. Match by naming convention — extract keywords from the changed file path and look for test DLLs with matching names:

      • Changed dxaml/xcp/components/<area>/** → look for *Isolated.<Area>* or *External.<Area>*
      • Changed dxaml/test/native/external/<area>/** → match *External.<Area>*
      • Changed dxaml/test/native/isolated/<area>/** → match *Isolated.<Area>*
      • Changed controls/dev/<ControlName>/** → match *MUXControls* and *<ControlName>*

   3. Use TAEF metadata for fine-grained matching when the DLL is found but you want to narrow to specific tests:

      .\te.exe Test\<matched-dll> /list /select:"@Name='*<keyword>*'"
      

      This confirms the DLL actually contains tests related to the changed area.

   Step 3c: User fallback (for unresolvable changes)

   If any changed files still can't be mapped after Steps 3a and 3b, ask the user via ask_user:

   "I found changes in <path> but couldn't auto-detect the relevant tests. Which test suite covers this area?" Choices: list of all discovered test DLLs + "Skip — no tests needed" + "Let me specify"

   If no source code files changed (only docs, config, build scripts), tell the user no functional tests are needed.

   If changes span multiple areas, union all the recommended test DLLs from Steps 3a, 3b, and 3c.

4. Present the recommendation using ask_user. Show:

   • The files that changed (summarized)
   • The recommended test suites based on the mapping
   • Choices: Run recommended tests (Recommended), Choose different tests, Run all tests

   If user picks "Choose different tests", dynamically list all available test DLLs from the TestPayload:

   Get-ChildItem "<repo-root>\TestPayload\x64chk\Test\Microsoft.UI.Xaml.Tests.*.dll" |
     ForEach-Object { $_.BaseName -replace '^Microsoft\.UI\.Xaml\.Tests\.', '' }
   

   Present each discovered suite as a choice, plus:

   • All Tests (all hosting modes) — runs everything
   • Custom query — let user specify a wildcard pattern like *CommandBar*

   Hosting mode inference for dynamically discovered DLLs:

   • DLL name contains External. → WPF
   • DLL name contains Isolated. → None
   • DLL name contains Managed. → WPF
   • MUXControls.Test.dll, UnitTests.dll → None
   • Unknown → ask user or try without hosting mode

5. Run the selected tests using te.exe directly for specific DLLs (more precise than runtests.cmd):

   cd <repo-root>\TestPayload\x64chk
   
   # For a specific test DLL (preferred for targeted runs):
   .\te.exe Test\Microsoft.UI.Xaml.Tests.Isolated.Foundation.Imaging.dll /p:SkipConsoleWindowMinimize
   
   # For External.* with WPF hosting:
   .\te.exe Test\Microsoft.UI.Xaml.Tests.External.Controls.dll /p:SkipConsoleWindowMinimize /p:HostingMode=WPF
   
   # For Managed.* with WPF hosting:
   .\te.exe Test\Microsoft.UI.Xaml.Tests.Managed.Controls.dll /p:SkipConsoleWindowMinimize /p:HostingMode=WPF
   
   # For multiple DLLs at once:
   .\te.exe Test\Microsoft.UI.Xaml.Tests.Isolated.Foundation.Imaging.dll Test\Microsoft.UI.Xaml.Tests.External.Foundation.dll /p:SkipConsoleWindowMinimize
   
   # Using runtests.cmd with wildcard (runs across all DLLs):
   .\runtests.cmd *AnimatedImage* -HostingMode:WPF
   
   # For ALL tests — run each hosting mode separately:
   .\runtests.cmd * -HostingMode:UAP
   .\runtests.cmd * -HostingMode:WPF
   .\runtests.cmd * -HostingMode:Win32Explicit
   .\runtests.cmd * -HostingMode:None
   

6. After tests complete, provide a detailed results summary. Parse the TAEF output and present:

   Required summary format:

   • A results table per test suite with columns: Total, Passed, Failed, Blocked, Skipped
   • For each failed test: the full test name, the failure reason (e.g., assertion message, HRESULT, CRC mismatch), and the source file/line if available
   • For each blocked test: the full test name and the blocking reason (e.g., missing privileges, missing Te.Service, test marked Ignore)
   • An overall verdict: whether the changes are safe (no new failures) or need attention
   • Actionable next steps for any failures or blocked tests (e.g., "run from admin prompt", "update master image", "pre-existing failure unrelated to your changes")

   How to extract failure details: Search the TAEF output for lines containing [Failed], [Blocked], Error:, Verify: (failed assertions), and Summary:. Use Select-String or Select-Object -Last to find relevant lines. For each failed test, look for the lines between StartGroup: <testname> and EndGroup: <testname> [Failed] to capture the full failure context.

   Example summary:

   ### Test Results: Isolated.Foundation.Imaging
   | Total | Passed | Failed | Blocked | Skipped |
   |-------|--------|--------|---------|---------|
   | 39    | 38     | 1      | 0       | 0       |
   
   #### Failed Tests
   1. **DecodingUnitTests::Jpeg**
      - Reason: CRC mismatch (Expected 0x5fd3ea9b, Actual 0xda5a7f30)
      - Source: DecodingUnitTests.cpp, Line 111
      - Action: Pre-existing environment-specific failure, unrelated to your changes. Can be ignored.
   
   #### Blocked Tests
   (None)
   
   ### Overall Verdict
   ✅ Your changes did not introduce any new test failures.
   

   Classify failures as:

   • New failure (likely caused by the current changes) — needs investigation
   • Pre-existing failure (exists on main branch too) — note it but don't block
   • Environment-specific (CRC mismatch, timing, machine config) — suggest rerun or admin prompt

   If any tests are blocked due to missing admin privileges or Te.Service, provide the exact command to rerun them from an elevated prompt.

Hosting mode rules:

• External.* tests → use /p:HostingMode=WPF for local runs (UAP requires full UWP container)
• Isolated.* tests → no hosting mode needed (unit tests, no UI)
• Managed.* tests → use /p:HostingMode=WPF (C# tests only work in WPF mode)
• MUXControls.Test.dll / UnitTests.dll → no hosting mode needed
• When user selects "All Tests", run all 4 hosting modes sequentially

Prerequisites

• Windows 10/11 (20H2 or later recommended)
• Visual Studio 2022 with C++ workloads and Spectre mitigation libs
  • If missing Spectre libs, import the .vsconfig file from the repo root via VS Installer
• Admin privileges (required for test machine setup and running tests)
• A visible desktop session (tests launch UI apps and send input — do NOT run over minimized RDP or locked screen)

Step 1–2: Build the Repository (Delegate to Build Skill)

Before creating the test payload, the repo must be initialized and built (product + tests).

Invoke the build skill to handle this. When delegating, request:

• A full repo build (product + tests): .\initrun.ps1 .\build.cmd /q /b
• The /b flag is recommended for test-suite prep to avoid PCH memory exhaustion on limited-memory machines.
• For a clean first-time build, use /c as well: .\initrun.ps1 .\build.cmd /q /c /b

The build skill handles init, flavor selection, troubleshooting (PCH errors, stale PCH, missing Spectre libs, etc.).

Note: amd64 and x64 are used interchangeably. init.ps1 uses amd64chk while CreateTestPayload.ps1 uses -Platform x64 -Configuration chk, but they produce output in the same TestPayload\x64chk\ directory. Ensure the build flavor matches the payload platform/configuration.

Step 3: Create the Test Payload

The test payload aggregates all built test binaries, TAEF infrastructure, and dependencies into a single directory.

.\initrun.ps1 powershell -ExecutionPolicy Bypass -File test\CreateTestPayload.ps1 -Platform x64 -Configuration chk

Parameters:

• -Platform: x86, x64, arm64, arm64ec (must match your build flavor)
• -Configuration: chk or fre (must match your build flavor)
• -Mode: DevTestSuite (default, for functional tests) or ScenarioTestSuite (for sample app tests)
• -Clean: Wipe the TestPayload directory before creating

The payload is created at <repo-root>\TestPayload\<platform><config>\ (e.g., TestPayload\x64chk\).

Step 4: One-Time Machine Setup (Admin Required)

This sets up crash dump collection, installs test certificates, installs required AppX framework packages, and kills processes known to interfere with tests.

# Open an ADMIN PowerShell prompt, then:
cd <repo-root>\TestPayload\x64chk
.\testmachine-prerun.cmd

What it does:

• Enables developer mode (sideloading without dev license)
• Disables network flyout
• Adds Windows Defender exclusion for test directory
• Kills interfering processes (Photos, Skype, YourPhone, etc.)
• Installs VCLibs and .NET runtime AppX packages
• Configures crash dump collection for te.exe and te.processhost.exe

This only needs to be run once per machine (or after OS reinstall).

Step 5: Install .NET Desktop Runtime (if needed)

If managed (C#) tests fail to launch, install the .NET Desktop Runtime:

cd <repo-root>\TestPayload\x64chk
.\dotnet-windowsdesktop-runtime-installer.exe /install /quiet /norestart

Step 6: Install VC++ Redistributable (if needed)

cd <repo-root>\TestPayload\x64chk
.\vc_redist.x64.exe /install /quiet /norestart

Step 7: Run All Tests

From the TestPayload directory, use runtests.cmd to run tests. Tests use the TAEF (te.exe) framework.

Run ALL tests (all hosting modes):

WinUI tests run in different hosting modes (UAP, WPF, Win32Explicit). You need to run each mode separately.

cd <repo-root>\TestPayload\x64chk

# Run all UAP-hosted tests (native C++ tests):
.\runtests.cmd * -HostingMode:UAP

# Run all WPF-hosted tests (native + managed C# tests):
.\runtests.cmd * -HostingMode:WPF

# Run all Win32Explicit-hosted tests:
.\runtests.cmd * -HostingMode:Win32Explicit

# Run tests with no hosting mode (unit tests, app tests):
.\runtests.cmd * -HostingMode:None

Run a specific test or subset:

# Run tests matching a name pattern:
.\runtests.cmd *CommandBar*

# Run a specific test class:
.\runtests.cmd *CommandBarAutomationIntegrationTests*

# Run with WPF hosting:
.\runtests.cmd *MenuFlyout* -HostingMode:WPF

Useful flags:

# See test statistics without running:
.\runtests.cmd * -Stat

# List tests without running them:
.\runtests.cmd *CommandBar* /list

# Wait for debugger attachment:
.\runtests.cmd *MyTest* -WaitForDebugger

# Run including disabled/ignored tests:
.\runtests.cmd *MyTest* -RunIgnoredTests

# Run tests in a loop (for reliability testing):
.\runtests.cmd *MyTest* -RunTestsInALoop

# Stop on first failure:
.\runtests.cmd *MyTest* -TerminateOnFirstFailure

# Slow down test execution to watch it:
.\runtests.cmd *MyTest* /p:GoSlow

Using te.exe directly (advanced):

cd <repo-root>\TestPayload\x64chk

# Run a specific test DLL with WPF hosting:
.\te.exe Test\Microsoft.UI.Xaml.Tests.External.Controls.dll /p:SkipConsoleWindowMinimize /p:HostingMode=WPF

# Run with a TAEF select query:
.\te.exe Test\Microsoft.UI.Xaml.Tests.External.Controls.dll /select:"@Name='*CommandBar*'" /p:SkipConsoleWindowMinimize

# Run managed tests in WPF mode:
.\te.exe Test\Microsoft.UI.Xaml.Tests.Managed.Controls.dll /p:HostingMode=WPF /p:SkipConsoleWindowMinimize

Full Automated Script (All Steps)

Below is a complete PowerShell script that runs all steps end-to-end. Must be run from an admin PowerShell prompt.

Adjust $platform and $config if targeting a different architecture (e.g., arm64 + chk for ARM64 debug).

# === WinUI3 Full Test Suite — End-to-End ===
# Run from admin PowerShell at repo root

$ErrorActionPreference = "Stop"
$repoRoot = Get-Location

# --- Configuration ---
# Change these to target a different architecture/configuration
$platform = "x64"       # x86, x64, arm64, arm64ec
$config   = "chk"       # chk (debug) or fre (release)
$flavor   = "${platform}${config}"   # e.g., x64chk, arm64chk

# --- Step 1: Init ---
Write-Host "=== Step 1: Initializing build environment ===" -ForegroundColor Cyan
.\init.ps1

# --- Step 2: Build (clean, reduced parallelism) ---
Write-Host "=== Step 2: Building repository (product + tests) ===" -ForegroundColor Cyan
.\initrun.ps1 .\build.cmd /q /c /b

# --- Step 3: Create Test Payload ---
Write-Host "=== Step 3: Creating test payload ===" -ForegroundColor Cyan
.\initrun.ps1 powershell -ExecutionPolicy Bypass -File test\CreateTestPayload.ps1 -Platform $platform -Configuration $config -Clean

# --- Step 4: Machine Setup ---
Write-Host "=== Step 4: One-time machine setup ===" -ForegroundColor Cyan
Push-Location "$repoRoot\TestPayload\$flavor"
.\testmachine-prerun.cmd

# --- Step 5: Install runtimes if present ---
Write-Host "=== Step 5: Installing runtimes ===" -ForegroundColor Cyan
if (Test-Path ".\dotnet-windowsdesktop-runtime-installer.exe") {
    Start-Process -Wait -FilePath ".\dotnet-windowsdesktop-runtime-installer.exe" -ArgumentList "/install /quiet /norestart"
}
if (Test-Path ".\vc_redist.x64.exe") {
    Start-Process -Wait -FilePath ".\vc_redist.x64.exe" -ArgumentList "/install /quiet /norestart"
}

# --- Step 6: Run Tests ---
Write-Host "=== Step 6: Running all tests ===" -ForegroundColor Cyan

# Run UAP tests
Write-Host "--- Running UAP tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:UAP

# Run WPF tests
Write-Host "--- Running WPF tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:WPF

# Run Win32Explicit tests
Write-Host "--- Running Win32Explicit tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:Win32Explicit

# Run no-hosting-mode tests (unit tests, app tests)
Write-Host "--- Running unit/app tests ---" -ForegroundColor Yellow
.\runtests.cmd * -HostingMode:None

Pop-Location
Write-Host "=== All tests complete ===" -ForegroundColor Green

Notes

• ~8000 tests total, taking ~8 hours to run serially across all hosting modes.
• Tests require a visible desktop session. UI tests will fail if the Remote Desktop window is minimized, the screen is locked, or another window has focus. Look for errors like RpcSetForegroundWindow or SetForegroundWindow: Failed.
• Managed tests (C#) can only run in WPF hosting mode (not UAP).
• Native tests (C++) can run in both UAP and WPF modes.
• Use -TerminateOnFirstFailure for quick validation.
• Use -Stat to see test statistics before running.
• Test output is in the TestPayload\<platform><config>\Test\WexLogFileOutput\ folder.

Test Suite Reference

All test DLLs are in TestPayload\<platform><config>\Test\. Here is the complete listing:

External Tests (E2E UI integration — C++ native, require UAP or WPF hosting)

Isolated Tests (Unit tests — C++ native, no hosting mode needed)

Managed Tests (C# — WPF hosting mode only)

Other Test DLLs

Mapping: Source Code → Test DLL

OSS Package Coherence

The OSS build of WinUI pins a small set of dependency packages directly in eng/Versions.props (the internal build ignores these and uses Maestro/Darc-managed versions instead). When those pins are not from the same internal build run, the symptoms are dramatic but the diagnosis is subtle.

Symptom recognition

If you see any of these, suspect package incoherence before suspecting your code:

• Mass 0xC0000602 (RoFailFastWithErrorContext) crashes on test startup — combase reports WinRT activation metadata mismatch.
• External category pass rate drops below ~80% (healthy is ~91%).
• Adaptability / Automation / Tools / Enterprise DLLs show cascading "Blocked" or crashes.
• Microsoft.UI.Xaml.dll loads but a controls type fails to activate at runtime.
• Same test passes in the internal CI but crashes locally on an OSS-restored build.

The IsInternalWinUIBuild gating pattern

eng/Versions.props (around line 21) defines:

<IsInternalWinUIBuild Condition="'$(IsInternalWinUIBuild)' == '' AND Exists('..\.azuredevops\')">true</IsInternalWinUIBuild>

The flag is true whenever ..\.azuredevops\ exists (i.e., when the repo is cloned from the ADO mirror, not from GitHub). Lines under Condition="'$(IsInternalWinUIBuild)' != 'true'" are the OSS-only pins — typically 5–7 packages:

• Microsoft.Internal.FrameworkUdk
• Microsoft.WindowsAppSDK.InteractiveExperiences (IXP) and its transport package
• Microsoft.WindowsAppSDK.Foundation and its transport package
• Microsoft.WindowsAppSDK.Base
• Microsoft.WindowsAppSDK.WinUIDetails

Internal builds read versions from eng/Version.Details.xml (Maestro dep-flow). OSS builds read the literal pins.

What "coherent" means

Two packages are coherent when their version strings prove they were produced by the same internal build run (byte-identical date stamp / build label, e.g. version suffixes ending in the same YYMMDD-HHMM.B token). Examples of incoherence that look fine to a human reviewer but break at runtime:

• FrameworkUdk from build A combined with the FrameworkUdk DLL embedded inside the IXP package from build B — both packages claim the same nominal version, but the embedded DLL differs by KB.
• MUX compiled against WinUIDetails headers from build A, then linked at deployment time against the runtime IXP from build B.
• Foundation and Base snapped from different days.

WinUIDetails is the secret killer

WinUIDetails is a headers-only NuGet package, but the headers ARE the compile-time ABI surface (vtable layouts, IIDs, struct sizes). If WinUIDetails is from a different build run than the runtime IXP / Foundation packages, MUX compiles successfully against one set of vtable layouts and then encounters a different set at activation time → 0xC0000602.

How to find a coherent snapshot

Two paths:

1. Dep-flow / BAR (preferred) — query the BAR (Build Asset Registry) or use Darc to find a recent build that produced all the packages together. The build label suffix is the coherence proof.
2. Cache inspection — check what's already on the machine: packages\, packages_temp\, ~\.nuget\packages\ for matching date stamps across the OSS-pinned package set.

Validating the snapshot is published to the OSS feed

OSS builds restore from the WinUI-Dependencies feed on dev.azure.com/shine-oss. To verify a specific version is published before pinning:

$tok = (az account get-access-token --resource '499b84ac-1321-427f-aa17-267ca6975798' --query accessToken -o tsv)
$pkg = 'microsoft.internal.frameworkudk'  # lowercase
Invoke-RestMethod -Uri "https://pkgs.dev.azure.com/shine-oss/microsoft-ui-xaml/_packaging/WinUI-Dependencies/nuget/v3-flatcontainer/$pkg/index.json" -Headers @{ Authorization = "Bearer $tok" }

A 200 with the version listed = ready to pin. A 404 = package needs to be promoted/published first.

The two-tier package consumption model

• Product DLL (Microsoft.UI.Xaml.dll) consumes transport packages (*TransportPackage), gated by IsInternalWinUIBuild.
• Test/sample apps consume the flat packages (LiftedIXPPackageName, FoundationPackageName) — and these versions are typically set unconditionally, not gated.

Both tiers must be coherent or runtime DLLs in TestPayload\ will conflict.

PR pattern when bumping OSS pins

A PR that updates the OSS-pinned versions:

• Touches only eng/Versions.props (and eng/Version.Details.xml if needed for OSS-feed mirroring).
• All changes gated by Condition="'$(IsInternalWinUIBuild)' != 'true'" so internal CI is unaffected.
• Must include a pass-rate validation table (see "PR + Validation Template" below).
• Acceptance bar: zero new failures vs unchanged main.

CI Test Invocation Map

When you need to know what CI runs and how it runs it, these two files are the source of truth:

• build/AzurePipelinesTemplates/WinUI-CreateTestPayload-Job.yml — defines each test work item: testFilePathPattern (which DLL or AppX) + hostingMode.

• Helix/GenerateHelixWorkItems.ps1 — look for the line that generates the TAEF command:

  $taefExtraParameters = "/p:HostingMode=$HostingMode"
  

  Every CI test invocation passes a HostingMode value to TAEF via /p:.

CI ↔ local mapping

The Managed-tests gotcha (READ THIS BEFORE DEBUGGING "BLOCKED" MANAGED TESTS)

Managed tests are not AppX-hosted in CI. They run via te.processhost.exe with WPF XamlIsland hosting — i.e., they need /p:HostingMode=WPF.

A local runner that omits this flag will appear to "block all 700+ Managed tests" with 0x80070002 / 0x8007007E / 0x80000003 and lead the investigator down a .rd.xml / .NET Native rabbit hole that does not exist. The fix is one flag: /p:HostingMode=WPF.

This was a multi-week false wall. Don't repeat it.

ADO ↔ GitHub Mirror Reference

The WinUI3 source code lives in two clones, kept in sync by a one-way mirror:

• ADO microsoft.visualstudio.com/WinUI/_git/microsoft-ui-xaml-lift (branch main) — source of truth for all mirrored content.
• GitHub github.com/microsoft/microsoft-ui-xaml (branch winui3/main) — read-only mirror for mirrored files. A small set of GitHub-only files (typically OSS-only docs and samples not present in ADO) can and should be edited directly on GitHub.

The mirror pipeline is build/WinUI-MirrorMainSourceToExternalRepo-Official.yml.

Layout difference

The mirror uses targetRepositorySubdirectory: "src". A file at ADO eng/Versions.props lands at GitHub src/eng/Versions.props. The GitHub repo's main branch is a legacy OSS-only branch — do not edit it; the mirror writes to winui3/main.

Exclusions

build/PipelineScripts/WinUISourceMirroringExclusions.txt controls what does NOT mirror.

Useful when re-enabling a small subset of an excluded tree (e.g., excluding :/test/ but re-including :!/test/scripts/ for the local runner scripts).

"Which clone do I edit?" decision rule

• File exists in ADO under any mirrored path (most product/test source, this skill, etc.) → edit in ADO, create an ADO PR. Editing the GitHub copy will be silently overwritten on the next mirror sync.
• File exists only on GitHub (rare — typically OSS-only docs or samples added directly to the GitHub repo) → edit on GitHub.
• Submodules — Samples/WinUIGallery is a submodule pointer to a separate repo. The submodule contents are not mirrored as plain files; only the pointer commit is.

Cross-Category Pass-Rate Reference

Expected pass-rate bands and approximate wall times for a healthy local validation run. Use these to calibrate "is my run healthy" before chasing failures.

A pass rate that falls dramatically below the band for a category is almost always either:

1. A package coherence issue (see OSS Package Coherence), or
2. The runner being killed by idle-detection (see Reliability Tweaks).

It is rarely "your code change broke 30% of tests."

Troubleshooting & Known Issues

Build Errors

For build-related issues (PCH virtual memory exhaustion, stale precompiled headers, missing Spectre libs, Run a full init first, etc.), refer to the build skill which covers all build troubleshooting.

Test Payload Errors

CreateTestPayload.ps1 — robocopy exit codes

Symptom: Script appears to "fail" but robocopy exit codes below 8 are actually success.

Root Cause: Robocopy uses non-standard exit codes where 0-7 indicate various forms of success. The script handles this internally.

Fix: Only exit codes 8+ are actual failures. Check CreateTestPayload.log in the TestPayload directory for details.

TestPayload is empty or missing files

Symptom: Test payload directory exists but is missing test DLLs or TAEF binaries.

Root Cause: Build didn't produce all outputs (partial build, or wrong build target).

Fix: Ensure you ran a full build (not just product or mux):

.\initrun.ps1 .\build.cmd /q /b   # builds product + tests

Then recreate the payload:

.\initrun.ps1 powershell -ExecutionPolicy Bypass -File test\CreateTestPayload.ps1 -Platform x64 -Configuration chk -Clean

Machine Setup Errors

testmachine-prerun.cmd fails with access denied

Symptom: Registry operations or AppX installs fail with permission errors.

Fix: Must run from an elevated (admin) PowerShell or cmd prompt.

AppX package install error 0x80073D06 or 0x80073CFB

Symptom: Add-AppxPackage reports an error during machine setup.

Root Cause: A same or newer version of the package is already installed.

Fix: Safe to ignore — the setup script already handles this and prints "The same or higher version of this package is already installed."

Test Execution Errors

SetForegroundWindow: Failed / RpcSetForegroundWindow errors

Symptom: Tests fail because the test app window cannot be brought to the foreground.

Root Cause: UI tests require an active, visible desktop session. The session is either minimized, locked, or another window has focus.

Fix:

• If using Hyper-V: Use basic "Virtual Machine Connection" (vmconnect.exe), NOT Enhanced Session
• If using Remote Desktop: Keep the RDP window visible and focused, do not minimize it
• Do not lock the screen while tests are running
• Check View menu in VM connect to ensure you're NOT using Enhanced Session

Tests show "blocked" status instead of running

Symptom: Managed C# tests report "blocked" and don't execute.

Root Cause: Managed (.NET) tests cannot run in UAP hosting mode. They must use WPF mode.

Fix: Run managed tests with WPF hosting:

.\runtests.cmd *ManagedTestName* -HostingMode:WPF

COM_END WindowHelper::VerifyTestCleanup: Error: Caught Platform::Exception^: Unspecified error

Symptom: Tests fail with COM/platform errors during cleanup.

Root Cause: Usually caused by running tests over Remote Desktop or Hyper-V Enhanced Session.

Fix: Switch to basic VM connection (not Enhanced Session, not RDP). See the SetForegroundWindow fix above.

Tests fail to find test DLLs

Symptom: te.exe reports it cannot find test binaries.

Root Cause: Running runtests.cmd from the wrong directory.

Fix: You must cd into the TestPayload directory first:

cd <repo-root>\TestPayload\x64chk
.\runtests.cmd *MyTest*

Query matched tests requiring conflicting HostingModes

Symptom: runtests.cmd auto-detection fails because the query matches tests in multiple hosting modes.

Root Cause: A broad wildcard (e.g., *) matches tests that require different hosting modes, and the script can only run one mode at a time.

Fix: Explicitly specify the hosting mode:

.\runtests.cmd * -HostingMode:UAP
.\runtests.cmd * -HostingMode:WPF
.\runtests.cmd * -HostingMode:Win32Explicit
.\runtests.cmd * -HostingMode:None

Init / Build Errors

For init and build-related issues (PCH virtual memory exhaustion, stale precompiled headers, missing Spectre libs, NuGet authentication, dotnet-install failures, etc.), refer to the build skill (/.github/skills/build/SKILL.md) which covers all build and init troubleshooting.

HRESULT Decoder Table

Every crash code we've encountered, what it actually means, and the first thing to try.

Baseline Failure Catalog

Before treating any failure as "caused by my change", reproduce it on unchanged main with the same package pins and the same TestPayload state. If it reproduces, it's pre-existing baseline drift, not your bug.

Categories of baseline failures we've consistently observed

• External (component) DLLs — typically 80–110 pre-existing failures on a healthy ~91% pass rate. Distribution roughly:
  • Test, Win32, Convergence — 3–4 failures each.
  • Adaptability — ~6 failures.
  • Enterprise — ~58–74 failures (largest contributor).
  • Tools — ~4–11 failures.
  • These persist across package version changes and across many weeks.
• MUXControlsTestApp (~95% expected) — UI-assertion drift is expected. Examples we've reproduced on unchanged main: RatingControlTests tap-and-return offsets, asynchronous-timing assertions in animation tests.
• Isolated — 1–2 single flakes (often clean on retry). Expected ≥99% pass.
• Managed (HostingMode=WPF) — some Win32.Common deterministic failures show up in the ~79% pass-rate sample.

What is NOT a baseline failure

• Any cascading 0xC0000602 activation crash — that's package incoherence.
• A sudden surge of "Blocked" results across multiple DLLs — that's the runner being killed by idle-detection (see Reliability Tweaks), not a real failure.
• A whole category dropping below its expected band — investigate the package set, not the individual tests.

Test Runner Reliability Tweaks

Patterns that prevent the most common run-time waste during long local validation runs.

5-second keepalive for long External runs

Without keepalive, idle-detection kills External DLL runs after a period of no input — and produces hundreds of "Blocked" results instead of real pass/fail. Two options:

1. Pass a TAEF foreground / idle-keepalive option appropriate for your TAEF version.
2. Run a parallel mouse-wiggle script: move the cursor by 1 pixel every ~5–15 seconds. This keeps SetForegroundWindow permission alive on the active session, which the UI tests need.

Background-shell pattern for long-running DLLs

Anything that runs for ≥5 minutes (Controls = 7+ min, Enterprise / Tools / Framework, full External run) should be launched detached so it survives Copilot CLI / shell restart:

Start-Process pwsh -WindowStyle Normal -ArgumentList '-NoExit','-Command',"<run command> *> '<per-DLL log>'; Add-Content '<csv>' '<dll>,<status>'"

Tee output to a per-DLL log file and append a one-line CSV summary at completion — that gives you an audit trail across the full multi-hour run.

Crash dumps via WER LocalDumps

Configure HKCU once so any test-host crash produces a dump:

HKCU\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\te.processhost.exe
HKCU\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\TE.exe
   DumpFolder = <writable path>
   DumpType   = 2  (full dump)

Critical when chasing combase fast-fail crashes.

TAEF /select filter is unreliable

TE.exe <dll> /select:"@Name='X' OR @Name='Y'" sometimes runs the full suite anyway. When sampling for diagnostic signal:

1. Enumerate first: TE.exe <dll> /list → capture the full test name list.
2. Run the first N test names individually, not via /select.

A 10-tests-per-DLL sample across all DLLs in a category gives ~80% of the diagnostic signal in 10–15% of the wall time. Sample by running fewer DLLs, not by trying to filter inside a DLL.

TestPayload directory naming

Build outputs land at BuildOutput\bin\amd64chk\ but TestPayload lands at TestPayload\x64chk\ — same architecture, different naming convention. This is normal; don't try to "fix" it.

AppX-hosted tests need the right deployment root

For AppX-hosted tests, target the test DLL inside the registered loose layout, not under the raw TestPayload\ directory. The deployment root for DeploymentItem resolution is the parent of the AppX folder.

PR + Validation Template

For changes that affect the OSS test pipeline (Versions.props pins, mirror exclusions, test infra scripts):

Always include a pass-rate validation table

Category                    Total  Run    Pass   Fail   Pass%    Time
Isolated (unit)               718   718    717      1   99.86%   ~2m
IXMPTestApp                     1     1      1      0  100.00%   ~30s
MUXControls.Test (sample)     825    54     50      4   92.59%   ~19m
MUXControlsTestApp            601   543    517     26   95.21%   ~38m
External (component)        1,225 1,225  1,121    104   91.51%   ~3h
Managed (.NET, WPF host)      724   ...    ...    ...    ~79%   ~60-90m
TOTAL                       4,094 ...    ...    ...    ≥94%    ~4.5h

Failure analysis paragraph

State explicitly whether failures reproduce on unchanged main:

• Reproduces on unchanged main → pre-existing baseline drift, not introduced by this PR.
• Does not reproduce on unchanged main → real regression, must be investigated before merge.

The acceptance bar is "PR introduces zero new failures."

Gating note

When the change is gated by Condition="'$(IsInternalWinUIBuild)' != 'true'", say so explicitly so reviewers know the internal CI build is unaffected and only the GitHub OSS build picks up the change.

ADO PR description size

ADO PR descriptions have a 4000-character hard limit. If your validation report is larger:

• Truncate aggressively in the PR body.
• Keep the validation table.
• Drop per-test failure breakdowns; link instead.

Which repo to PR to

Per the ADO ↔ GitHub Mirror Reference above:

• File exists in ADO under any mirrored path → ADO PR.
• File exists only on GitHub → GitHub PR.
• When in doubt, ADO PR — the mirror will propagate it.