Glass Tests
Testing & Qualityglass-testsGlass Tests is an Agent Skill for testing & quality published in the microsoft/PTVS repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Use when working with PTVS Glass tests: setup_glass.py, run_glass.py, Glass.TestAdapter, Newtonsoft.Json assembly mismatches, Azure Pipelines Glass cache, TestScript.xml, glass2.exe, and mixed-mode Python debugger tests.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/microsoft/PTVS/blob/HEAD/.claude/skills/glass-tests/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/glass-tests, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
PTVS Glass Tests
When to Use
Use this skill when setting up, running, debugging, or investigating CI failures for PTVS Glass tests. Glass tests exercise Visual Studio debugger behavior, including mixed-mode Python/native scenarios, and require Microsoft-internal drops for the Glass runner.
Link to existing human-facing documentation instead of duplicating it. Start with Python/Tests/GlassTests/readme.txt, which links the internal Concord Glass docs and presentation.
Key Files
- Build/setup_glass.py: downloads
drop.exe, Glass, NuGet tools, and vstest drops; extracts VSIX payloads; copies PTVS debugger bits; discovers installed Python versions; generates.GlassTestPropsfiles; verifies the test listing. - Build/run_glass.py: calls
copy_ptvs_output(), verifies the runtime test root, and invokesvstest.console.exewith/Paralleland TRX logging. - Build/templates/run_tests.yml: installs Python versions, restores or populates the
GlassTestscache, acquires the Azure DevOps token, runs setup, runs tests, and publishes TRX results. - Python/Tests/GlassTests/PythonTests/PythonConcord.GlassTestRoot: root Glass test configuration.
Python/Tests/GlassTests/PythonTests/**/TestScript.xml: individual debugger test scripts with commands, expected events, breakpoints, and source expectations.GlassTests/: generated/runtime directory for downloaded Glass bits and copied tests; do not treat it as the source of truth for test definitions.
Local Workflow
-
Build PTVS first so the chosen build output contains
EnvironmentDiscover.exe,Microsoft.Python*, andDkmDebu*files. -
Prefer passing the build output explicitly, especially for VS 18 builds:
python Build\setup_glass.py --buildOutput BuildOutput\Debug18.0\raw\binaries -
If setup needs internal drop access, provide an auth token environment variable or allow the interactive credential flow:
python Build\setup_glass.py --authTokenVariable SYSTEM_ACCESSTOKEN --buildOutput <path-to-raw-binaries> -
List discovered tests after setup:
python Build\setup_glass.py --verifyListing -
Run all tests or a single test:
python Build\run_glass.py --buildOutput BuildOutput\Debug18.0\raw\binaries python Build\run_glass.py --buildOutput BuildOutput\Debug18.0\raw\binaries Repr_Bytes-311-64
CI Failure Workflow
- Start with the PR log and
TestResults/PythonTests.trx. - Classify the failure:
- Setup/drop/auth failure before
vstest.console.exestarts. - Discovery/adapter failure after
vstest.console.exestarts but before individual tests run. glass2.exeorTestScript.xmlfailure inside a specific test case.
- Setup/drop/auth failure before
- Check Build/templates/run_tests.yml to see whether
GlassTestscame from cache. Cache hits skipsetup_glass.py, butrun_glass.pystill callscopy_ptvs_output(). - For assembly binding errors, inspect every
Newtonsoft.Json.dllunderGlassTestsand compare assembly versions.Glass.TestAdapter.dllexpectsNewtonsoft.Json, Version=13.0.0.0; NuGet/vstest VSIX extraction can place older 9.x or 10.x copies in the runner tree. The compatible copy usually comes fromGlassTests\Glass\Newtonsoft.Json.dll. - Do not assume a PTVS NuGet restore or package update will fix Glass runner dependency conflicts. The failing assembly may come from downloaded Glass/NuGet/vstest VSIX payloads, not from
packages/.
Debugging TestScript Failures
- Reproduce one test at a time with
python Build\run_glass.py --buildOutput <path> <test-name>. - Inspect the failing test's
TestScript.xmland source files underPython/Tests/GlassTests/PythonTests/. - Compare the error log against expected events, source file names, line numbers, breakpoints, and function names in the script.
- Look for generated logs, dumps, and copied debuggee files under the runtime
GlassTests\PythonTests\...\obj\<configuration>\andbin\<configuration>\paths. - To debug Glass itself, set
GLASS_DEBUG=1, rerun one test, attach Visual Studio toglass2.exe, and check assertions or event handling. - To debug with the real product, follow Python/Tests/GlassTests/readme.txt: run PTVS under the debugger, open the generated test folder, mirror the
TestScript.xmlbreakpoints, and uselaunch.vs.jsonwith the intended interpreter.
Common Pitfalls
- Full setup is Microsoft-internal because
drop.exedownloads from internal DevDiv artifact drops. run_glass.pywithout--buildOutputhistorically probes 17.0 build output paths; pass--buildOutputfor local VS 18 builds.EnvironmentDiscover.exe not foundusually means the build output path is wrong or stale.PythonConcord.GlassTestRootorPythonEngine.regdefmissing means the runtime test copy is incomplete or setup did not run successfully.- Cache hits can preserve stale
GlassTestscontents. Fixes that must run before test execution should be in therun_glass.pypath, usually viacopy_ptvs_output(). - Runtime files under
GlassTests/are generated and cacheable; source test changes belong underPython/Tests/GlassTests/.