Compare Python Typecheckers
Testing & Qualitycompare-python-typecheckersCompare Python Typecheckers is an Agent Skill for testing & quality published in the JetBrains/intellij-community repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Run a Python file or snippet through the real third-party type checkers (ty, pyrefly, basedpyright, mypy, zuban) via uvx and collate their output into one Markdown report. Use when evaluating how a type should be inferred, debugging PyCharm code-insight / type-engine behaviour, or checking where the external checkers agree or disagree on a snippet.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/JetBrains/intellij-community/blob/HEAD/.claude/skills/compare-python-typecheckers/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/compare-python-typecheckers, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Compare Python type checkers
When reasoning about how PyCharm should infer a type or whether something ought to be a type error, cross-checking against the real third-party checkers is the fastest way to gain insight — and to see where they disagree. This skill runs them all on a scratch file or snippet and collates the output into a single report.
Run it
${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py invokes each checker
through uvx (fetched on demand — no install; the network is used on first run)
and prints a Markdown report: a summary table (exit code / verdict / timing per
tool) followed by each tool's raw output.
# Inline snippet:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py -c 'def f(x: int) -> int:
return x
f("a")'
# An existing file:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py path/to/test.py
# A subset, written to a file:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py test.py --tools ty,mypy -o /tmp/report.md
Flags: -c/--code (inline snippet), -t/--tools ty,mypy,… (subset), -o/--output
(write to file), --timeout (per-checker seconds, default 180). The script exits
0 whenever it produced a report — findings live in the report, not the exit code.
Reading the result
- The summary table shows each tool's exit code (0 = nothing flagged, non-zero = the checker reported an issue or failed to run), a verdict, and time.
- Output formats differ per tool, so read each section, not just the table:
ty(Astral) andpyrefly(Meta) each have their own diagnostic style.basedpyrightis pyright-based and also emits extra diagnostics such asreportUnusedCallResult.mypyandzubanshare the same message format (zubanis mypy-compatible).
- The checkers track their latest releases through
uvx, so behaviour can shift over time — record the date when capturing results in a YouTrack issue or test.
Manual one-offs
The exact per-tool invocations, if you want to run just one by hand (note
basedpyright takes the file positionally — no check subcommand):
| Tool | Command |
|---|---|
| ty | uvx ty check test.py |
| pyrefly | uvx pyrefly check test.py |
| basedpyright | uvx basedpyright test.py |
| mypy | uvx mypy test.py |
| zuban | uvx zuban check test.py |