OpenParable
Skip to content
← Back to skills

Compare Python Typecheckers

Testing & Quality
compare-python-typecheckers

Compare Python Typecheckers is an Agent Skill for testing & quality published in the JetBrains/intellij-community repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Run a Python file or snippet through the real third-party type checkers (ty, pyrefly, basedpyright, mypy, zuban) via uvx and collate their output into one Markdown report. Use when evaluating how a type should be inferred, debugging PyCharm code-insight / type-engine behaviour, or checking where the external checkers agree or disagree on a snippet.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/JetBrains/intellij-community/blob/HEAD/.claude/skills/compare-python-typecheckers/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/compare-python-typecheckers, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Compare Python type checkers

When reasoning about how PyCharm should infer a type or whether something ought to be a type error, cross-checking against the real third-party checkers is the fastest way to gain insight — and to see where they disagree. This skill runs them all on a scratch file or snippet and collates the output into a single report.

Run it

${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py invokes each checker through uvx (fetched on demand — no install; the network is used on first run) and prints a Markdown report: a summary table (exit code / verdict / timing per tool) followed by each tool's raw output.

# Inline snippet:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py -c 'def f(x: int) -> int:
    return x

f("a")'

# An existing file:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py path/to/test.py

# A subset, written to a file:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py test.py --tools ty,mypy -o /tmp/report.md

Flags: -c/--code (inline snippet), -t/--tools ty,mypy,… (subset), -o/--output (write to file), --timeout (per-checker seconds, default 180). The script exits 0 whenever it produced a report — findings live in the report, not the exit code.

Reading the result

  • The summary table shows each tool's exit code (0 = nothing flagged, non-zero = the checker reported an issue or failed to run), a verdict, and time.
  • Output formats differ per tool, so read each section, not just the table:
    • ty (Astral) and pyrefly (Meta) each have their own diagnostic style.
    • basedpyright is pyright-based and also emits extra diagnostics such as reportUnusedCallResult.
    • mypy and zuban share the same message format (zuban is mypy-compatible).
  • The checkers track their latest releases through uvx, so behaviour can shift over time — record the date when capturing results in a YouTrack issue or test.

Manual one-offs

The exact per-tool invocations, if you want to run just one by hand (note basedpyright takes the file positionally — no check subcommand):

ToolCommand
tyuvx ty check test.py
pyreflyuvx pyrefly check test.py
basedpyrightuvx basedpyright test.py
mypyuvx mypy test.py
zubanuvx zuban check test.py