OpenParable
Skip to content
← Back to skills

Causal Detective

Testing & Quality
causal-detective

Causal Detective is an Agent Skill for testing & quality published in the pymc-labs/CausalPy repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Challenge causal claims through structured threat assessment, counterfactual reasoning, and CausalPy falsification checks. Use when validating whether a causal effect is real or when the user asks "is this effect real?" or "can I trust this result?"

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/pymc-labs/CausalPy/blob/HEAD/causalpy/skills/causal-detective/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/causal-detective, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Causal Detective

Use this skill to stress-test a causal claim before trusting or communicating it. The workflow combines qualitative causal reasoning with CausalPy sensitivity and diagnostic checks.

Investigation Workflow

  1. Frame the claim: state the treatment, outcome, estimand, fitted method, and proxy counterfactual.
  2. Evaluate the counterfactual: ask how close the proxy is to the ideal parallel-world comparison.
  3. Hunt for alternatives: identify concrete confounders, selection effects, reverse causation, measurement issues, common shocks, and external-validity limits.
  4. Map threats to tests: choose CausalPy checks that would be expected to fail if each alternative explanation were true.
  5. Interpret the evidence: separate threats ruled out by the data from threats that remain untested or unresolved.
  6. Communicate the verdict: use cautious language that reflects the strength of the causal evidence rather than treating a fitted effect as proof.

Core Questions

  • What is the counterfactual, and how far is it from the ideal comparison?
  • Is there something else that could affect both treatment assignment and the outcome?
  • Could the outcome be influencing the treatment, or could the timing be ambiguous?
  • If bias exists, would it inflate the effect, shrink it, or make the direction unclear?
  • Can this result be generalized across populations, time periods, geographies, or treatment scales?

CausalPy Checks

Alternative explanationUseful check
Effect existed before treatmentcp.checks.PreTreatmentPlaceboCheck
Model detects fake effects in untreated periodscp.checks.PlaceboInTime
Result depends on one donor or observationcp.checks.LeaveOneOut
Common shocks affect untreated units toocp.checks.PlaceboInSpace
Effect appears on outcomes that should not movecp.checks.OutcomeFalsification
RD/RK estimate depends on bandwidthcp.checks.BandwidthSensitivity
Bayesian result depends on prior choicescp.checks.PriorSensitivity
RD threshold may be manipulatedcp.checks.McCraryDensityTest
Synthetic control extrapolates beyond donorscp.checks.ConvexHullCheck
Effect fades, reverses, or is window-specificcp.checks.PersistenceCheck

Output Pattern

Return:

  • Claim: one sentence.
  • Counterfactual quality: good, moderate, or poor with reasoning.
  • Threat inventory: named threats with severity, bias direction, and whether each is testable.
  • Tests to run or tests run: CausalPy check names and the alternative each test targets.
  • Verdict: strong, moderate, suggestive but inconclusive, weak, or likely non-causal.
  • What would change the verdict: specific additional data, checks, or domain evidence.

References