OpenParable
Skip to content
← Back to skills

A/B Test Statistical Analyzer

Business

A/B Test Statistical Analyzer is an Agent Skill for business published in the a5c-ai/babysitter repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Performs statistical analysis for A/B testing experiments

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/a5c-ai/babysitter/blob/HEAD/library/specializations/data-engineering-analytics/skills/ab-test-statistical-analyzer/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/a-b-test-statistical-analyzer, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

A/B Test Statistical Analyzer

Overview

Performs statistical analysis for A/B testing experiments. This skill provides rigorous statistical methods to determine experiment validity and significance.

Capabilities

  • Sample size calculation
  • Statistical significance testing
  • Bayesian analysis
  • Sequential testing
  • Multi-armed bandit analysis
  • Segment analysis
  • Novelty/primacy effect detection
  • SRM (Sample Ratio Mismatch) detection
  • Confidence interval calculation
  • Power analysis

Input Schema

{
  "experimentData": {
    "control": "object",
    "variants": ["object"]
  },
  "metrics": [{
    "name": "string",
    "type": "conversion|continuous|ratio"
  }],
  "analysisType": "frequentist|bayesian|sequential"
}

Output Schema

{
  "results": [{
    "metric": "string",
    "controlValue": "number",
    "variantValues": ["number"],
    "pValue": "number",
    "confidenceInterval": "object",
    "significant": "boolean"
  }],
  "srmCheck": "object",
  "recommendation": "string"
}

Target Processes

  • A/B Testing Pipeline
  • Feature Store Setup

Usage Guidelines

  1. Provide complete experiment data for control and variants
  2. Define metrics with appropriate types
  3. Select analysis methodology based on requirements
  4. Review SRM checks before interpreting results

Best Practices

  • Always check for sample ratio mismatch before analysis
  • Use appropriate statistical tests for metric types
  • Consider practical significance alongside statistical significance
  • Account for multiple comparison corrections
  • Document assumptions and limitations