Apache Spark Optimizer

DevOps & Security

Apache Spark Optimizer is an Agent Skill for devops & security published in the a5c-ai/babysitter repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Analyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization

View on GitHub ↗

QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

Open your project in Codex.
Copy and paste the prompt below.
Review files and risks before approving any action.

Prompt to paste

Inspect the Agent Skill at https://github.com/a5c-ai/babysitter/blob/HEAD/library/specializations/data-engineering-analytics/skills/apache-spark-optimizer/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/apache-spark-optimizer, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Apache Spark Optimizer

Overview

Analyzes and optimizes Apache Spark jobs for performance, cost, and resource utilization. This skill provides deep expertise in Spark execution plans, partitioning strategies, and resource configuration to maximize efficiency.

Capabilities

Spark execution plan analysis and optimization
Partition strategy recommendations
Shuffle reduction techniques
Memory and executor configuration tuning
Catalyst optimizer hints generation
Data skew detection and mitigation
Broadcast join optimization
Caching strategy recommendations

Input Schema

{
  "sparkCode": "string",
  "clusterConfig": "object",
  "executionMetrics": "object",
  "dataCharacteristics": {
    "volumeGB": "number",
    "partitionCount": "number",
    "skewFactor": "number"
  }
}

Output Schema

{
  "optimizedCode": "string",
  "recommendations": ["string"],
  "expectedImprovement": {
    "executionTime": "percentage",
    "resourceUsage": "percentage",
    "cost": "percentage"
  },
  "configChanges": "object"
}

Target Processes

ETL/ELT Pipeline
Streaming Pipeline
Feature Store Setup
Pipeline Migration

Usage Guidelines

Provide the Spark code or job definition for analysis
Include cluster configuration details (executors, memory, cores)
Share execution metrics if available (from Spark UI or history server)
Describe data characteristics including volume, partitions, and known skew

Best Practices

Always analyze execution plans before and after optimization
Test optimizations on representative data samples first
Monitor resource utilization during optimization validation
Document configuration changes for reproducibility
Consider cost implications alongside performance gains