OpenParable
Skip to content
← Back to skills

Java Development

Development
java-development

Java Development is an Agent Skill for development published in the apache/beam repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Guides Java SDK development in Apache Beam, including building, testing, running examples, and understanding the project structure. Use when working with Java code in sdks/java/, runners/, or examples/java/.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/apache/beam/blob/HEAD/.agent/skills/java-development/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/java-development, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Java Development in Apache Beam

Project Structure

Key Directories

  • sdks/java/core - Core Java SDK (PCollection, PTransform, Pipeline)
  • sdks/java/harness - SDK harness (container entrypoint)
  • sdks/java/io/ - I/O connectors (51+ connectors including BigQuery, Kafka, JDBC, etc.)
  • sdks/java/extensions/ - Extensions (SQL, ML, protobuf, etc.)
  • runners/ - Runner implementations:
    • runners/direct-java - Direct Runner (local execution)
    • runners/flink/ - Flink Runner
    • runners/spark/ - Spark Runner
    • runners/google-cloud-dataflow-java/ - Dataflow Runner
  • examples/java/ - Java examples including WordCount

Build System

Apache Beam uses Gradle with a custom BeamModulePlugin. Every Java project's build.gradle starts with:

apply plugin: 'org.apache.beam.module'
applyJavaNature( ... )

Common Commands

Build Commands

# Compile a specific project
./gradlew -p sdks/java/core compileJava

# Build a project (compile + tests)
./gradlew :sdks:java:harness:build

# Run WordCount example
./gradlew :examples:java:wordCount

Running Unit Tests

# Run all tests in a project
./gradlew :sdks:java:harness:test

# Run a specific test class
./gradlew :sdks:java:harness:test --tests org.apache.beam.fn.harness.CachesTest

# Run tests matching a pattern
./gradlew :sdks:java:harness:test --tests *CachesTest

# Run a specific test method
./gradlew :sdks:java:harness:test --tests *CachesTest.testClearableCache

Running Integration Tests

Integration tests have filenames ending in IT.java and use TestPipeline.

# Run I/O integration tests on Direct Runner
./gradlew :sdks:java:io:google-cloud-platform:integrationTest

# Run with custom GCP project
./gradlew :sdks:java:io:google-cloud-platform:integrationTest \
  -PgcpProject=<project> -PgcpTempRoot=gs://<bucket>/path

# Run on Dataflow Runner
./gradlew :runners:google-cloud-dataflow-java:examplesJavaRunnerV2IntegrationTest \
  -PdisableSpotlessCheck=true -PdisableCheckStyle=true -PskipCheckerFramework \
  -PgcpProject=<project> -PgcpRegion=us-central1 -PgcsTempRoot=gs://<bucket>/tmp

Code Formatting

# Format Java code
./gradlew spotlessApply

Writing Integration Tests

@Rule public TestPipeline pipeline = TestPipeline.create();

@Test
public void testSomething() {
  pipeline.apply(...);
  pipeline.run().waitUntilFinish();
}

Set pipeline options via -DbeamTestPipelineOptions='[...]':

-DbeamTestPipelineOptions='["--runner=TestDataflowRunner","--project=myproject","--region=us-central1","--stagingLocation=gs://bucket/path"]'

Using Modified Beam Code

Publish to Maven Local

# Publish a specific module
./gradlew -Ppublishing -p sdks/java/io/kafka publishToMavenLocal

# Publish all modules
./gradlew -Ppublishing publishToMavenLocal

Building SDK Container

# Build Java SDK container (for Runner v2)
./gradlew :sdks:java:container:java11:docker

# Tag and push
docker tag apache/beam_java11_sdk:2.XX.0.dev \
  "us-docker.pkg.dev/your-project/beam/beam_java11_sdk:custom"
docker push "us-docker.pkg.dev/your-project/beam/beam_java11_sdk:custom"

Building Dataflow Worker Jar

./gradlew :runners:google-cloud-dataflow-java:worker:shadowJar

Test Naming Conventions

  • Unit tests: *Test.java
  • Integration tests: *IT.java

JUnit Report Location

After running tests, find HTML reports at: <project>/build/reports/tests/test/index.html

IDE Setup (IntelliJ)

  1. Open /beam (the repository root, NOT sdks/java)
  2. Wait for indexing to complete
  3. Find examples/java/build.gradle and click Run next to wordCount task to verify setup