run_tests

Run Tests Skill

This skill provides comprehensive instructions for executing tests within the AndroidX repository. It covers module discovery, unit testing, instrumentation testing on connected devices, and remote testing via Firebase Test Lab (FTL).

1. How to Find and Run a Specific Test

If you have a specific failing test (e.g., BasicTextFieldTest#longText_doesNotCrash_singleLine) and want to execute it:

1. Find the Test File: Use code search, find_declaration, or find_files (e.g., search for BasicTextFieldTest.kt).
2. Identify the Module: The file path determines the Gradle project (e.g., compose/foundation/foundation/src/... belongs to :compose:foundation:foundation).
3. Identify Test Type:
   • If the test is in a test, androidHostTest, or jvmTest folder, it is a Unit Test.
   • If the test is in androidTest or androidDeviceTest, it is an Instrumentation Test.
4. Identify Module Type: Check if the module is KMP or standard Android (see details below) by inspecting build.gradle for androidXMultiplatform { or running ./gradlew <module>:tasks | grep "test".
5. Formulate the Task: Select the proper Gradle task (e.g., testAndroidHostTest vs test, connectedAndroidDeviceTest vs connectedAndroidTest).
6. Filter by Class/Method:
   • Class: Append filtering arguments. E.g., for instrumentation: -Pandroid.testInstrumentationRunnerArguments.class=androidx...BasicTextFieldTest. For unit tests: --tests "androidx...BasicTextFieldTest".
   • Method: For instrumentation, append #<method_name> to the class name (e.g., ...BasicTextFieldTest#longText_doesNotCrash_singleLine). For unit tests, append .<method_name> to the class name in --tests (e.g., --tests "...BasicTextFieldTest.longText_doesNotCrash_singleLine").

2. Module Discovery

Before running tests, you must identify the Gradle project name (module) associated with the code you are testing.

• Mapping a path to a project: Most directories in this repository are Gradle projects. Use the directory structure as a guide (e.g., appcompat/appcompat corresponds to :appcompat:appcompat).
• Verification: Run ./gradlew projects to see a full list of projects. Because the output is very long, consider chaining a grep command to filter the output if you are looking for a specific module (e.g., ./gradlew projects | grep appcompat).
• Module Type: Be aware if the module is a standard Android library (like appcompat) or a Kotlin Multiplatform (KMP) library (like compose:foundation:foundation). Task names differ significantly. To determine if a module is KMP, you can run ./gradlew <project-name>:tasks | grep "test": if you see tasks like testAndroidHostTest or jvmTest, it is a KMP module. If you see standard test and connectedAndroidTest tasks, it is a standard Android module. Alternatively, inspect its build.gradle file for androidXMultiplatform {.

3. Unit Testing (JVM)

Unit tests run on the host machine and are the fastest way to verify logic.

• Standard Android Modules:

  ./gradlew <project-name>:test
  

• KMP Modules:

  ./gradlew <project-name>:testAndroidHostTest
  ./gradlew <project-name>:jvmStubsTest
  

• Filtering: Use the --tests filter. You can also append .<method_name> for specific methods.

  ./gradlew <project-name>:test --tests "androidx.example.MyTest"
  ./gradlew <project-name>:testAndroidHostTest --tests "androidx.example.MyTest"
  ./gradlew <project-name>:test --tests "androidx.example.MyTest.myMethod"
  

4. Instrumentation Testing (Connected Devices)

These tests run on a physical device or emulator connected via ADB.

• Standard Android Modules:

  ./gradlew <project-name>:connectedAndroidTest
  

• KMP Modules:

  ./gradlew <project-name>:connectedAndroidDeviceTest
  

• Filtering: Use the android.testInstrumentationRunnerArguments.class property. For specific methods, append #<method_name>.

  ./gradlew <project-name>:connectedAndroidTest \
      -Pandroid.testInstrumentationRunnerArguments.class=androidx.example.MyTest
  ./gradlew <project-name>:connectedAndroidDeviceTest \
      -Pandroid.testInstrumentationRunnerArguments.class=androidx.example.MyTest#myMethod
  

5. Remote Testing (Firebase Test Lab)

AndroidX provides specialized tasks for running instrumentation tests on Firebase Test Lab (FTL). FTL is not used for local unit tests. The task suffix changes based on module type.

• Standard Android Modules: ftl<device><api>releaseAndroidTest
• KMP Modules: ftl<device><api>androidDeviceTest
• Listing Available Combinations: To see the full list of available FTL tasks for a specific project, you can run:

  ./gradlew <project-name>:tasks --all | grep ftl
  

• Common Device/API combinations:
  • mediumphoneapi36
  • mediumphoneapi35
  • mediumphoneapi34
  • mediumphoneapi30
  • nexus5api23
• Example (KMP):

  ./gradlew <project-name>:ftlmediumphoneapi35androidDeviceTest --className="androidx.example.MyTest"
  

5.1. Reproducing Flakes on FTL

To verify a flaky test, you can run it multiple times on Firebase Test Lab using the ftlOnApis task variant.

1. Parameterize the Test: To run a test N times, temporarily modify the test class to be parameterized:
   • Add @RunWith(Parameterized::class) to the class.
   • Update the constructor to accept a repetition parameter: class MyTest(private val repetition: Int).
   • Add the companion object for data generation:

     import org.junit.runners.Parameterized
     
     companion object {
         private const val RUNS = 100 // Adjust as needed
         @JvmStatic
         @Parameterized.Parameters
         fun data(): Array<Int> = Array(RUNS) { 0 }
     }
     

   • Note: If there are many tests in the class but you only want to focus on one, comment out the @Test annotation on the irrelevant ones. DO NOT REPLACE ANY LINES OF CODE OR VARIABLES NAMES in the test class aside from making it parameterized. Test class name should remain the same.
2. Run Locally (Optional): Verify it runs a few times locally before deploying to FTL. E.g., for KMP:

   ./gradlew <project-name>:connectedAndroidDeviceTest -Pandroid.testInstrumentationRunnerArguments.class=androidx.example.MyTest
   

3. Run on FTL: Use the ftlOnApis task. Specify the API level(s) with --api and add a longer timeout --testTimeout=1h (if the API level is not provided, you must ask the user for it).

   # Example for KMP (append 'releaseAndroidTest' instead of 'androidDeviceTest' for standard Android)
   ./gradlew <project-name>:ftlOnApisandroidDeviceTest --testTimeout=1h --api 28 --api 30 --className=androidx.example.MyTest
   

6. Screenshot / Visual Tests

Screenshot tests (e.g., using AndroidXScreenshotTestRule, which extends the core ScreenshotTestRule.kt) compare the rendered UI against approved "golden" reference images to detect visual regressions.

• Emulator Requirement: Screenshot tests must be executed on a specific emulator configuration to ensure consistent rendering.

  • Verify the required configuration: If the required emulator configuration is not found in the test class or local documentation, fall back to verifying it by reading ScreenshotTestRule.kt. Check the ScreenshotTestStatement class for the required API level (typically API 35) and device model.
  • Locally, a Medium Phone API 35 emulator is typically required.

• Managing the Emulator:

  • Recommend the use of the android-cli tool to manage emulators.
  • Check the environment: Check if the user has android-cli installed and if they have a compatible emulator (e.g., Medium Phone API 35).
    • To list available virtual devices:

      android emulator list
      

  • If the user does not have android-cli installed, ask them if they want to install it.
  • If they have a compatible emulator configured (e.g., Medium Phone API 35), it is safe to use.
  • If they do not have a compatible emulator, or if they prefer not to use a local emulator, ask the user if they would prefer to run the tests via Firebase Test Lab (FTL) (see Section 5) or if they want to set up a local emulator.
  • To launch the emulator using android-cli:

    android emulator start <device_name>
    

• Running the Tests: Once the emulator is running (or if using FTL), execute the screenshot tests. If running locally, they are executed as standard instrumentation tests using Gradle (see Section 4).

• Troubleshooting Missing Screenshots:

  • If the tests run but screenshots do not show up or cannot be pulled, follow the code in the test and ScreenshotTestRule.kt to understand how they are configured and where they are being saved on the device.

• Updating Screenshot Goldens: If a screenshot test fails due to intentional UI changes, you must update the golden reference images in the support-goldens repository (which is checked out as a sibling golden directory to frameworks/support).

  1. Run Tests (Trigger Failure): Execute the screenshot tests locally on the required emulator (or FTL). If you are introducing a new screenshot or expecting a change, the test must fail to write output files. If the test passes but you want to recreate the golden, delete the local golden image first to trigger a MISSING_REFERENCE failure.

  2. Pull Test Outputs: When a test fails, ScreenshotTestRule writes the actual screenshot, expected screenshot, and a mapping .textproto file to the device.

     • To pull these output files to your workstation, run:

       adb pull /sdcard/Android/data/<test_package_name>/cache/androidx_screenshots/
       

       (Note: <test_package_name> is the identifier of the test APK, e.g., androidx.compose.material.test)

  3. Map and Copy to Goldens Repo: Locate the generated *_diffResult_goldResult.textproto file for the failed test. It explicitly maps the actual screenshot filename on the device to its repo destination. For example:

     • image_location_test: "androidx.compose.material.CheckboxScreenshotTest_checkBoxTest_checked_emulator_b294d33ecd4f4764_actual.png"
     • image_location_golden: "compose/material/material/checkbox_checked_emulator.png"

     Rename the pulled *_actual.png image to match the filename in image_location_golden (e.g., checkbox_checked_emulator.png) and copy it to its location in the sibling golden project repository (e.g., ../../golden/compose/material/material/).

  4. Submit Linked CLs via Shared Topic: To submit your code changes and golden image updates together (ensuring presubmit runs them as a unit), upload them to Gerrit using a shared topic:

     • Code CL (under frameworks/support): repo upload --cbr -t <my_shared_topic>
     • Golden CL (under golden/): repo upload --cbr -t <my_shared_topic>

     [!NOTE] Some modules may use custom wrapper rules (such as RemoteScreenshotTestRule in wear/compose/remote/remote-material3). These custom rules delegate to the same core ScreenshotTestRule library under the hood. They write their outputs to the same androidx_screenshots cache directory on the device and generate identical .textproto mapping files.