README.md

Setting up Jetpack Macrobenchmarks for CI

This document describes how to use library with Github Actions; running benchmarks on Firebase Test Lab and using Google Cloud Monitoring to detect regressions, and track app performance over time.

Overview

To set up Macrobenchmarks on Firebase Test Lab as a post-submit action, you will need to configure three components:

1. Github Workflow

• Builds the app (under test) and the macrobenchmarks.
• Submits the app APK, and the macrobenchmark APK to Firebase Test Lab.

2. A Google Cloud function which automatically pulls results from Firebase Test Lab, and creates custom metrics using Google Cloud Monitoring.

3. Google Cloud Monitoring dashboards

• Set up custom dashboards
• Set up alerts

Project Setup

You will need to create a Google Account to setup your Google Cloud Platform project.

If you don't have a Firebase project for your app, go to the Firebase Console and click Create New Project to create one now. You will need ownership or edit permissions in your project. You might also want to upgrade to the Blaze plan, if you run out of free quota.

One Time Setup

Create Secrets

Create a Google Cloud IAM service account key. This allows your GitHub project to authenticate with the Google Cloud Platform. Store this service account key using GitHub Secrets.

Do not check in this key to your repository, or share with anyone outside your organization.

Create 2 secrets:

• GCP_PROJECT_ID represents your Google Cloud Platform project id.
• GCP_CREDENTIALS represents the service account key exported to JSON.

Now that you have a service account, you also need to grant the service account access to the Firebase Test Lab APIs. For more information, please refer to enabling an API for your project.

Create a Cloud Storage Bucket

Create a Google Cloud Storage bucket in your project called macrobenchmark-results. This will be the location where the results of test runs from Firebase Test Lab are stored.

Deploy the Google Cloud Function

A Google Cloud function is used to process the results of test runs from Firebase Test Lab, and process the Macrobenchmark results json file. The cloud function also captures all the quantitative metrics from the json file, and uploads them to Google Cloud Monitoring.

The cloud function uses the default service account associated with the Google Cloud Platform project. This service account should be assigned roles that give it access to Google Cloud Storage (so it has access to the created buckets and results stored by Firebase Test Lab), and Google Cloud Monitoring. For more information, please refer to enabling an API for your project.

GitHub Workflow

You can create your GitHub workflow, based on this sample.

Let's look at the sample workflow in more detail, and break it down.

name: Build and Run Macrobenchmarks
on:
  push:
    branches: [ macrobenchmark ]
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 1
    
      - name: Setup JDK
        id: setup-java
        uses: actions/setup-java@v1
        with:
          java-version: "11"
            
      - name: Build Macrobenchmark Sample
        uses: eskatos/gradle-command-action@v1
        env:
          JAVA_HOME: ${{ steps.setup-java.outputs.path }}
        with:
          arguments: build
          build-root-directory: ${{ github.workspace }}/...
          gradle-executable: ${{ github.workspace }}/.../gradlew
          wrapper-directory: ${{ github.workspace }}/.../gradle/wrapper

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@master
        with:
          project_id: ${{ secrets.GCP_PROJECT_ID }}
          service_account_key: ${{ secrets.GCP_SA_KEY }}
          export_default_credentials: true

      - name: Run Macrobenchmarks
        run: |
          gcloud firebase test android run \
            --type instrumentation \
            --app ${{ github.workspace }}/MacrobenchmarkSample/app/build/outputs/apk/benchmark/app-benchmark.apk \
            --test ${{ github.workspace }}/MacrobenchmarkSample/macrobenchmark/build/outputs/apk/benchmark/macrobenchmark-benchmark.apk \
            --device model=redfin,version=30,locale=en,orientation=portrait \
            --directories-to-pull /sdcard/Download \
            --results-bucket gs://macrobenchmark-results \
            --environment-variables additionalTestOutputDir=/sdcard/Download,no-isolated-storage=true \
            --timeout 20m

Events that trigger the workflow

name: Build and Run Macrobenchmarks
on:
  push:
    branches: [ macrobenchmark ]
  workflow_dispatch:

This part of the workflow defines the events that trigger a Macrobenchmark run. For our workflow we want to trigger a workflow when a pull request is merged or pushed to the branch macrobenchmark.

Building Macrobenchmark Tests

Here we define the jobs that are required to produce the artifacts necessary to kick off a Macrobenchmark run. We need to checkout the source, set up JDK and Gradle before we can build the Macrobenchmark test APK. Once everything is set up we can run ./gradlew build.

- name: Checkout
  uses: actions/checkout@v2
  with:
    fetch-depth: 1

- name: Setup JDK
  id: setup-java
  uses: actions/setup-java@v1
  with:
    java-version: "11"
      
- name: Build Macrobenchmark Sample
  uses: eskatos/gradle-command-action@v1
  env:
    JAVA_HOME: ${{ steps.setup-java.outputs.path }}
  with:
    arguments: build
    build-root-directory: ${{ github.workspace }}/...
    gradle-executable: ${{ github.workspace }}/.../gradlew
    wrapper-directory: ${{ github.workspace }}/.../gradle/wrapper

Kick off a Firebase Test Lab Run

Now that the APKs are built, we are ready to kick off tests using Firebase Test Lab. For this, we will use the gcloud CLI. The CLI will use the GitHub secrets that you previously created to authenticate with Google Cloud Platform. The following GitHub action sets up the gcloud CLI.

- name: Set up Cloud SDK
  uses: google-github-actions/setup-gcloud@master
  with:
    project_id: ${{ secrets.GCP_PROJECT_ID }}
    service_account_key: ${{ secrets.GCP_SA_KEY }}
    export_default_credentials: true

We can now kick execute tests using Firebase Test Lab with the following job:

- name: Run Macrobenchmarks
  run: |
    gcloud firebase test android run \
      --type instrumentation \
      --app ${{ github.workspace }}/.../benchmark/app-benchmark.apk \
      --test ${{ github.workspace }}/.../macrobenchmark-benchmark.apk \
      --device model=redfin,version=30,locale=en,orientation=portrait \
      --directories-to-pull /sdcard/Download \
      --results-bucket gs://macrobenchmark-results \
      --environment-variables additionalTestOutputDir=/sdcard/Download,no-isolated-storage=true \
      --timeout 20m

A couple of important things to keep in mind are:

• We are specifying the type of tests to run (instrumentation).

• We are specifying a device make and model that we are interested in using. Here the job wants to use redfin (a Pixel 5), API 30. It’s best to run tests against a consistent set of devices so you can compare subsequent runs and set up alerts for regressions.

• We are specifying an environment variable additionalTestOutputDir as an argument to the Macrobenchmark Library. This tells the library to store the JSON outputs and associated traces in the external storage directory. These artifacts are eventually copied to the Google Cloud Storage bucket we created.

• We are specifying a test timeout of 20 minutes. If your tests take longer, choose a longer timeout.

Google Cloud Function

The cloud function is notified every time the GitHub workflow triggers a Firebase Test Lab run. Once a test run is complete, the function processes the resulting JSON file, and uploads all the relevant metrics to Google Cloud Monitoring.

One Time Setup

Deploying the cloud function

The first step is to deploy the Google Cloud Functions to your GCP project. Make sure you have downloaded the gcloud [CLI])(https://cloud.google.com/sdk/gcloud) and the Firebase CLI setup.

Copy the cloud function from here.

Then, initialize your project using the commands below.

# Initialize gcloud CLI with a GCP admin credentials/
# Will launch a browser and complete an OAuth flow.
gcloud auth login

# Login to firebase using the CLI.
# Will launch a browser and complete an OAuth flow.
firebase login

# Choose a default GCP project
gcloud config set project <your_project_id>

Update the .firebaserc file with your project_id.

{
  "projects": {
    "default": "<your_project_id>"
  }
}

Now, deploy the Google Cloud Function.

# Will deploy the cloud functions for the selected GCP project.
firebase deploy --only functions

Setting up environment variables

The cloud function uses a couple of environment variables, to determine the package_name and the device_configurations. After you deploy the cloud function for the first time, you will need to add these environment variables.

To add the variables, go to the Google Cloud Platform Console and search for Cloud Functions. Then select a function called completionHandler and click on Edit.

Now click on Runtime, Build and Connection Settings. Update the settings to the following:

• Update Memory allocated to 4GiB.
• Update Timeout to 500 seconds.
• Add the following environment variables.

# The Macrobenchmark target app package name
# Example:
package_name = "com.example.macrobenchmark"

# The device configurations you are interested in tracking metrics for.
# This should match the device that you are testing on in the GitHub workflow.
# Example:
device_configurations = ["redfin-30-en-portrait"]

Invoking the Cloud Function

The Google Cloud Function deployed should automatically get invoked after a pull request is merged. You can track progress using the GitHub Actions tab.

Here is an example.

Once a Firebase Test Lab run is complete, the cloud function will be invoked. Under Cloud Functions, when you click on Logs you should see something like

Google Cloud Monitoring

All Macrobenchmark results are stored as custom metrics using Google Cloud Monitoring APIs. These metrics have a namespace custom.googleapis.com so you can use the Metrics Query Editor to query and create dashboards and alerts to catch regressions.

For more information on how you can use the MQL (query language) to create alerts, please refer to the following documentation.

Monitoring Dashboard

Example Metrics