Introduce repo configuration with pull request

[codecovdesign]

Problem to solve

Today, when a developer configures a repository with Codecov, the process begins on the Codecov app. However, this experience often requires users to navigate between the app, documentation, their CI provider, and their repository provider, resulting in a fragmented workflow. During user testing, we noticed that users can become overwhelmed by extensive blocks of text and may even anticipate a simpler "on switch" solution. The challenge lies in the fact that relying solely on written documentation and multiple steps is not conducive to a fast and efficient setup process.

Solution

Offer users a smoother "learn-by-doing" configuration experience, where we provide a lower-friction setup process with an intuitive "on switch" approach.

• On the repository guide page, we can include a clear call-to-action (CTA) that allows users to configure their repository with ease, starting with a pull request (PR).
• The system can then automatically generate a PR on behalf of the user, which adds the necessary YAML input to the corresponding YAML file (such as the GitHub workflow file, CircleCI YAML, etc.).
• To minimize the chances of incorrect placement and optimize for successful configuration in the majority of cases, the input is placed at the end of the YAML file.
• To ensure visibility and prevent breaking changes merging, the PR can be labeled as "Work in Progress" (WIP), and a note can be added to the description, guiding reviewers to review the input placement before merging the changes.

Open questions

• What is the expected time and effort investment for implementation work?
• What ux and/or configuration challenges might developers encounter during this process?
• Can we track the frequency of YAML file edits after setting up a PR? Is it possible to measure the time between PR creation and PR merge?

Prototype:

Screen.Recording.2023-09-07.at.5.37.47.AM.mov

Tasks

Preview Give feedback

1. Investigate requirements for creating repo config PR #155
   investigation requirements-gathering +2
   
2. Create framework for adding Codecov config via PR #150
   deprioritized enhancement +2
   
3. Codecov config via PR for Github Actions #151
   deprioritized enhancement +2
4. Codecov config via PR for Circleci #152
   deprioritized enhancement +2

[nicholas-codecov]

So imo this ticket should be moved to probably codecov-api as 99% of the work will need to be done there, and a simple API call will need to be done in Gazebo.

This is definitely something we're gonna have to learn how to tackle if we move over to GitHub issues as we do not have a monorepo where everything lives and we can just throw things in there.

[codecovdesign]

ticket should be moved to probably codecov-api

@nicholas-codecov makes sense, I'll move it over there. As the issue creator, there is a "transfer issue" action that sends it to another repo. Though it requires write access to do so; I'll make the request and transfer it over there.

In terms of feedback about the issue and implementation investigation; should we also rope in Scott? Or were you going to comment on that separately?

[nicholas-codecov]

In terms of feedback about the issue and implementation investigation; should we also rope in Scott? Or were you going to comment on that separately?

I don't have much knowledge on the API side of things, so I would recommend @scott-codecov, @adrian-codecov, or @RulaKhaled, they should be able to help out with the estimation and implementation on the API side of things.

Frontend wise, this is a fairly simple exercise as we're just making a mutation request on an event, then have the API return the URL for the pull request then direct the user to that location once the API response is successful.

[trent-codecov]

Moved this to platform as 95% of the work will be done there.

[giovanni-guidini]

Just throwing this out there cause it doesn't exactly match the design that we have on the ticket. I discussed this both with @dana-yaish and @adrian-codecov , and we kinda come to the agreement that (1) it's hard to produce a useful workflow file, but we think it can be done AND (2) producing a codecov.yml is very nice and useful.

We kinda envisioned a part of the UI where you have forms or buttons or toggles or things of the nature that you can edit and see a real-time template example of what your PR comment will look like with those settings, for example.

Also things like flags and components and what not.
Then we create a codecov.yml that reflects those choices and open a PR with it

[giovanni-guidini]

Allow me to express my opinions on this question

• What is the expected time and effort investment for implementation work?

TL;DR
From an implementation perspective we should do this incrementally. We should select 1-3 CI providers to be the supported ones (at least to start). I imagine GH Actions and CircleCI.

---

Long version

• What is the expected time and effort investment for implementation work?

Depends on the level of accuracy you want the end result to have. That is, assuming no mods on the part of the user, how useful do you want the proposed PR to be?

There's a lot of complexity that goes into modifying CI files automatically, for example:

• There are many different CI providers, each with its details in terms of syntax and file structure that need to be accounted for. We can't support them all.
• Users (specially large users) have many different CI files, each doing something different that may or may not have to do with running tests.
• Inevitably, specially in the beginning of this feature, the user will have to make changes in the PR. Don't expect that to go away forever right from the start.

So we should probably take on this effort incrementally. Some example stages:

1. Have a set of templates, one per CI provider we want to support. Each template creates a new workflow with the steps (I) download project code; (II) placeholder for building the project; (III) placeholder for running tests; (IV) upload to codecov. We might make educated guesses in the placeholders, but make it very clear that the user needs to edit those steps to match their needs. When the user clicks the action button there will be a dropdown to choose the CI used by them.

2. Same as 1, but we try to detect the user's CI provider automatically by checking their repo file structure (looking for some specific folder or file).

3. Here we start parsing the user's existing CI files to try and figure out where they test their code. Then we add the upload to Codecov step below the test step. This is a complex step, but it's where we propose really meaningful changes with minimal user interaction.

So... we start with GitHub Actions and CircleCI I assume?

[aj-codecov]

@giovanni-guidini Thank you for the write up!
I agree the complexity of this is more than we anticipated - based on our conversation in Slack and the above, this initiative sounds like:

• GHActions first and then CircleCI second
• Figure out which file(s) to update - even if we have a user tell us what the file (or files) is to start, that's fine.
• Figure out how to parse CI steps
• Figure out how to determine which step is the testing step
• Insert (CLI) upload step into the file after the testing step

I am comfortable with less than 100% accuracy, my hunch is that most sane engineers aren't going to just blindly push merge on something without at least giving it a glance first. In this case I would strive to not let perfect be the enemy of good enough.

I'm not sure I fully understand the template piece - are you able to give me a little more info there?

[giovanni-guidini]

A template is some generic configuration that we could use as the changes to propose.
We propose that in a new file instead of trying to modify any of the user's existing files.
For example, for GH Actions (and a python project) we could propose

name: Build-Test-Coverage

on:
  pull_request:

jobs:
  build-test-coverage:
    runs-on: ubuntu-latest
    steps:
    # Checkout code
    - uses: actions/checkout@v3
    # Install Python
    - uses: actions/setup-python@v2
      with:
        python-version: '3.10.10'
    # EDITME - build project
    - name: Build project
      run: pip install -r requirements.txt
    # EDITME - ruin tests and generate coverage reports
    - name: Test with coverage
      run: python -m pytest --cov=./
    # Upload coverage reports to codecov
    - name: Upload to codecov
      run: |
        pip install codecov-cli
        codecovcli upload-process --flag unit --token=${{ secrets.CODECOV_TOKEN }} --fail-on-error

[katia-sentry]

[Applications] Deprioritized for Q3