Creating a GitHub Action to run automated tests

[prestoncarman]

Issues Fixed

• Fixes Add GitHub Actions to enable testing on forks of arduino_ci #199

[prestoncarman]

The GitHub Action ran on my forked repository, but it does not seems to have run on PR for Arduino CI. Are GitHub Actions turned on? I think they are on by default.

[jgfoster]

@prestoncarman, I don't think it will run until it is part of the code being merged into. So, once this PR is accepted, then subsequent PRs will be tested.

[prestoncarman]

Ah... this must be a security issue. This allows the repository owner to review the action before allowing it to run on their repository.

[ianfixes]

Is this intended as a guide to how a library owner might set up Arduino CI as a GitHub action in their own repo?

[jgfoster]

See the discussion in #199.

[prestoncarman]

The PR includes documentation for a library owner who wants to use Arduino CI in a GitHub Action.

Also, the Arduino CI tests in this repository are running as a GitHub Action in Ubuntu. Its nice that these tests will run in a fork without any additional set up. Both Travis CI and Appveyor CI need to be configured on a third party site. Issue #199 shares more reasoning.

[ianfixes]

I'm on board with this change, but not sure why the check isn't running. According to the settings, this should be all set right?

[jgfoster]

My observation (see Keypad) is that the Actions are not installed untill they have been accepted. There is a bit of a leap-of-faith, but you can go look at the Actions in the source repository and have some confidence that they will behave the same when accepted to your repository.

[prestoncarman]

After reading your comments about running in different OSes. I started looking into what GitHub Actions supports. This PR's GitHub Action now includes tests on Windows Server 2019, Ubuntu 18.04, and macOS Catalina 10.15. The version of macOS required updating the Arduino IDE to the latest version. It was nice to see that the tests pass without any other changes.

[jgfoster]

Running on macOS 10.15 will require addressing #155.

[prestoncarman]

Also addresses #185.

[jgfoster]

You can get macOS to pass now by explicitly downloading a more recent version of Arduino.app in your Action script. See what was added to Blink.

[ianfixes]

I think this just needs some steps to reflect what was added in #180

[ianfixes]

Also, this should contain a follow-up (if it can't be done already) to add the badges to the README.md

[ianfixes]

Deleted a comment that was incorrect.

I think this is missing a workspace-cleanup step: https://github.com/marketplace/actions/clean-workspace

[per1234]

add the badges to the README.md

Here's the Markdown for the badge:

[![Arduino CI](https://github.com/Arduino-CI/arduino_ci/workflows/Arduino%20CI/badge.svg)](https://github.com/Arduino-CI/arduino_ci/actions?workflow=Arduino+CI)

I think this is missing a workspace-cleanup step

Each run on the GitHub hosted runners starts in a fresh environment. Each of the matrix items here:

  runTest: 
    strategy:
      matrix:
        os: [windows, ubuntu, macos]

Is actually a separate job: runTest (windows), runTest (ubuntu), runTest (macos)

So, on a GitHub Actions hosted runner, the only time you would need to clean up is if a step in a job could be negatively effected by changes to the system made by a previous step. Normally, you would just put that in a separate job, if needed, passing files from one job to the other via workflow artifacts, or data via job outputs.

From the description of the AutoModality/action-clean action, as well as some of my own experiences, it seems that, on self-hosted runners, the environment can be left dirty from previous workflow runs. But you aren't using a self-hosted runner, so that is not relevant here.

[ianfixes]

Each run on the GitHub hosted runners starts in a fresh environment

Aah, OK. That's good to know

os: [windows, ubuntu, macos]

I definitely prefer having 3 separate badges (one per OS) surfaced in the README, even if it means 3 config files with roughly the same contents.

As far as getting the actions running in the first place, I assume my settings are correct but I don't see them firing for this PR. Does that mean they need to be in the master branch before they can start working?

[jgfoster]

Does that mean they need to be in the master branch before they can start working?

Yes. If you want to see that they pass you can see that here.

[per1234]

I definitely prefer having 3 separate badges (one per OS) surfaced in the README, even if it means 3 config files with roughly the same contents.

The badges created by GitHub Actions are per-workflow, not per-job, so it would require three workflow files to create three badges. I think there is probably a service to generate arbitrary badges which could be controlled by the job results in a single workflow though.

However, I see that prestoncarman has split the single workflow into a linux and a windows workflow since the last time I commented, so this could be done with the GitHub Actions-generated badges if a macOS version of the workflow was added

[jgfoster]

Good catch on the typo. As to macOS, it will fail until we get #155 and/or #185. GitHub Actions use macOS 10.15, which fails. Of course, maybe the failing badge would be an indication that macOS doesn't work completely, which could be useful information!

[per1234]

maybe the failing badge would be an indication that macOS doesn't work completely, which could be useful information!

This is my philosophy when setting up CI. The purpose of the check is to report any problems that might exist with the project, which means that a legitimate CI setup may result in a ❌ status. After that, work can be done on the project to try to get that lovely ✔️!

[ianfixes]

Oh wow, I posted the same screenshot twice 🤦
apologies.

[prestoncarman]

A few notes about this change:

• GitHub Actions have been set up for Windows and Linux.
• Badges have been added to the README for these two actions.
• The README has a GitHub Actions example for a library creator.
• Removed deprecated TravisCI option sudo: false.
• Each workflow has two jobs: rubocop, TestSomething, and Network. These have been separated out ensure each has a clean environment. (related to Add files needed to compile Ethernet library #180) Also to allow parallel execurtion.

The change is ready for a final review.

Side Note: It would be easy to add a workflow for macOS, but at this time I left that workflow out to allow all tests to pass. It seemed like moving to a new Arduino IDE version may be a significant change which is required for macOS (related to #185).

[per1234]

Thanks @prestoncarman!

[ianfixes]

I'm switching the base on this to the TDD branch to combine it with the macos action, and from there we'll track it on #226