CI for other platforms (Mac, more Unixes, Windows, mobile, ARM...)

[dr-orlovsky]

No description provided.

[nicbus]

trying to wrap my head around this issue

in my opinion, the CI/CD tool preference heavily depends on the choice of underlying git infrastructure, as that's where it all hooks up

so, as a first step, I'd question the basic git repository hosting and then what kind of CI setup is needed

are the LNP-BP projects going to stay on github or are there plans to move them elsewhere (e.g. gitlab.com, self-hosted, etc.)?

[dr-orlovsky]

With the current release deadlines we just have no options for doing any kind of migration for now, so we need this actions to run in order to get wallet and exchange devs starting integrating and making sure the code does build on all required platforms. It is done very simply by adding more build targes with matrix in GitHub workflow dir build.yml file, like we already have for different rust versions.

As for long-term perspective, RGB gives a way to design decentralized GitHub collaborating tool (PRs, issues) with it's recent "public state extensions" advancement; we briefly discussed with @giacomozucco the general approach to that. Combined with Prometheus it gives a way for having decentralized CI (and with Storm, for storing large BLOBs and repos). But it will take time to develop (my company, Pandora Core, is looking into that, but we are very opened to having a joint venture or child company dedicated to this project if anyone is interested)

In the mid-term, after the first release, we will be happy to have a GitLab self-hosted thing; but the problem with that is the fact that we need GitHub for network effect, not the code storage, so in self-hosted world there will be much harder to attract more collaborators on the project. So it will be rather a mirror for a worst-case scenario of GitHub censorhip.

[nicbus]

had a look at the current CI build workflow, adding missing architectures (macos, arm linux and windows) to the matrix strategy

• macos appears to build just fine
• arm linux also builds fine but requires the use of a self-hosted runner (used a test one of my own)
• windows is not as easy it seems, haven't found the right combination of deps to make it work yet (no windows host here to debug)

talking about build failures, with the matrix getting bigger it might make sense to switch to fail-fast: false (default true) so a failing build (on one architecture) doesn't cancel all other jobs

as I see it, macos can be added right away, while arm and windows require some discussion

I can open a PR to add macos and possibly change fail-fast to false, what are your thoughts?

[dr-orlovsky]

Of course you are welcome to open a PR. Replying to your questions:

• Don't know what self-hosted Mac OS actually is. I have MacBook, and we have a Unix server, but I assume neither would work for CI?
• yeah, Windows is tricky b/c of rust, ZMQ etc... I was able to make it working once, may be I can help later
• Agree with fail-fast: true

We can start with what we can and add more targets later (i.e. no windows/ARM in the first version)

BTW re ARM we already have builds happening within the scope of RGB SDK https://github.com/LNP-BP/rgb-sdk, @zoedberg knows how to make LNP/BP Core lib to compile on Android and iOS :)

[nicbus]

just a couple clarifications:

Don't know what self-hosted Mac OS actually is

self-hosted runners are only required for arm linux
macos runners are hosted by github

Agree with fail-fast: true

true is already the default, I was asking if false made sense

[dr-orlovsky]

1. How we can do self-hosted Android runners? We have a server with plenty of cores and gb of memory + ubuntu. What to do next?
2. Sure I meant false! Not true.. sorry, too tired

[nicbus]

1. How we can do self-hosted Android runners? We have a server with plenty of cores and gb of memory + ubuntu. What to do next?

self-hosted runners are required to run the workflows on ARM hardware

if the goal is to make sure that builds run fine for android and ios, cross-building from a standard hosted runner should be enough in my opinion and I can add it

[dr-orlovsky]

@dieeasy yes, the goal is that. Will appreciate PR.

Also, had a thought about platform matrix. We do not need actually to test each feature on each platform. The idea with the testing features one by one separately is that we make sure that the project builds. For platform testing we need just to use --all-features and make sure that all dependencies build. So we can have two one-dimensional matrixes instead of one two-dimensional and reduce number of parallel builds by the square root

[nicbus]

after some discussion, here's an updated view on the issue, for future reference

1. building on other platforms is required to make sure the project works for as many users as possible, so it make sense to only build all features on other platforms

   • drop the OS matrix for current jobs
   • create a new platforms job to build all features, using rust stable, on more platforms

2. fail-fast: false allows to continue other matrix jobs after one has failed, which is useful for cross-platform builds but in general helps identify more than one issue at a time

   • keep fail-fast: false on all jobs using a matrix (unavailable otherwise)

3. github hosted runners offer more than one OS version for ubuntu and macos and we want to check build works on all of them

   • drop ubuntu-latest (currently 18.04) and macos-latest (currently 10.15) versions
   • build on ubuntu-18.04, ubuntu-20.04, macos-10.15, macos-11.0

[nicbus]

with PR #141 merged I think we can now close this