Multi-repo using environment-variables ability

[ibrah3m]

Is your feature request related to a problem? Please describe

Inside .gitpod.yml
The multi-repo parameter does not support any type of environment variables now

Describe the behavior you'd like

I recommend providing an option to make us able to pass/get the main configuration from
environment-variables like the example below

mainConfiguration: $MY_VAR // should already exist or be passed by the URL

[geropl]

@ibrah3m Interesting. Could you provide an example of how you'd want to use this concretely? 🤔

[ibrah3m]

@ibrah3m Interesting. Could you provide an example of how you'd want to use this concretely? 🤔

Sure, instead of injecting this env var in a shell script to clone the repo and rearrange the directories like the example below uses a shell script to achieve that
https://gitpod.io/#PROJECT_REPO=https%3A%2F%2Fgithub.com%2Fibrah3m%2Flaravel-vue-test-project.git,PROJECT_TYPE=laravel/https://github.com/ibrah3m/IQdevs

as you can see I'm passing the repo link as an encoded URL to pass it again in the script from the init parameter inside YAML in this repo https://github.com/ibrah3m/IQdevs

To make it short we can add one line inside. gitpod.yml by using both Multi-repo & environment-variables
just like this one
mainConfiguration: $PROJECT_REPO
please let me know if there is anything need to clarify and ill be happy If I can contribute to achieving that.

[david-bakin]

I'd like to see this more generally. For example, in the additionalRepositories tag - and in fact throughout .gitpod.yml.

For additionalRepositories - I'm writing a blog post on this (will update here later) but basically consider the case where you're working on a project which does not have a .gitpod.yml (or .gitpod.Dockerfile, or anything else you want) checked in anywhere, and you're not likely to get it to happen.

For example, some open source project you'd like to contribute to. In my case, Bitcoin Core.

It is not ideal at all to check in those files to your own fork because they have to be in whatever branch you're working on - so you have to keep them there in each and every short-lived branch you want to create/use.

Instead, I'm using the great multi-repo feature. Gitpod configuration files and anything else I want to have over there in my own repo, and it points to my fork of the repo I'm working with. Part of the init start task is to copy my scripts over there. (But not .gitpod.*, that's not needed.) Which works well.

Trouble is, I'd like to share my Bitcoin Core Gitpod setup - but right now since I can only embed a hard string for the additionalRepository there's an extra step for everyone who wants to use it: They've got to go edit the .gitpod.yml to point to their own fork. Which means they've got to fork my Bitcoin Core Gitpod setup repo also.

Expanding project-specific environment variables in additionalRepository would fix that. They could simply launch my repo off of GitLab after having set the project-specific env var to point to their fork of Bitcoin Core. (Or put it in the url I guess.)

N.B.: This following idea might actually already work but I haven't tested it yet, because dockerfiles support "environment replacement". I'll update the next paragraph once I test this, if it works.)

And there are other uses for project-specific environment var expansion elsewhere too, for example, in .gitpod.Dockerfile I'd like to make the Docker tag pulled be a project-specific environment var. Because I'm pulling a Docker image that I create myself from gitpod/workspace-full, it has lots of additional stuff I want yet it is fairly static so no need to do it with the "normal" .gitpod.Dockerfile which rebuilds too often (when it doesn't need to) yet there seems to be a problem where Gitpod, when using a custom docker image, doesn't check the digest of the named base image (FROM) to see if it has changed - so it can build with an out-of-date base image. At least, this was happening when I was overwriting that base image but using :latest. So now I have to use commit SHAs as the tag for my base image, and that means I have to edit .gitpod.Dockerfile each time the base image changes. I'd rather put that in a project-specific env var. (Actually, I'd rather Gitpod looked at the digest of the named base image to see if it needs to pull it again...)

[ibrah3m]

HI thanks for your contribute.
For example, some open source project you'd like to contribute to. In my case, Bitcoin Core.
This is actually what I'm doing to achieve if you see the example bellow doesn't include any gitpod and docker files,

https://gitpod.io/#PROJECT_REPO=https%3A%2F%2Fgithub.com%2Fgothinkster%2Flaravel-realworld-example-app,PROJECT_TYPE=laravel/https://github.com/ibrah3m/IQdevs
its auto merged from the 2nd repo for deploy the project itself i used some package called DDEV its manage the connection of the DB and servethe project.

so my solution already works and desgined for that purpose , but until now its only for web developers and the frameworks that ddev support it and needs to add more options like running custom commands and stuff like that in the next versions

[geropl]

Thank you @ibrah3m and @david-bakin for sharing.

Do I understand correctly that you're using the multi-repo feature to emulate ".gitpod.yml injection" of sorts?
Are you aware of definitely-gp? We're going to deprecate it (because it's badly structured, and has maintenance issues), but I mentioned it here because it sounds like a similar use-case: share a Gitpod setup without being able to push it to the original repo.

[ibrah3m]

Thank you @ibrah3m and @david-bakin for sharing.

Do I understand correctly that you're using the multi-repo feature to emulate ".gitpod.yml injection" of sorts?
Are you aware of definitely-gp? We're going to deprecate it (because it's badly structured, and has maintenance issues), but I mentioned it here because it sounds like a similar use-case: share a Gitpod setup without being able to push it to the original repo.

it's kinda similar but not exactly the same , I think in that case they need to pass 2 arguments #1 is the Gitpod.yml #2 the repo

this my face lack of resources in case they need shell script / custom docker image

were in my case you have 2 repos , you can put everything and after the prebuild the origin of the git will be replaced automatically to the targeted project
hope that's explain everything.

[david-bakin]

@geropl - yes, injection of .gitpod.yml and also .gitpod.Dockerfile. It is kind of like definitely-gp - except I'm not sure how you're expected to use those thing. Maintain them separately but copy them into your forked repo?

Here, by the way, is my main repo that I'm currently doing this with: https://gitlab.com/bakins-bits/gitpod-setup-for-bitcoin-core-dev

Works great! Except I can't directly trigger prebuilds of the main repo when committing/pushing to the sub repo (because the sub repo doesn't _point back to the main one because the .gitpod.yml is not committed there). I'm discussing that in comments here: #9183 (comment)

A problem with the definitely-gp approach, as I understand it, is that the basic gitpod approach (and open source generally) is to do all work in a (very) temporary branch, resulting in a PR then a merge. This means that mostly you want to do the prebuild in the temporary branch you're working in at that moment. You can do that by editing .gitpod.yml to checkout the branch you're interested in and then building it - but that's inconvenient, especially if you're working on multiple PRs at a time. And, unfortunately, the processing of .gitpod.yml doesn't do env var expansion, which I pointed out above, which would fix this issue (I could pass the env var as part of the URL, or if the "temp" branch was longer lived than usual, just set it as a project env var).

Unless you can suggest a way to do it?

[geropl]

were in my case you have 2 repos , you can put everything and after the prebuild the origin of the git will be replaced automatically to the targeted project

@geropl - yes, injection of .gitpod.yml and also .gitpod.Dockerfile. It is kind of like definitely-gp

@ibrah3m I'm still struggling to understand what this is more about: Is it:

1. injecting/overlaying a certain config (.gitpod.yml + Dockerfile) into a repository you don't have control over
2. or sharing a certain config between multiple repos: that's what multi-repo support is for. Except that atm, it's limited to a 1-n connection, and the only way we can make prebuilds work.

Honestly I don't think we'll introduce something like variables in .gitpod.yml, because it opens up too many degrees of freedom, and it still feels we could solve all of this in easier ways. 🤔

[ibrah3m]

@geropl

> 1. injecting/overlaying a certain config (.gitpod.yml + Dockerfile) into a repository you don't have control over

Hi again, for me we don't need to Dockerfile but the main goal was this one, yes to be able to work on a public repository which not prepared to run in both gitpod and ddev (doesn't contain any yml & docker & ddev configs), we achieve that by mix the public repo with repo we already made contains all config needed such as https://github.com/ibrah3m/IQdevs

right now we use the environment variables and we pass the URL of the public repo into the shell file https://github.com/ibrah3m/IQdevs/blob/main/.beforebuild.sh
which is responsible to clone and merge the public one + our config repo.
this way is a bit longer and to minimize it prefers to use the way we mentioned above it will be more efficient

mainConfiguration: $MY_VAR // should already exist or be passed by the URL

the difference is the definitely-gp pass only the (.gitpod.yml + Dockerfile) directly to the public repo

[ibrah3m]

>2- or sharing a certain config between multiple repos: that's what multi-repo support is for. Except that atm, it's limited to a 1-n connection, and the only way we can make prebuilds work.

that's actually what we do we need to merge our configs in the public repo as I explained above . :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[david-bakin]

@geropl - any further ideas on how to solve this use case "in easier ways"? My use case in particular is: Working with bitcoin core, which I can't add a .gitpod.yml or custom dockerfile to - so I created a repo which sets up a multi repo with it and with a fork of bitcoin core. Now that repo must of necessity point to the fork in the additionalRepositories key of .gitpod.yml, as seen here - https://gitlab.com/bakins-bits/gitpod-setup-for-bitcoin-core-dev/-/blob/main/.gitpod.yml#L9 .

As time passes I update this repo so it has the latest dev tools including additional bitcoin-specific tools.

Now this should work for anyone wanted to work with bitcoin core, and I'd like to encourage it's adoption by other people wishing to work with bitcoin core. (And you would like to encourage its adoption too because it uses gitpod.io!)

If such a dev wants to use the current setup he has to clone not only bitcoin core but also my "main" repo here. And then change the .gitpod.yml to point to his fork. And then, as time passes, remember occasionally to sync his fork of my "main" repo to get the latest and greatest stuff. Kind of tedious, and all overhead, when all he really wants to do is work seamlessly with bitcoin core, using the awesome gitpod.io cloud dev facility.

If instead the additionalRepositories key could be configured from outside the workspace, e.g., by passing env vars in through the URL and then expanding them in .gitpod.yml - this could all be seamless to such a dev. He'd just have a simple instruction on how to build a url pointing gitpod to my "main" repo and his bitcoin core fork, which he would then bookmark to keep it safe, and from them on he can launch his own bitcoin core workspace on gitpod seamlessly.

Env var expansion in .gitpod.yml seem good to me but you had concerns that it opens up too much. So I'm not fixated on it. It's just what seemed available.

Note that I also have a reason to want some kind of external customization of the image tag too: so I can provide the base image for my workspace externally. But this may be due to a gitpod.io bug: I can't use the latest tag because it seems that gitpod.io won't always go and check the last-update-date or the image checksum or something of the image it pull to build the workspace during prebuilds, thus, if I update the image then the next prebuild won't get it. (I don't ask that the prebuild be triggered off my updating the image out there in some container repository - just that it check to see if its been updated when pre-building.)