Migrate Prebuild settings – EXP-672 #18842

AlexTugarev · 2023-09-29T06:55:30Z

Description

This PR introduces a PrebuildSettings model to encapsulate prebuild settings of a project in Gitpod. Existing projects will get updated on access of the project, i.e. once the Project model is accesses, the settings are updated. The Project Settings page uses the new data structure to render and update prebuild settings.

As this is primarily a backend change, there are no UX changes to expect.

Regarding system behavior of this one time migration the noteworthy impact is the following:

Prebuilds will be explicitly disabled on pre-existing projects, if the project is not active and no prebuilds were registered in the past 7 days, which is the regular period Gitpod uses to start considering projects inactive.
- Using this predicate allows to safely assume, that if we mark prebuilds as enabled, the underlying prebuild events are working on those projects.
- We need to follow up with announcements and docs on the new prebuild settings, to make users aware of the new explicit enablement of prebuilds on projects.

Summary generated by Copilot

`🤖 Generated by Copilot at 9024949`

Refactor prebuild settings data model and logic to support more flexible and configurable prebuild options for projects. Update the UI, the protocol, the API, and the server components to use the new prebuilds property of the project settings, and migrate the existing settings of inactive projects.

Related Issue(s)

Fixes EXP-672

How to test

Documentation

Preview status

Gitpod was successfully deployed to your preview environment.

🏷️ Name - at-migrate8f2bf942a7
🔗 URL - at-migrate8f2bf942a7.preview.gitpod-dev.com/workspaces.
📚 Documentation - See our internal documentation for information on how to interact with your preview environment.
📦 Version - at-migrate-prebuild-settings-gha.17944
🗒️ Logs - GCP Logs Explorer

Build Options

Build

/werft with-werft
Run the build with werft instead of GHA
leeway-no-cache
/werft no-test
Run Leeway with --dont-test

Publish

/werft publish-to-npm
/werft publish-to-jb-marketplace

Installer

analytics=segment
with-dedicated-emulation
workspace-feature-flags
Add desired feature flags to the end of the line above, space separated

Preview Environment / Integration Tests

/werft with-local-preview
If enabled this will build install/preview
/werft with-preview
/werft with-large-vm
/werft with-gce-vm
If enabled this will create the environment on GCE infra
with-integration-tests=all
Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
with-monitoring

/hold

components/dashboard/src/projects/ProjectSettings.tsx

components/server/src/prebuilds/prebuild-manager.ts

components/server/src/projects/projects-service.ts

AlexTugarev · 2023-09-29T13:32:06Z

components/gitpod-protocol/src/teams-projects-protocol.ts

+     */
+    prebuilds?: PrebuildSettings;
+
+    /** @deprecated see `Project.settings.prebuilds.enabled` instead. */


FYI, not removing here, as these are required for the migration phase

AlexTugarev · 2023-09-29T13:37:30Z

components/server/src/prebuilds/github-enterprise-app.ts

@@ -174,10 +172,7 @@ export class GitHubEnterpriseApp {
                        context,
                    });

-                    const shouldRun = Project.hasPrebuildSettings(project)
-                        ? prebuildPrecondition.shouldRun
-                        : this.appRules.shouldRunPrebuild(config, CommitContext.isDefaultBranch(context), false, false);


FYI, this does effectively disables comparable settings in .gitpod.yml, which we want to deprecate in #18788

AlexTugarev · 2023-09-29T15:46:08Z

Header	Header

Briefly checked that the filtering feature is still working, e.g. build only on feature-* branches.

svenefftinge · 2023-09-30T07:36:27Z

We should remove these three settings with this PR (at least from the UI) and turn the defaults from

to

svenefftinge · 2023-09-30T07:56:46Z

components/gitpod-protocol/src/teams-projects-protocol.ts

+    export function getPrebuildSettings(project: Project): PrebuildSettingsWithDefaults {
+        const effective = {
+            ...PREBUILD_SETTINGS_DEFAULTS,
+            ...project.settings?.prebuilds,


This overrides properties that are set to undefined.
We need to filoter those properties out first.

/** * Returns effective prebuild settings for the given project. The resulting settings * contain default values for properties which are not set explicitly for this project. */ export function getPrebuildSettings(project: Project): PrebuildSettingsWithDefaults { const overrides = Object.fromEntries( Object.entries(project.settings?.prebuilds ?? {}).filter(([_, value]) => value !== undefined), ); return { ...PREBUILD_SETTINGS_DEFAULTS, ...overrides, }; } ```

svenefftinge · 2023-09-30T08:02:56Z

I propose to change the settings UI like this:

svenefftinge · 2023-09-30T08:04:06Z

components/gitpod-db/src/typeorm/workspace-db-impl.ts

@@ -805,7 +805,7 @@ export class TypeORMWorkspaceDBImpl extends TransactionalDBImpl<WorkspaceDB> imp
        const repo = await this.getPrebuiltWorkspaceRepo();

        let query = repo.createQueryBuilder("pws");
-        query = query.where("pws.projectId != :projectId", { projectId });
+        query = query.where("pws.projectId = :projectId", { projectId });


AlexTugarev · 2023-09-30T10:33:03Z

Alright, let me try to add those changes.

AlexTugarev · 2023-10-02T07:27:34Z

@svenefftinge, could we apply changes to the layout and inlining of settings in a separate PR 🙏🏻. This one is already 1K changes and it took a while to test ride it. The mentioned application of defaults is related but not affected by the settings migration in general. It wouldn't change much to do that separately.

easyCZ · 2023-10-05T08:33:44Z

components/public-api/gitpod/experimental/v1/projects.proto

-    bool keep_outdated_prebuilds_running = 2;
-    bool use_previous_prebuilds = 3;
-    int32 prebuild_every_nth = 4;
+    reserved 1; // enable_incremental_prebuilds


easyCZ · 2023-10-05T08:34:57Z

components/public-api-server/pkg/apiv1/project_test.go

@@ -328,22 +328,22 @@ func setupProjectsService(t *testing.T) (*protocol.MockAPIInterface, v1connect.P

 func newProject(p *protocol.Project) *protocol.Project {
 	r := rand.Int()
+	b_false := false


[nit] Underscore is an anti-pattern in go. camelCase is the expected

loujaybee · 2023-10-05T10:11:25Z

Linking through the related documentation and changelog updates related to changing the prebuild settings.

Given the potential impact on users workflows from this change, I would suggest that we make sure to incorporate documentation of the changes into the linked docs pull request, so that users are aware of the impact and actions on their end. Some areas that we'll want to clarify in comms (questions are rhetorical, so don't need directly answering here):

Will this impact only existing, or new projects?
What actions will a user have to take (given the behavioural changes e.g by enabling incremental prebuilds)?
What are the potential impacts on cost for these defaults? Whether adverse, or positive changes does it introduce?

... I will try to think through other impacts + questions.

Some further discussion, questions and clarifications in this internal slack thread.

and

- no more incremental prebuilds - always incremental workspace - never wait for running prebuilds

svenefftinge · 2023-10-05T14:31:45Z

/unhold

AlexTugarev requested a review from a team as a code owner September 29, 2023 06:55

roboquat added do-not-merge/hold size/XL labels Sep 29, 2023

github-actions bot added the team: team-experience label Sep 29, 2023

AlexTugarev marked this pull request as draft September 29, 2023 07:00

roboquat added the do-not-merge/work-in-progress label Sep 29, 2023

svenefftinge reviewed Sep 29, 2023

View reviewed changes

components/dashboard/src/projects/ProjectSettings.tsx Outdated Show resolved Hide resolved

svenefftinge reviewed Sep 29, 2023

View reviewed changes

components/server/src/prebuilds/prebuild-manager.ts Outdated Show resolved Hide resolved

svenefftinge reviewed Sep 29, 2023

View reviewed changes

components/server/src/projects/projects-service.ts Outdated Show resolved Hide resolved

roboquat added size/XXL and removed size/XL labels Sep 29, 2023

AlexTugarev commented Sep 29, 2023

View reviewed changes

AlexTugarev changed the title ~~Migrate Prebuild settings~~ Migrate Prebuild settings – EXP-672 Sep 29, 2023

AlexTugarev marked this pull request as ready for review September 29, 2023 16:11

roboquat removed the do-not-merge/work-in-progress label Sep 29, 2023

svenefftinge reviewed Sep 30, 2023

View reviewed changes

svenefftinge force-pushed the at/migrate-prebuild-settings branch from 1f1715d to b88a21f Compare October 5, 2023 08:30

AlexTugarev requested a review from a team as a code owner October 5, 2023 08:30

github-actions bot added the team: team-engine label Oct 5, 2023

easyCZ approved these changes Oct 5, 2023

View reviewed changes

svenefftinge force-pushed the at/migrate-prebuild-settings branch from b88a21f to d0decdb Compare October 5, 2023 09:59

svenefftinge force-pushed the at/migrate-prebuild-settings branch from d0decdb to b6466c4 Compare October 5, 2023 10:20

AlexTugarev and others added 24 commits October 5, 2023 12:46

[protocol] Add PrebuildSettings to ProjectSettings

95c458d

add migration function

4e0d852

add migration function and test

8ae49f2

update protocol helper functions

df06300

update prebuild manager

87047f1

use new prebuild setting in frontend

08c4c5a

update papi

0d12585

fix papi test

318c52c

rename var

8916a4b

fix papi mapping

df97e8c

deprecation note

e976d40

update migration function

813b55f

fix prebuild-manager

6201906

fixup server

7e7556d

fix Enable Prebuilds on Project Settings

1fa2565

Use Project.getPrebuildSettings

905d6cc

and

fix countUnabortedPrebuildsSince

0e630e8

improve usage of default settings

373ec10

fix Build Branches select

a72a96b

fix papi conversion 🤯

77ba20e

don't save empty pattern strings

56f41f1

fix papi-server conversion and align naming of prebuild settings

e2eefef

fix getPrebuildSettings

f111f23

[prebuilds] changed prebuild semantics

63c3c6d

- no more incremental prebuilds - always incremental workspace - never wait for running prebuilds

svenefftinge force-pushed the at/migrate-prebuild-settings branch from d29f3b8 to 63c3c6d Compare October 5, 2023 12:46

roboquat removed the do-not-merge/hold label Oct 5, 2023

svenefftinge approved these changes Oct 5, 2023

View reviewed changes

roboquat merged commit b2260d0 into main Oct 5, 2023
15 checks passed

roboquat deleted the at/migrate-prebuild-settings branch October 5, 2023 14:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate Prebuild settings – EXP-672 #18842

Migrate Prebuild settings – EXP-672 #18842

AlexTugarev commented Sep 29, 2023 •

edited

Loading

AlexTugarev Sep 29, 2023

AlexTugarev Sep 29, 2023

AlexTugarev commented Sep 29, 2023 •

edited

Loading

svenefftinge commented Sep 30, 2023

svenefftinge Sep 30, 2023

AlexTugarev Oct 2, 2023

svenefftinge commented Sep 30, 2023

svenefftinge Sep 30, 2023

AlexTugarev commented Sep 30, 2023

AlexTugarev commented Oct 2, 2023

easyCZ Oct 5, 2023

easyCZ Oct 5, 2023

loujaybee commented Oct 5, 2023

svenefftinge commented Oct 5, 2023

Migrate Prebuild settings – EXP-672 #18842

Migrate Prebuild settings – EXP-672 #18842

Conversation

AlexTugarev commented Sep 29, 2023 • edited Loading

Description

🤖 Generated by Copilot at 9024949

Related Issue(s)

How to test

Documentation

Preview status

Build Options

AlexTugarev Sep 29, 2023

Choose a reason for hiding this comment

AlexTugarev Sep 29, 2023

Choose a reason for hiding this comment

AlexTugarev commented Sep 29, 2023 • edited Loading

svenefftinge commented Sep 30, 2023

svenefftinge Sep 30, 2023

Choose a reason for hiding this comment

AlexTugarev Oct 2, 2023

Choose a reason for hiding this comment

svenefftinge commented Sep 30, 2023

svenefftinge Sep 30, 2023

Choose a reason for hiding this comment

AlexTugarev commented Sep 30, 2023

AlexTugarev commented Oct 2, 2023

easyCZ Oct 5, 2023

Choose a reason for hiding this comment

easyCZ Oct 5, 2023

Choose a reason for hiding this comment

loujaybee commented Oct 5, 2023

svenefftinge commented Oct 5, 2023

AlexTugarev commented Sep 29, 2023 •

edited

Loading

`🤖 Generated by Copilot at 9024949`

AlexTugarev commented Sep 29, 2023 •

edited

Loading