Release workflow

[cbrnr]

Following up on #12616, I'd like to discuss our current release workflow. I'm curious about why we need to backport changes. Would it be possible to streamline our process by merging everything into the main branch and releasing more frequently, including minor releases? Given that we don't strictly adhere to semantic versioning anyway, I propose adopting a rolling release model. This would eliminate the need for separate backport branches, leaving us with just the main branch from which we periodically issue new releases, including minor updates. This approach would even be compatible with our existing version numbering scheme (meaning that I'm fine with whatever scheme we choose). As a positive side effect, we could have more frequent (minor/patch) releases, because the process is much less cumbersome.

[hoechenberger]

I agree, but we'd probably need to come up with a time-based (not release-based) schedule for deprecations in that case.

[cbrnr]

If we keep our current schedule of two major releases per year (and we only increment the minor version), then we could still deprecate on a release basis, or no?

[hoechenberger]

Ah, I thought you were suggesting to release more frequently

[drammock]

Would it be possible to streamline our process by merging everything into the main branch and releasing more frequently, including minor releases?

IMO the main downside to "never backport anything, and release more frequently" is that we occasionally merge PRs that are large or complex enough that we're not 100% confident that things won't break in some corner cases. We try to merge PRs like that near the beginning of a release cycle, so that folks using main have more time to notice and report bugs. A shorter release cycle would mean a higher likelihood that such bugs would make it into released "stable" versions of the software.

our current schedule of two major releases per year

(nitpick: current schedule is 3x / year)

quoting from #12616 (comment):

I don't really have time to go through this [backporting] process

An uncomplicated backport takes me <5 minutes. The more complicated ones I've done (involving merge conflicts / test fails) have been 10-30 minutes. We have instructions on the wiki (though I'll admit they're a bit out of date). Is 5 minutes really too long, or is there some other barrier? Would better documentation of the backporting process change your willingness to do them?

[hoechenberger]

I guess the question is really if backports are necessary at all. Usually if a problem is discovered briefly after a release, one could just cut a new minor release from main. Of course you'd then have the problem that larger, less-tested changes may have found their way into main in the meantime, too, yes.

A potential way around this is having a dev and a release branch. Dev branch would be main, and anytime a feature seems to be ready for end users, it gets pulled into the release branch.

I'm not sure if that wouldn't create even more overhead than backporting, though.

Personally, I believe anything that enters main should always be good enough for a release, or in other words: at any point in time, it should be safe to cut a new release directly from main.And I believe we're actually super close to this ... "Hotfixes" would always just be new minor releases from main.

[drammock]

Dev branch would be main, and anytime a feature seems to be ready for end users, it gets pulled into the release branch.
I'm not sure if that wouldn't create even more overhead than backporting, though.

To me it seems almost certain that this would be more work than our current backporting approach.

anything that enters main should always be good enough for a release [...] "Hotfixes" would always just be new minor releases from main.

With our current release cadence of ~4 months, and the relative infrequency of finding bugs serious enough to trigger a backport, those "hotfixes" could potentially have a couple months' worth of changes in them (including new features, not just bugfixes / doc corrections). That's not necessarily a problem, but it's not consistent with what "hotfix" usually means in my experience. Rather, to me this sounds like "just release more often" which I'm reluctant to do given the time our release process takes (even after improvements like #12037 and #12299).

[cbrnr]

I guess the question is really if backports are necessary at all.

Exactly!

To me it seems almost certain that this would be more work than our current backporting approach.

Yes, agreed!

Personally, I believe anything that enters main should always be good enough for a release

💯

IMO the main downside to "never backport anything, and release more frequently" is that we occasionally merge PRs that are large or complex enough that we're not 100% confident that things won't break in some corner cases. We try to merge PRs like that near the beginning of a release cycle, so that folks using main have more time to notice and report bugs. A shorter release cycle would mean a higher likelihood that such bugs would make it into released "stable" versions of the software.

I could live with things breaking in some rare corner cases once in a while. After all, with a new release process skipping backports, we'd be able to quickly push a hotfix release. Plus we'd be able to push new features more quickly, which is something users will surely appreciate.

An uncomplicated backport takes me <5 minutes. The more complicated ones I've done (involving merge conflicts / test fails) have been 10-30 minutes. We have instructions on the wiki (though I'll admit they're a bit out of date). Is 5 minutes really too long, or is there some other barrier? Would better documentation of the backporting process change your willingness to do them?

Yes, better documentation would certainly help. I've done it only once or twice, so I'd have to do everything from scratch, and I'm pretty sure it would take me more than five minutes. But this raises another question: I don't think that every dev needs to be able to perform every kind of maintenance tasks in the first place. I know we don't have dev roles (yet?), but this is really a great example for a "release manager" role. Someone who already knows how to backport can do it in 5 minutes, whereas it would take everyone else a significantly longer time, which is probably better spent elsewhere.

[hoechenberger]

@cbrnr There's some docs in the Wiki, https://github.com/mne-tools/mne-python/wiki/How-to-backport-a-change

[cbrnr]

I know, Dan already linked them.

[hoechenberger]

I know, Dan already linked them.

Whoops, right. My bad

[larsoner]

So far I think our system is less painful than the other different ones proposed, given the end-user expectations we try to satisfy and have tried to deliver on for roughly a decade now. Especially if the main gain is avoiding backporting work -- as a maintainer, backporting fixes is one of the least painful pain points I encounter.

Nonetheless what we could do is look into something like a backporting bot. Like if you label "backport candidate" (which we do already and is the easy step!) it could maybe pick this up and try to open a PR to maint/<whatever> to backport the change. This would already save a bit of time. I assume this exists but I'll also possibly look into it as part of scientific-python/summit-2024#14

[drammock]

it does exist if you want to try it now: https://github.com/MeeseeksBox/MeeseeksDev

After the summit it will have a new home in a repo in the Scientific Python org, and will hopefully get some upgrades.

[cbrnr]

A bot would be great!

[larsoner]

Bot seems to work with a simple command

#12634

And I added a description to backport-candidate label that we usually use that should make a backport PR automatically open once a PR is merged (but didn't for #12633 because I bungled the label description). Okay to close this issue?

[cbrnr]

This is a great improvement, thanks @larsoner!

[larsoner]

@mscheltienne this bot was configured just by following the instructions on https://meeseeksbox.github.io/. Relevant commands are listed at https://github.com/MeeseeksBox/MeeseeksDev?tab=readme-ov-file#what-can-a-meeseeksbox-do-