Release velocity for instrumentation bugfixes is poor

[ivomagi]

Description of the status quo:

1. Customer reports a bug in a specific OpenTelemetry instrumentation
2. Our engineering verifies the claim and ships a PR in a few days
3. It gets merged in Otel in few more days
4. ... weeks pass by, waiting for next Otel release, which in worst case is 30d away and on average 14d away.
5. Otel releases new version
6. Splunk-otel-x releases new version
7. Bugfix shipped.

Problem: very long passive wait time where we have fixed the bug but just have not released it. Meanwhile, customer is helplessly waiting.

Aspirational dream status: Whenever a bugfix in instrumentation is merged in Otel instrumentations, then either on-demand or better yet, every night, we run the tests, verify the stability of the snapshot and build a new splunk-otel-x based on the snapshot, not waiting for the Otel release to be cut. Alternatively there might be ways to make Otel ship new releases faster, but I guess this is not directly under our control.

[MrAlias]

Thanks for opening this topic up. It is definitely something we should have a solution for. We can continue the discussion of how to achieve this here, but when we have a general idea of what we want to do we need to open this up as a feature request issue.

That issue is not something we will include in the 1.0 milestone, so there is not a huge rush.

[iNikem]

One comment about very frequent (especially nightly) releases: this will make harder providing good release notes/changelog. We do want to have user-oriented changelog, which usually means writing it by hand (and not collecting it automatically from commit messages or PR titles). Thus every release will continue requiring manual work.

[mateuszrzeszutek]

And you still need to do a thorough check just before the release: somebody could've forgotten to update the changelog, some items may need to get merged together, etc.

[pellared]

because it will lead to a lot of merging conflicts and a need to constantly rebase your PRs

Come on, I guess that it is usually a 1 line conflict and you can simply "accept both" 😄 Like it would probably take a few seconds to resolve it and you get "always up to date" changelog and you can provide a release faster 😉

[iNikem]

• You have to merge and then wait again for CI pipeline to run
• If/when we start accepting external contributions it will be hard to require from people outside Splunk to always provide changelod and to reject PRs if not provided.
• From this very spec: "The file SHOULD contain an Unreleased section at the top, which includes changes that have not yet been released". Thus every release still require changes in CHANGELOG file.

[pellared]

@iNikem

1. Fair enough only if you are NOT using Require branches to be up to date before merging option. I was always enabling this option 😄
2. The approver can always help and suggest what to put in the changelog.
3. True. I never said it will be fully automated. I just find it easier (for myself) to update changelog as part of my work so I feel confident that I won't miss it later 😉

Of course, I am leaving everything to you as you know a lot better what will be best for you 👍

[iNikem]

That means that we still cannot have fully-automatic nightly releases. That was my main point :)

[mateuszrzeszutek]

In the Java world depending on snapshots might cause our build to become not repeatable, as snapshots are usually pruned after a week or so. It might be better to just copy the instrumentation (or bugfix) over to our repo if we really need it.

[ivomagi]

It would still be a friction and decisions on case-by-case basis, requiring multiple people to handle the escalation and involve engineering. There might be perfectly good reasons why achieving the dream state is hard, but what if we can move closer towards it and avoid the WIP just sitting there, helping no-one?

[seemk]

With js-contrib it really is too slow to get fixes out. The patch version numbering does not get used at all - the releases are done manually (perhaps once a month) which increases the minor version along with a changelog. I'd be happy to see more patch version releases which contain small non-breaking fixes, something to talk about at the next SIG.

[tigrannajaryan]

Do our (Splunk) distro releases need to wait for Otel releases? Can we release without waiting for official Otel release and just incorporate the latest Otel commit?

For the Collector we are able to if we need to. Once the change is committed to Otel Collector we can update our Splunk Collector dependency to point to the new Otel commit and make our release.

This way we decouple our releases from Otel releases if necessary. There are a couple downsides (maybe more):

• We potentially depend on a less stable Otel version, but I don't know if Otel releases today are tested any more thoroughly then individual commits (is there a manual testing process for Otel instrumentation that anybody does pre-release?).
• We can no longer say that our release version x.y.z matches Otel release version a.b.c (I don't know if we care about it).

[iNikem]

Second point: we do care, see #39 (comment)

In general, basing on commit may depend on language distribution/upstream incorporation way. E.g. in Java this may be problematic: #67 (comment) . But this very same Java problem can be solved by using Splunk Artifactory as a proxy... @mateuszrzeszutek All in all I am slightly in favour of having this option.

[mateuszrzeszutek]

Hmm, if we solve the problem of disappearing artifacts this might be a good option.

There is also another problem of breaking changes in all javaagent internal APIs in every second commit 😂 But once we stabilize that (1.2.0 release?) then we could depend on snapshots without major issues.