Bump beam to 2.61, pin beam dependencies, reduce dependabot noise #925

kellen · 2024-11-21T07:39:31Z

Note that this involves downgrading some libraries to be in sync with the beam release

codecov · 2024-11-21T07:41:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.65%. Comparing base (addacc3) to head (429dc21).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master     #925   +/-   ##
=========================================
  Coverage     91.65%   91.65%           
  Complexity      252      252           
=========================================
  Files            26       26           
  Lines           947      947           
  Branches         71       71           
=========================================
  Hits            868      868           
  Misses           52       52           
  Partials         27       27

pom.xml

labianchin

Reducing dependabot noise looks good.
Still, can you expand on why are dependencies being downgraded? Or why syncing with with older versions Beam release?

In my experience Beam does not keep dependencies up to date very well, sometimes carrying vulnerabilities. The up to date versions here have been "more compatible" with maven enforcer, and had no runtime issues (mostly minor bumps, no breaking changes).

labianchin · 2024-11-27T08:51:03Z

pom.xml

+    <!-- https://github.com/apache/beam/blob/release-2.60.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L586 -->
+    <beam.version>2.60.0</beam.version>


Suggested change



<beam.version>2.60.0</beam.version>



<beam.version>2.61.0</beam.version>

beam 2.61.0 artifact has been promoted yesterday. When updating to 2.61, we'll make sure all deps are in sync

labianchin · 2024-11-27T08:52:25Z

pom.xml

+    <hamcrest.version>2.1</hamcrest.version>
+    <httpclient.version>4.5.13</httpclient.version>
+    <httpcore.version>4.4.14</httpcore.version>
+    <jackson.version>2.15.4</jackson.version>


Suggested change

<jackson.version>2.15.4</jackson.version>

<jackson.version>2.17.3</jackson.version>

Keep up to date?

I'd rather keep beam's version to not mix with jackson modules. If we upgrade, we should use jackson BOM instead.

labianchin · 2024-11-27T08:53:07Z

pom.xml

    <!-- https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/26.45.0/index.html -->
    <google-cloud-libraries-bom.version>26.45.0</google-cloud-libraries-bom.version>


Suggested change



<google-cloud-libraries-bom.version>26.45.0</google-cloud-libraries-bom.version>



<google-cloud-libraries-bom.version>26.50.0</google-cloud-libraries-bom.version>

We're syncing with beam's version

labianchin · 2024-11-27T08:54:12Z

pom.xml

+    <commons-codec.version>1.17.1</commons-codec.version> <!-- libraries-bom version -->
+    <commons-compress.version>1.26.2</commons-compress.version>
+    <errorprone.version>2.10.0</errorprone.version>
+    <guava.version>33.1.0-jre</guava.version>


Suggested change

<guava.version>33.1.0-jre</guava.version>

<guava.version>33.3.1-jre</guava.version>

labianchin · 2024-11-27T08:55:04Z

pom.xml

+    <netty.version>4.1.100.Final</netty.version>
+    <slf4j.version>1.7.30</slf4j.version>
+    <threetenbp.version>1.6.8</threetenbp.version>
+    <zstd-jni.version>1.5.6-3</zstd-jni.version>


Suggested change

<zstd-jni.version>1.5.6-3</zstd-jni.version>

<zstd-jni.version>1.5.6-8</zstd-jni.version>

labianchin · 2024-11-27T08:56:58Z

pom.xml

@@ -554,6 +543,32 @@
                    <!-- Keep aligned with prerequisite section below. -->
                    <version>[3.3.9,)</version>
                  </requireMavenVersion>
+                  <requireUpperBoundDeps>
+                    <excludes>
+                      <!-- managed by beam BOM -->


Why?

AFAIU maven enforcer helps to keep dependency compatibilities. With these many exclusions wouldn't the risk of post release runtime errors increase?

This is the strategy we used with scio. We upgrade those libs along with beam

labianchin · 2024-11-27T08:57:54Z

.github/dependabot.yml

@@ -12,15 +12,34 @@ updates:
      - dependency-name: com.fasterxml.jackson.core:jackson-annotations
      - dependency-name: com.fasterxml.jackson.core:jackson-core
      - dependency-name: com.fasterxml.jackson.core:jackson-databind
+      - dependency-name: com.fasterxml.jackson:jackson-bom
+      - dependency-name: com.github.luben:zstd-jni


Suggested change

- dependency-name: com.github.luben:zstd-jni

I think we can keep zstd updates, as they are decoupled with Beam.

zstd-jni is not following classic semver. the postfix -3 causes sometimes conflict with dependency resolvers. We prefer to stick with on the common version with beam

labianchin · 2024-11-27T09:00:14Z

pom.xml

+    <httpcore.version>4.4.14</httpcore.version>
+    <jackson.version>2.15.4</jackson.version>
+    <joda-time.version>2.10.14</joda-time.version>
+    <netty.version>4.1.100.Final</netty.version>


Suggested change

<netty.version>4.1.100.Final</netty.version>

<netty.version>4.1.115.Final</netty.version>

Same as jackson

RustedBones · 2024-12-11T07:57:53Z

.github/dependabot.yml

@@ -12,15 +12,34 @@ updates:
      - dependency-name: com.fasterxml.jackson.core:jackson-annotations
      - dependency-name: com.fasterxml.jackson.core:jackson-core
      - dependency-name: com.fasterxml.jackson.core:jackson-databind
+      - dependency-name: com.fasterxml.jackson:jackson-bom
+      - dependency-name: com.github.luben:zstd-jni


zstd-jni is not following classic semver. the postfix -3 causes sometimes conflict with dependency resolvers. We prefer to stick with on the common version with beam

RustedBones · 2024-12-11T07:59:40Z

pom.xml

+    <hamcrest.version>2.1</hamcrest.version>
+    <httpclient.version>4.5.13</httpclient.version>
+    <httpcore.version>4.4.14</httpcore.version>
+    <jackson.version>2.15.4</jackson.version>


I'd rather keep beam's version to not mix with jackson modules. If we upgrade, we should use jackson BOM instead.

RustedBones · 2024-12-11T07:59:54Z

pom.xml

+    <httpcore.version>4.4.14</httpcore.version>
+    <jackson.version>2.15.4</jackson.version>
+    <joda-time.version>2.10.14</joda-time.version>
+    <netty.version>4.1.100.Final</netty.version>


Same as jackson

RustedBones · 2024-12-11T08:00:08Z

pom.xml

+    <netty.version>4.1.100.Final</netty.version>
+    <slf4j.version>1.7.30</slf4j.version>
+    <threetenbp.version>1.6.8</threetenbp.version>
+    <zstd-jni.version>1.5.6-3</zstd-jni.version>


RustedBones · 2024-12-11T08:02:27Z

pom.xml

    <!-- https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/26.45.0/index.html -->
    <google-cloud-libraries-bom.version>26.45.0</google-cloud-libraries-bom.version>


We're syncing with beam's version

RustedBones · 2024-12-11T08:03:32Z

pom.xml

@@ -554,6 +543,32 @@
                    <!-- Keep aligned with prerequisite section below. -->
                    <version>[3.3.9,)</version>
                  </requireMavenVersion>
+                  <requireUpperBoundDeps>
+                    <excludes>
+                      <!-- managed by beam BOM -->


This is the strategy we used with scio. We upgrade those libs along with beam

kellen added 2 commits November 21, 2024 02:16

wip

1f130c3

wip

ac8f258

wip

dcc9fca

kellen requested a review from RustedBones November 21, 2024 12:43

RustedBones reviewed Nov 22, 2024

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

pom.xml Show resolved Hide resolved

avro

8dd394c

labianchin reviewed Nov 27, 2024

View reviewed changes

Update to beam 2.61

1d174c2

RustedBones changed the title ~~Bump beam to 2.60, pin beam dependencies, reduce dependabot noise~~ Bump beam to 2.61, pin beam dependencies, reduce dependabot noise Dec 11, 2024

RustedBones approved these changes Dec 11, 2024

View reviewed changes

Merge remote-tracking branch 'origin/master' into kellen/deps

429dc21

RustedBones merged commit 1f38aa8 into master Dec 11, 2024
8 checks passed

RustedBones deleted the kellen/deps branch December 11, 2024 08:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump beam to 2.61, pin beam dependencies, reduce dependabot noise #925

Bump beam to 2.61, pin beam dependencies, reduce dependabot noise #925

kellen commented Nov 21, 2024 •

edited

Loading

codecov bot commented Nov 21, 2024 •

edited

Loading

labianchin left a comment

labianchin Nov 27, 2024

RustedBones Nov 27, 2024

labianchin Nov 27, 2024

RustedBones Dec 11, 2024

labianchin Nov 27, 2024

RustedBones Dec 11, 2024

labianchin Nov 27, 2024

labianchin Nov 27, 2024

RustedBones Dec 11, 2024

labianchin Nov 27, 2024

RustedBones Dec 11, 2024

labianchin Nov 27, 2024

RustedBones Dec 11, 2024

labianchin Nov 27, 2024

RustedBones Dec 11, 2024

RustedBones Dec 11, 2024

RustedBones Dec 11, 2024

RustedBones Dec 11, 2024

RustedBones Dec 11, 2024

RustedBones Dec 11, 2024

RustedBones Dec 11, 2024

		<!-- https://github.com/apache/beam/blob/release-2.60.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L586 -->
		<beam.version>2.60.0</beam.version>

	<jackson.version>2.15.4</jackson.version>
	<jackson.version>2.17.3</jackson.version>

		<!-- https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/26.45.0/index.html -->
		<google-cloud-libraries-bom.version>26.45.0</google-cloud-libraries-bom.version>

	<guava.version>33.1.0-jre</guava.version>
	<guava.version>33.3.1-jre</guava.version>

	<zstd-jni.version>1.5.6-3</zstd-jni.version>
	<zstd-jni.version>1.5.6-8</zstd-jni.version>

	<netty.version>4.1.100.Final</netty.version>
	<netty.version>4.1.115.Final</netty.version>

Bump beam to 2.61, pin beam dependencies, reduce dependabot noise #925

Bump beam to 2.61, pin beam dependencies, reduce dependabot noise #925

Conversation

kellen commented Nov 21, 2024 • edited Loading

codecov bot commented Nov 21, 2024 • edited Loading

Codecov Report

labianchin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kellen commented Nov 21, 2024 •

edited

Loading

codecov bot commented Nov 21, 2024 •

edited

Loading