Upgrade snakeyaml to 2.0 #24636

zhfeng · 2023-03-16T03:24:18Z

Fixes CVE-2022-1471

Changes proposed in this pull request:

upgrade snakeyaml to 2.0

Before committing this PR, I'm sure that I have checked the following options:

My code follows the code of conduct of this project.
I have self-reviewed the commit code.
I have (or in comment I request) added corresponding labels for the pull request.
I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
I have made corresponding changes to the documentation.
I have added corresponding unit tests for my changes.

zhfeng · 2023-03-16T03:32:40Z

The more information could be found here: https://www.veracode.com/blog/research/resolving-cve-2022-1471-snakeyaml-20-release-0

I'm not sure shardingsphere allows global tags? I run some related tests with Yaml and they all work.

zhfeng · 2023-03-16T04:08:18Z

The test fails

Error:  org.apache.shardingsphere.test.it.data.pipeline.scenario.consistencycheck.api.impl.ConsistencyCheckJobAPITest.assertDropByParentJobId  Time elapsed: 0.374 s  <<< ERROR!
java.lang.NoSuchMethodError: org.yaml.snakeyaml.representer.Representer: method 'void <init>()' not found
	at org.apache.shardingsphere.elasticjob.infra.yaml.representer.ElasticJobYamlRepresenter.<init>(ElasticJobYamlRepresenter.java:28)
	at org.apache.shardingsphere.elasticjob.infra.yaml.YamlEngine.marshal(YamlEngine.java:38)
	at org.apache.shardingsphere.elasticjob.lite.lifecycle.internal.settings.JobConfigurationAPIImpl.updateJobConfiguration(JobConfigurationAPIImpl.java:51)
	at org.apache.shardingsphere.data.pipeline.core.api.impl.AbstractPipelineJobAPIImpl.stop(AbstractPipelineJobAPIImpl.java:151)
	at org.apache.shardingsphere.data.pipeline.scenario.consistencycheck.api.impl.ConsistencyCheckJobAPI.dropByParentJobId(ConsistencyCheckJobAPI.java:207)
	at org.apache.shardingsphere.test.it.data.pipeline.scenario.consistencycheck.api.impl.ConsistencyCheckJobAPITest.assertDropByParentJobId(ConsistencyCheckJobAPITest.java:108)

It seems that we need to fix in shardingsphere-elasticjob as well.

zhfeng · 2023-03-16T05:26:18Z

Well, shardingsphere-elasticjob::elasticjob-lite-spring-boot-starter depends on spring-boot:2.3.x which has not been fixed. According to https://spring.io/projects/spring-boot#support, 2.3.x is EOL so I think we should consider to upgrade to 2.7.x at least.

terrymanu · 2023-03-19T16:51:24Z

I am considering shading all dependencies of ShardingSphere at a later time so that I won't need to worry about compatibility issues with Spring or other third-party libraries.
However, this is a long-term job that may take several months to complete.

zhfeng · 2023-03-19T17:17:55Z

Thanks @terrymanu for taking care of the upgarding.

I did some works on shardingsphere-elasticjob to test with spring-boot 2.7.10-SNAPSHOT and snakeyaml 2.0. See zhfeng/shardingsphere-elasticjob@0c2d7b3

We need to be very careful about these CVE issues. Anyway, if I understand correctly, we only use snakeyaml for parsing the configuration files. So it could be low risk in this case.

zhfeng · 2023-03-19T17:22:54Z

Also do we consider to enable Vulnerability alerts in github which could notify us when our dependency libraies have some security issues which need to be fixed. Something like https://github.com/apache/camel/security/dependabot

Maybe we need to open a ticket for Apache Infra team?

TeslaCN · 2023-03-20T05:07:38Z

Also do we consider to enable Vulnerability alerts in github which could notify us when our dependency libraies have some security issues which need to be fixed. Something like https://github.com/apache/camel/security/dependabot

Maybe we need to open a ticket for Apache Infra team?

Thanks for your suggestion. Vulnerability alerts was enabled in ShardingSphere repo.

zhfeng · 2023-03-20T05:22:06Z

Hmm, I can not see it because I'm not a committor?

TeslaCN · 2023-03-20T06:02:54Z

Hmm, I can not see it because I'm not a committor?

I cannot see camel's security tab either.

zhfeng · 2023-03-20T07:57:25Z

Yeah, that makes sense. Anyway, CVE-2022-1471 is marked as High due to allow remote code execution. But in shardingshpere, I think we only use it to parse the yaml configuration files, so it might be low risk in this case?

TeslaCN · 2023-03-21T01:38:11Z

Yeah, that makes sense. Anyway, CVE-2022-1471 is marked as High due to allow remote code execution. But in shardingshpere, I think we only use it to parse the yaml configuration files, so it might be low risk in this case?

Yes.

linghengqian

As an additional topic, consider that the Spring team doesn't expect to make big changes in SnakeYAML, they expect to handle it in Spring Framework 6.1, refer to Switch to Yaml 1.2 with snakeyaml-engine spring-projects/spring-framework#28349 and Upgrade to SnakeYAML 2.0 spring-projects/spring-framework#30048 , should I close Markup SpringBoot users need to specify SnakeYAML version in Github Wiki #21476 to wait for our shade operation on SnakeYAML?

lizongbo

yes need upgrade to support snakeyml 1.33 to 2.1

linghengqian · 2023-08-19T18:20:13Z

yes need upgrade to support snakeyml 1.33 to 2.1

@lizongbo Obviously we have two ways of thinking about dealing with downstream usage. One is to wait for the Release of ShardingSphere ElasticJob 3.0.4, and the other is to shade the SnakeYAML package, so that the downstream can use a higher version of the SnakeYAML API.
I assume you are interested in the first option, but I am not sure about the release plan of ShardingSphere ElasticJob 3.0.4. Would you like to start a discussion on the mailing list?

lizongbo · 2023-08-20T12:48:03Z

yes need upgrade to support snakeyml 1.33 to 2.1

@lizongbo Obviously we have two ways of thinking about dealing with downstream usage. One is to wait for the Release of ShardingSphere ElasticJob 3.0.4, and the other is to shade the SnakeYAML package, so that the downstream can use a higher version of the SnakeYAML API.

I assume you are interested in the first option, but I am not sure about the release plan of ShardingSphere ElasticJob 3.0.4. Would you like to start a discussion on the mailing list?

This changed code is fully compatible with snakeyml 1.33, 2.0, 2.1.

because The Representer class's old constructor Representer() was marked as Deprecated in 1.33 and then removed in version 2.0

"public Representer(DumperOptions options) " is exists in 1.33 ,2.0 and 2.1 .

linghengqian · 2023-08-21T01:51:45Z

yes need upgrade to support snakeyml 1.33 to 2.1

@lizongbo Obviously we have two ways of thinking about dealing with downstream usage. One is to wait for the Release of ShardingSphere ElasticJob 3.0.4, and the other is to shade the SnakeYAML package, so that the downstream can use a higher version of the SnakeYAML API.

I assume you are interested in the first option, but I am not sure about the release plan of ShardingSphere ElasticJob 3.0.4. Would you like to start a discussion on the mailing list?

This changed code is fully compatible with snakeyml 1.33, 2.0, 2.1.

because The Representer class's old constructor Representer() was marked as Deprecated in 1.33 and then removed in version 2.0

"public Representer(DumperOptions options) " is exists in 1.33 ,2.0 and 2.1 .

@lizongbo It looks like the deprecated method on SnakeYAML 1.33 is used because new method of SnakeYAML 1.33 caused CI execution to fail. fix ci #21480 reintroduced methods that were deprecated on SnakeYAML 1.33.
@totalo I wonder if fix ci #21480 has any background information? Add transaction IT test demo #19293 doesn't mention SnakeYAML.

zhfeng · 2023-08-21T07:15:21Z

So is there any update on a new release of ElasticJob? I think it has been upgrade.

linghengqian · 2023-08-21T07:33:08Z

So is there any update on a new release of ElasticJob? I think it has been upgrade.

@zhfeng Apparently ShardingSphere 5.4.1 will close the milestone on September 30 of this year, see https://github.com/apache/shardingsphere/milestone/27 . But ShardingSphere ElasticJob 3.0.4 doesn't have a specific release plan, see https://github.com/apache/shardingsphere-elasticjob/milestone/9 .
I assume there is a lack of a PMC responsible for the Release program?

zhfeng · 2023-08-21T07:36:50Z

Thanks @linghengqian - Please raise this concern on the dev mailing list. It should be important to fix the security issue and hope it can be included in 5.4.1.

linghengqian · 2023-08-21T14:46:46Z

Thanks @linghengqian - Please raise this concern on the dev mailing list. It should be important to fix the security issue and hope it can be included in 5.4.1.

@zhfeng I found that I wasn't quite sure how to rewrite the sender from outlook.com to my personal apache.org mailbox, outlook's PC client got a big update in August of this year.
I'm assuming you have time to send mail to the dev mailing list first?

zhfeng · 2023-08-22T06:00:08Z

@linghengqian NP - I will take care of it.

TeslaCN · 2023-09-12T02:43:54Z

Hi @linghengqian Are you willing to release ElasticJob 3.0.4?

linghengqian · 2023-09-12T02:47:18Z

Hi @linghengqian Are you willing to release ElasticJob 3.0.4?

@TeslaCN I have no experience working on releases at ASF, and my work has been scattered in recent months. I hope other committers can help with this process.

linghengqian

@zhfeng ShardingSphere ElasticJob 3.0.4 with SnakeYAML 2.x is now available in https://repo1.maven.org/maven2/org/apache/shardingsphere/elasticjob/elasticjob-lite-core/3.0.4/ .
Do you feel like this needs to be a separate PR, or is it handled in the current PR?

zhfeng · 2023-10-19T07:17:43Z

@linghengqian feel free to rasie a new PR!

linghengqian · 2023-10-19T13:23:14Z

@linghengqian feel free to rasie a new PR!

I have opened the Updates ElasticJob to 3.0.4 to block CVEs for SnakeYAML delivery #28805 .

zhfeng · 2023-10-19T13:49:00Z

Thanks @linghengqian and I close this PR.

linghengqian reviewed Mar 30, 2023

View reviewed changes

linghengqian mentioned this pull request Apr 23, 2023

Sharding startup failed after upgrading the snake yml component to version 2.0 #25291

Closed

linghengqian mentioned this pull request May 8, 2023

When could you update snakeyaml version? apache/shardingsphere-elasticjob#2215

Closed

github-actions bot added the stale label Jul 12, 2023

zhfeng mentioned this pull request Jul 31, 2023

quarkus 3.2.2 无法正常加载数据库 quarkiverse/quarkus-shardingsphere-jdbc#139

Closed

lizongbo approved these changes Aug 19, 2023

View reviewed changes

github-actions bot removed the stale label Aug 19, 2023

Upgrade snakeyaml to 2.0

f189047

zhfeng force-pushed the upgrade_snakeyaml_2.0 branch from 0f4153b to f189047 Compare August 22, 2023 06:06

linghengqian mentioned this pull request Aug 28, 2023

Snakeyml 2.1 compatibility issue with shardingsphere-jdbc-core-spring-boot-starter 5.2.1 #28233

Closed

linghengqian mentioned this pull request Sep 11, 2023

[DISCUSS] Release Plan of ShardingSphere 5.4.1 #28411

Closed

RaigorJiang mentioned this pull request Oct 12, 2023

Snakeyaml has a high version conflict, and now relies on version 1.33 to have vulnerabilities. #28717

Closed

linghengqian reviewed Oct 18, 2023

View reviewed changes

linghengqian mentioned this pull request Oct 19, 2023

Updates ElasticJob to 3.0.4 to block CVEs for SnakeYAML delivery #28805

Merged

6 tasks

zhfeng closed this Oct 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade snakeyaml to 2.0 #24636

Upgrade snakeyaml to 2.0 #24636

zhfeng commented Mar 16, 2023

zhfeng commented Mar 16, 2023

zhfeng commented Mar 16, 2023

zhfeng commented Mar 16, 2023

terrymanu commented Mar 19, 2023

zhfeng commented Mar 19, 2023

zhfeng commented Mar 19, 2023

TeslaCN commented Mar 20, 2023

zhfeng commented Mar 20, 2023

TeslaCN commented Mar 20, 2023

zhfeng commented Mar 20, 2023 •

edited

Loading

TeslaCN commented Mar 21, 2023

linghengqian left a comment

lizongbo left a comment

linghengqian commented Aug 19, 2023

lizongbo commented Aug 20, 2023

linghengqian commented Aug 21, 2023

zhfeng commented Aug 21, 2023

linghengqian commented Aug 21, 2023

zhfeng commented Aug 21, 2023

linghengqian commented Aug 21, 2023

zhfeng commented Aug 22, 2023

TeslaCN commented Sep 12, 2023

linghengqian commented Sep 12, 2023

linghengqian left a comment

zhfeng commented Oct 19, 2023

linghengqian commented Oct 19, 2023

zhfeng commented Oct 19, 2023

Upgrade snakeyaml to 2.0 #24636

Upgrade snakeyaml to 2.0 #24636

Conversation

zhfeng commented Mar 16, 2023

zhfeng commented Mar 16, 2023

zhfeng commented Mar 16, 2023

zhfeng commented Mar 16, 2023

terrymanu commented Mar 19, 2023

zhfeng commented Mar 19, 2023

zhfeng commented Mar 19, 2023

TeslaCN commented Mar 20, 2023

zhfeng commented Mar 20, 2023

TeslaCN commented Mar 20, 2023

zhfeng commented Mar 20, 2023 • edited Loading

TeslaCN commented Mar 21, 2023

linghengqian left a comment

Choose a reason for hiding this comment

lizongbo left a comment

Choose a reason for hiding this comment

linghengqian commented Aug 19, 2023

lizongbo commented Aug 20, 2023

linghengqian commented Aug 21, 2023

zhfeng commented Aug 21, 2023

linghengqian commented Aug 21, 2023

zhfeng commented Aug 21, 2023

linghengqian commented Aug 21, 2023

zhfeng commented Aug 22, 2023

TeslaCN commented Sep 12, 2023

linghengqian commented Sep 12, 2023

linghengqian left a comment

Choose a reason for hiding this comment

zhfeng commented Oct 19, 2023

linghengqian commented Oct 19, 2023

zhfeng commented Oct 19, 2023

zhfeng commented Mar 20, 2023 •

edited

Loading