Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use jsonpb AllowUnknownFields everywhere #5521

Merged
merged 3 commits into from
Jul 4, 2024

Conversation

andrewwdye
Copy link
Contributor

Why are the changes needed?

By default jsonpb.Unmarshal(), jsonpb.UnmarshalString, and jsonpb.Unmarshaler{} sets AllowUnknownFields to false, which means that unmarshaling a proto message with a newly added field will fail here.

What changes were proposed in this pull request?

This change updates all use of jsonpb.Unmarshal* to use the flytestlib/utils.Unmarshal* abstractions, which set AllowUnknownFields to true.

There is some risk that this break existing assumptions about new field handling, but since protobuf silently drops fields, it seems reasonable to default to this behavior across the flyte codebase.

How was this patch tested?

Unittests

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Copy link

codecov bot commented Jun 26, 2024

Codecov Report

Attention: Patch coverage is 58.82353% with 21 lines in your changes missing coverage. Please review.

Project coverage is 61.05%. Comparing base (ce5eb03) to head (a4aec9c).
Report is 121 commits behind head on master.

Files with missing lines Patch % Lines
...epropeller/pkg/apis/flyteworkflow/v1alpha1/gate.go 0.00% 6 Missing ⚠️
...ller/pkg/apis/flyteworkflow/v1alpha1/identifier.go 0.00% 5 Missing ⚠️
...propeller/pkg/apis/flyteworkflow/v1alpha1/nodes.go 50.00% 4 Missing ⚠️
flytectl/cmd/get/node_execution.go 50.00% 2 Missing ⚠️
flytestdlib/utils/marshal_utils.go 84.61% 0 Missing and 2 partials ⚠️
...propeller/pkg/apis/flyteworkflow/v1alpha1/tasks.go 50.00% 1 Missing ⚠️
...peller/pkg/apis/flyteworkflow/v1alpha1/workflow.go 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5521      +/-   ##
==========================================
+ Coverage   60.99%   61.05%   +0.05%     
==========================================
  Files         794      794              
  Lines       51475    51433      -42     
==========================================
+ Hits        31397    31402       +5     
+ Misses      17186    17146      -40     
+ Partials     2892     2885       -7     
Flag Coverage Δ
unittests-datacatalog 69.31% <ø> (ø)
unittests-flyteadmin 58.73% <ø> (ø)
unittests-flytecopilot 17.79% <ø> (ø)
unittests-flytectl 68.06% <50.00%> (+0.02%) ⬆️
unittests-flyteidl 79.06% <100.00%> (+0.01%) ⬆️
unittests-flyteplugins 61.85% <ø> (ø)
unittests-flytepropeller 57.47% <46.87%> (+0.16%) ⬆️
unittests-flytestdlib 65.68% <84.61%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@andrewwdye andrewwdye force-pushed the andrew/use-jsonpb-allowunknownfields-everywhere branch from a854a08 to c996dcb Compare June 26, 2024 21:48
squiishyy
squiishyy previously approved these changes Jun 27, 2024
Copy link
Contributor

@squiishyy squiishyy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you for making the change!

Signed-off-by: Andrew Dye <[email protected]>
Signed-off-by: Andrew Dye <[email protected]>
Copy link
Member

@pingsutw pingsutw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you!

@eapolinario eapolinario merged commit e13bfe3 into master Jul 4, 2024
52 of 53 checks passed
@eapolinario eapolinario deleted the andrew/use-jsonpb-allowunknownfields-everywhere branch July 4, 2024 04:41
vlibov pushed a commit to vlibov/flyte that referenced this pull request Aug 16, 2024
* Use jsonpb AllowUnknownFields everywhere

Signed-off-by: Andrew Dye <[email protected]>

* Missing return

Signed-off-by: Andrew Dye <[email protected]>

* make lint-fix

Signed-off-by: Andrew Dye <[email protected]>

---------

Signed-off-by: Andrew Dye <[email protected]>
Signed-off-by: Vladyslav Libov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants