Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(deps): update bigdecimal from 0.4.1 to 0.4.6 #13569

Merged
merged 1 commit into from
Nov 27, 2024

Conversation

jonahgao
Copy link
Member

@jonahgao jonahgao commented Nov 26, 2024

Which issue does this PR close?

Closes #10001.

Rationale for this change

We use the bigdecimal crate to format decimal values to strings and display them in sqllogical tests(SLTs).
The new version of bigdecimal formats certain decimals in scientific notation, which will break many SLT tests.

There are two possible solutions:

  1. Set the compile-time environment variables for bigdecimal, such as RUST_BIGDECIMAL_FMT_EXPONENTIAL_LOWER_THRESHOLD. However, I'm a bit concerned about whether it's appropriate to operate environment variables in a library like DataFusion. Additionally, we might need to introduce a separate Cargo config.toml into the project root directory so that cargo test --test sqllogictests can work.
  2. Implement our own method for formatting decimals.

This PR takes the latter approach.

What changes are included in this PR?

Are these changes tested?

Yes. By new UTs and existing SLT tests.

Are there any user-facing changes?

No. Only modified the test engine.

@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Nov 26, 2024
Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @jonahgao for taking care on this

Copy link
Member

@Weijun-H Weijun-H left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @jonahgao, it is an elegant solution 👍

@jonahgao jonahgao merged commit acc2c7d into apache:main Nov 27, 2024
28 checks passed
@jonahgao
Copy link
Member Author

Thanks @comphead @Weijun-H

@jonahgao jonahgao deleted the update_bigdecimal branch November 27, 2024 13:37
@comphead
Copy link
Contributor

Btw, I was experimenting with env var, I didn't work well

@jonahgao
Copy link
Member Author

Btw, I was experimenting with env var, I didn't work well

I tested it on this branch, and it works except for one query.
cargo clean -p bigdecimal might be needed due to rust-lang/cargo#10358 (bigdecimal reads envs in build.rs)

@comphead
Copy link
Contributor

Good catch

// Round the value to limit the number of decimal places
let value = value.round(12).normalized();
// Format the value to a string
format_big_decimal(value)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could use bigdecimal to_plain_string (see akubera/bigdecimal-rs#135) -- but it's not released yet afaict

@findepi
Copy link
Member

findepi commented Dec 2, 2024

  1. Set the compile-time environment variables for bigdecimal, such as RUST_BIGDECIMAL_FMT_EXPONENTIAL_LOWER_THRESHOLD. However, I'm a bit concerned about whether it's appropriate to operate environment variables in a library like DataFusion.

fwiw, i faced same issuues when upgrading the bigdecimal dep in an internal project based on DataFusion and felt the same.
Especially that it could be impossible to "negotiate" value of RUST_BIGDECIMAL_FMT_EXPONENTIAL_LOWER_THRESHOLD if different parts of the code end up needing something different.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[TESTS] Fix sqllogictests to support bigdecimal 0.4.3 crate
4 participants