Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Value & Enable Lazy Ion literals. #523

Merged
merged 6 commits into from
Dec 5, 2024
Merged

Refactor Value & Enable Lazy Ion literals. #523

merged 6 commits into from
Dec 5, 2024

Conversation

jpschorr
Copy link
Contributor

@jpschorr jpschorr commented Dec 5, 2024

Merges the following previously PRed changes to main:

Also fixes a few minor lifetime warnings from newer versions of rust/clippy.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This changes the lexer and parser to pass through strings enclosed in backticks un-parsed. (At current, these documents are parsed during lowering).

Since embedded documents may themselves contain backticks, beginning and ending delimiters consist of an arbitrary odd numbers of backticks (e.g., `` ` ``, `` ``` ``, `` ````` `` etc.) that must be paired (e.g., `` `$ion_data_here::[]` ``, `` ```$ion_data_here::[ $string_with_embedded_backtick:"`" ]``` ``, etc.).

As opening and closing delimiters are required to be odd in count of backticks, a contiguous string of backticks that is even is interpreted as an empty document.
Change parsing and AST-modeling of literals to not share AST structures with non-scalar expressions.
Changes the logical plan to have a distinct `Lit` type to hold literals instead of embedded `Value`
@jpschorr jpschorr requested review from alancai98 and am357 December 5, 2024 21:28
@jpschorr jpschorr changed the title Dev ion doc Refactor Value & Enable Lazy Ion literals. Dec 5, 2024
Copy link

codecov bot commented Dec 5, 2024

Codecov Report

Attention: Patch coverage is 81.47448% with 196 lines in your changes missing coverage. Please review.

Project coverage is 80.75%. Comparing base (4a4b143) to head (2c7cb62).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
partiql-value/src/value.rs 81.12% 46 Missing and 1 partial ⚠️
partiql-logical/src/util.rs 61.53% 36 Missing and 4 partials ⚠️
partiql-value/src/value/math.rs 79.02% 30 Missing ⚠️
partiql-logical-planner/src/lower.rs 77.27% 12 Missing and 3 partials ⚠️
partiql-value/src/bindings.rs 40.00% 15 Missing ⚠️
partiql-logical-planner/src/typer.rs 0.00% 9 Missing ⚠️
partiql-parser/src/parse/parse_util.rs 83.67% 5 Missing and 3 partials ⚠️
partiql-value/src/value/iter.rs 76.66% 7 Missing ⚠️
partiql-value/src/sort.rs 77.77% 6 Missing ⚠️
partiql-value/src/util.rs 76.19% 5 Missing ⚠️
... and 7 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #523      +/-   ##
==========================================
- Coverage   80.76%   80.75%   -0.02%     
==========================================
  Files          80       89       +9     
  Lines       19405    19593     +188     
  Branches    19405    19593     +188     
==========================================
+ Hits        15673    15822     +149     
- Misses       3311     3352      +41     
+ Partials      421      419       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Dec 5, 2024

Conformance comparison report

Base (4a4b143) a11496b +/-
% Passing 86.97% 86.90% -0.06%
✅ Passing 5591 5587 -4
❌ Failing 838 842 4
🔶 Ignored 0 0 0
Total Tests 6429 6429 0

Number passing in both: 5587

Number failing in both: 838

Number passing in Base (4a4b143) but now fail: 4

Number failing in Base (4a4b143) but now pass: 0

⁉️ CONFORMANCE REPORT REGRESSION DETECTED ⁉️. The following test(s) were previously passing but now fail:

Click here to see
  • partiql_tests::fail::static_analysis::primitives::primitives::functions::lower::invalid_function_args::lower::lower_wrong_type_of_arguments_param
  • partiql_tests::fail::static_analysis::primitives::primitives::functions::upper::invalid_function_args::upper::upper_wrong_type_of_arguments_param
  • partiql_tests::fail::static_analysis::primitives::primitives::functions::char_length::invalid_function_args::char_length::char_length_wrong_types_in_
  • partiql_tests::fail::static_analysis::primitives::primitives::functions::char_length::invalid_function_args::char_length::character_length_wrong_types_in_

Copy link
Member

@alancai98 alancai98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like there's a slight regression in some conformance tests. Could take a brief look at those

@jpschorr
Copy link
Contributor Author

jpschorr commented Dec 5, 2024

Looks like there's a slight regression in some conformance tests. Could take a brief look at those

They are changes to static analysis tests. In making ion data lazy, the ion parsing is no longer happening at a time that triggers the failure for these tests.

e.g.,

fail_semantics("char_length(`()`)");

No longer immediately errors, since the () isn't parsed prior to lowering to a logical plan anymore. I'll either change the way semantics tests are validated as part of the larger lazy document initiative, or tests like these will be changed, as 'semantic' is a fuzzy line between syntax and evaluation anyway.

@jpschorr jpschorr merged commit a07bdfc into main Dec 5, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants