Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated DF (patched) branch to Nov-11. #50

Draft
wants to merge 9 commits into
base: base-nov-11
Choose a base branch
from
Draft

Updated DF (patched) branch to Nov-11. #50

wants to merge 9 commits into from

Conversation

wiedld
Copy link
Collaborator

@wiedld wiedld commented Dec 19, 2024 •
edited
Loading

This takes us up to Nov-11 SHA cadeb53

It advances DF by adding these oct-31 to nov-11 commits: f23360f...influxdata:arrow-datafusion:cadeb53e4ad36a6a12238e3acc833c6405c75dd2

EARs fixed:

  • we advance past the Nov-3 commit requested in this EAR: 5575
  • we cherry-picked Dec-3 SHA apache@86740bf for EAR 5776

Patches applied:

  • Patches already fixed in future upstream commits:

  • Patches still not fixed:

  • Patches specific to our DF branch (no need for upstream fix):

    • bit_length(utf8view) patch: 90423e2
    • we are keeping in the warn logging for the schema mismatch. (Upstream only returns error, or not, based on schema misalignment): our patch 69d6118
    • we are using an extra few bytes for the GroupedHashAggregateStream: 50e1209
    • NEWLY ADDED: our sqllogictests for string LIKE are now matching the updated output after the arrow-rs 53.3.0 upgrade. See commit with message and link: 7a0dd9d

  • tech debt to be fixed:

    • we are keeping in the warn logging for the schema mismatch. (Upstream only returns error, or not, based on schema misalignment): our patch 69d6118

alamb and others added 8 commits December 19, 2024 10:41
@alamb @wiedld
Test + workaround for SanityCheck plan
508ddc9
@alamb @wiedld
chore: workaround new clippy failures by disabling them
378aba3
@wiedld
fix: handle when the left side of the union has no fields (e.g. an em…
b94d515 
…pty projection)
@wiedld
chore: default=true for skip_physical_aggregate_schema_check, and add…
2f42c39 
… warn logging
@wiedld
Revert "Account for constant equivalence properties in union, tests (a…
836f92e 
…pache#12562)"

This reverts commit 577e4bb.
@wiedld
fix: temporary commit to update bit_length signature to only accept n…
4435c87 
…on-view Utf8s, and test does pass (due to coercion?)
@wiedld
fix: tmp commit since our local changes make us use slightly more mem…
3bdfa90 
…ory for the aggregates
@wiedld
fix: LIKE string comparisons now match DF main, based upon changes br…
7a0dd9d 
…ought in with DF main's arrow-rs upgrade to 53.3.0 on Nov 20th: apache@a2811fc
@wiedld
Copy link
Collaborator Author

wiedld commented Dec 19, 2024 •
edited
Loading

NEWLY ADDED: our sqllogictests for string LIKE are now matching the updated output after the arrow-rs 53.3.0 upgrade. See commit with message and link: 7a0dd9d

Make sure to pay attention to this newly added patch: 7a0dd9d

@2010YOUY01 @wiedld
Add generate_series() udtf (and introduce 'lazy' MemoryExec) (apach…
d01ae57 
…e#13540)

* Add generate_series() udtf

* liscence

* fix examples

* clippy

* comments

* singleton udtf init

* StreamingMemoryExec -> LazyMemoryExec

* use RwLock

* test udf+udtf generate_series() in the same sql

* CI

* CI

* small fixes
@wiedld wiedld force-pushed the chore/df-upgrade-nov-11-patched branch from d058adf to d01ae57 Compare December 19, 2024 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
catalog common core documentation Improvements or additions to documentation execution functions logical-expr optimizer physical-expr sql sqllogictest
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wiedld @alamb @2010YOUY01