Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Upgrade to DataFusion 44 #972

Closed
timsaucer opened this issue Dec 13, 2024 · 1 comment · Fixed by #973
Closed

chore: Upgrade to DataFusion 44 #972

timsaucer opened this issue Dec 13, 2024 · 1 comment · Fixed by #973
Labels
enhancement New feature or request

Comments

@timsaucer
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

DataFusion 44 is preparing for the next release. Let's start the upgrade process and document all changes we've needed to make to feed back to them as part of an upgrade guide.

Describe the solution you'd like

Keep a running branch with the upgrade differences and document in this issue what we've encountered as we go along. Ideally we will complete this evaluation before the datafusion release process ends so we can identify issues earlier.

@timsaucer timsaucer added the enhancement New feature or request label Dec 13, 2024
@timsaucer
Copy link
Contributor Author

timsaucer commented Dec 13, 2024

  1. The trait definition of ExecutionPlanProperties return type for output_ordering now is Option<&LexOrdering> instead of Option<&[PhysicalSortExpr]>. The recommendation is to use .into() to convert into the required output.
  2. find_df_window_func has now been removed from the logical_expr crate. If users require this functionality they can perform equivalent searches by either creating a default session context, or calling all_default_aggregate_functions and all_default_window_functions, or by searching their existing session context using ctx.udaf, ctx.window_functions, ctx.aggregate_functions, and so on.
  3. LogicalPlan::Prepare and LogicalPlan::Execute have been removed. We do not actively use these in datafusion-python and I didn't search further for figuring out migration strategy.
  4. to_substrait_plan and from_substrait_plan no longer take a session context. Rather they expect a SubstraitPlanningState which is implemented by SessionState. One recommended migration is to call ctx.ctx.state() and use a borrow of the returned SessionState.
  5. RuntimeConfig is now deprecated in favor of RuntimeEnvBuilder. The migration is fairly straightforward, except to create the RuntimeEnv you now call build().
  6. String concat of a string view and string returns a string view.
  7. date_part function returns int32 instead of float64.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant