Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid local functions in push #9856

Merged
merged 3 commits into from
Dec 5, 2024

Conversation

fjetter
Copy link
Contributor

@fjetter fjetter commented Dec 5, 2024

See dask/dask#11576 for the dask sibling

Using local functions is rather expensive to serialize because the function body has to be serialized instead of merely a reference. That's exacerbated if the function is accessing non-local state since that requires the serialization of the entire context that function is referring to. It is therefore always better for performance to define these functions, regardless of how small they are, as global, top-level functions such that pickle works properly.
The same is true for lambdas which have the same problem.

I'm looking into whether we can have automated tests for this (difficult since our and likely your tests suite is using locals all over the place...)

Copy link

welcome bot commented Dec 5, 2024

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

@fjetter
Copy link
Contributor Author

fjetter commented Dec 5, 2024

I think the test failures are unrelated (e.g. https://github.com/pydata/xarray/actions/runs/12170678414/job/33946120091)

@dcherian dcherian merged commit eac5105 into pydata:main Dec 5, 2024
29 checks passed
Copy link

welcome bot commented Dec 5, 2024

Congratulations on completing your first pull request! Welcome to Xarray! We are proud of you, and hope to see you again! celebration gif

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants