-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PSA: Improved graph construction time for calls including from_delayed
with dask-expr>=1.1.13
#52
Comments
@hendrikmakait. Thank you, I do confirm, that scheduling time (time between I hit |
That's great to hear! Please let us know if you notice any other performance bottlenecks. 12 minutes may still be somewhat long, depending on the number of total tasks. |
@hendrikmakait 12 minutes is significant amount of time, the "run time" was ~90 minutes on 60 3-thread workers, you can find more details in this notebook It is also worth to mention that the manager node used ~90GB of memory. The task graph had ~0.5M tasks. When I run a more complex notebook, "lazy cells" (where I plan dask computations before |
Is there a public dataset that one could access to reproduce this? If not, I recommend profiling your code with py-spy. We could do a sanity check if you can provide a profile in the |
@hendrikmakait The data is public, I made a notebook which uses the HTTPS data storage. The dataset volume is pretty large, but graph building part doesn't really touch any data files, so the manager node issues are reproducible without fetching any actual data files. The NB: It took 90GB of RAM and 7 minutes to schedule the graph. (Do I use the right terminology here? What I mean is the time between I hit I haven't tried |
@hendrikmakait I ran the code (up to the Here you can find the code and outputs for both profilers: I'd really appreciate it if you could take a look. I believe you would have much better insight into this! Update 2024.09.12 I tried to turn off optimizations with |
@hendrikmakait just a gentle reminder, could you please give a look to these profiler logs? Should I convert it to a Dask/ |
Thanks for creating the profiles! I've had a brief look at them and there's nothing that would point toward an obvious issue at first glance. I'm not sure if I'll have any time soon to dig deeper into this. It looks like I'd have to understand the graph structures you create in much more detail to point out issues. |
During today's Dask monthly meeting, people mentioned exceedingly long graph construction times related to delayed objects. I see that you are using
from_delayed
here. Please note thatdask-expr>=1.1.13
includes a fix that improves graph construction time for large collections of delayed objects indd.from_delayed
by orders of magnitude (dask/dask-expr#1132).I don't know your specific problem, so I'm not sure if this will fix it, but I wanted to point it out nonetheless in case it helps.
The text was updated successfully, but these errors were encountered: