Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove slow timing comparsion #425

Merged
merged 2 commits into from
Apr 4, 2024
Merged

remove slow timing comparsion #425

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e1ee53b
remove slow timing comparsion
dougbrn Apr 4, 2024
febcac6
Merge branch 'main' into remove_timing_comparison
dougbrn Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 1 addition & 70 deletions docs/tutorials/using_ray_with_the_ensemble.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"source": [
"# Using Dask on Ray with the Ensemble\n",
"\n",
"[Ray](https://docs.ray.io/en/latest/ray-overview/index.html) is an open-source unified framework for scaling AI and Python applications. Ray provides a scheduler for Dask ([dask_on_ray](https://docs.ray.io/en/latest/ray-more-libs/dask-on-ray.html)) which allows you to build data analyses using Dask’s collections and execute the underlying tasks on a Ray cluster. We have found with TAPE that the Ray scheduler is often more performant than Dasks scheduler. Ray can be used on TAPE using the setup shown in the following example."
"[Ray](https://docs.ray.io/en/latest/ray-overview/index.html) is an open-source unified framework for scaling AI and Python applications. Ray provides a scheduler for Dask ([dask_on_ray](https://docs.ray.io/en/latest/ray-more-libs/dask-on-ray.html)) which allows you to build data analyses using Dask’s collections and execute the underlying tasks on a Ray cluster. Ray can be used on TAPE using the setup shown in the following example."
]
},
{
Expand Down Expand Up @@ -86,75 +86,6 @@
" calc_sf2, use_map=False\n",
") # use_map is false as we repartition naively, splitting per-object sources across partitions"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c5692d75",
"metadata": {},
"source": [
"## Timing Comparison\n",
"\n",
"As mentioned above, we generally see that Ray is more performant than Dask. Below is a simple timing comparison."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f128cdbf",
"metadata": {},
"source": [
"### Ray Timing"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dd960e10",
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"\n",
"ens = Ensemble(client=False) # Do not use a client\n",
"ens.from_dataset(\"s82_qso\", sorted=True)\n",
"ens.source = ens.source.repartition(npartitions=10)\n",
"ens.batch(calc_sf2, use_map=False)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "228e5114",
"metadata": {},
"source": [
"### Dask Timing"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "24a8f466",
"metadata": {},
"outputs": [],
"source": [
"disable_dask_on_ray() # unsets the dask_on_ray configuration settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1552c2b8",
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"\n",
"ens = Ensemble()\n",
"ens.from_dataset(\"s82_qso\", sorted=True)\n",
"ens.source = ens.source.repartition(npartitions=10)\n",
"ens.batch(calc_sf2, use_map=False).compute()"
]
}
],
"metadata": {
Expand Down