Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EnsembleFrame Support to Tape #308

Merged
merged 50 commits into from
Dec 8, 2023
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
f8c420b
A minimal Dask Dataframe subclass for the Ensemble
wilsonbb Aug 25, 2023
740d2d7
Addressed comments, added test fixture.
wilsonbb Aug 28, 2023
d033607
Merge pull request #209 from lincc-frameworks/tape_ensemble_refactor_…
wilsonbb Aug 28, 2023
9a61392
Make convert_flux_to_mag part of the EnsembleFrame
wilsonbb Aug 29, 2023
17f9cc1
Merge pull request #212 from lincc-frameworks/tape_ensemble_refactor_…
wilsonbb Aug 29, 2023
72b8629
Ensembles can now track a group of labeled frames
wilsonbb Aug 30, 2023
1cd049e
Merge pull request #217 from lincc-frameworks/tape_ensemble_refactor_…
wilsonbb Aug 31, 2023
3128141
Preserve EnsembleFrame metadata after assign()
wilsonbb Aug 31, 2023
01a083a
Merge pull request #218 from lincc-frameworks/tape_ensemble_refactor_…
wilsonbb Aug 31, 2023
44ef76e
Merge pull request #219 from lincc-frameworks/main
wilsonbb Aug 31, 2023
423805a
Merge pull request #220 from lincc-frameworks/tape_ensemble_refactor
wilsonbb Aug 31, 2023
8db79e0
Parquet support for frame subclasses checkpoint
wilsonbb Sep 2, 2023
657a2a7
Reverting changes to tests
wilsonbb Sep 5, 2023
e8de263
Adds test for objsor_from_parquet
wilsonbb Sep 5, 2023
34e9bbd
Addressed comments
wilsonbb Sep 6, 2023
93abf4d
Removed adding column via apply
wilsonbb Sep 6, 2023
8c8e793
Fix comment typo
wilsonbb Sep 7, 2023
c586fa1
Merge pull request #222 from lincc-frameworks/tape_ensemble_refactor_…
wilsonbb Sep 7, 2023
068870a
Fix EnsembleFrame.set_index
wilsonbb Sep 19, 2023
db8a1ab
Add update_ensemble() and Use EnsembleFrames (#252)
wilsonbb Oct 5, 2023
13d507b
Propagate EnsembleFrame._is_dirty (#264)
wilsonbb Oct 6, 2023
578900f
Have update_frame mark frames as dirty (#267)
wilsonbb Oct 10, 2023
35de81c
Remove calls to set_dirty in ensemble (#269)
wilsonbb Oct 10, 2023
683c362
Update refactor (#274)
wilsonbb Oct 19, 2023
5a542f3
Merge main into tape_ensemble_refactor (#277)
wilsonbb Oct 24, 2023
0d4da10
Fix EnsembleFrame.set_dirty and map_partitions metadata propagation (…
wilsonbb Oct 27, 2023
c86d7ab
Ensemble.update_frame no longer infers if a frame is dirty by checkin…
wilsonbb Nov 1, 2023
a980349
Merge branch 'main' into big_merge
wilsonbb Nov 1, 2023
4b7d01e
Merge pull request #282 from lincc-frameworks/update_merge
wilsonbb Nov 1, 2023
0ce6f2e
Support storing batch results for custom meta (#285)
wilsonbb Nov 7, 2023
5ca0cc3
Update Remaining TAPE Documentation Notebooks for the Refactor (#298)
wilsonbb Nov 27, 2023
1dfa8df
Update Docs for TAPE EnsembleFrame Refactor (#290)
wilsonbb Nov 27, 2023
7e6abaf
Allow EnsembleFrame.compute to Trigger Object-Source Table Syncing (#…
wilsonbb Nov 28, 2023
a714b10
Add Explicit Metadata Propagation for EnsembleFrame joins (#301)
wilsonbb Nov 29, 2023
8535130
Merge branch 'main' into night_before_the_merge
wilsonbb Nov 29, 2023
d615572
Merge branch 'refactor_main' into night_before_the_merge
wilsonbb Nov 29, 2023
5bd7d6d
Merge branch 'main' into night_before_the_merge
wilsonbb Nov 29, 2023
8f8cc66
Update test
wilsonbb Dec 1, 2023
1b150eb
Merge branch 'main' into night_before_the_merge
wilsonbb Dec 1, 2023
5c847e1
Merge Main into Ensemble Refactor Branch (#304)
wilsonbb Dec 1, 2023
6779ba0
Revert "Merge Main into Ensemble Refactor Branch (#304)"
wilsonbb Dec 1, 2023
8154ee9
Merge pull request #306 from lincc-frameworks/revert-304-night_before…
wilsonbb Dec 1, 2023
3c4a484
Merge pull request #307 from lincc-frameworks/night_before_the_merge
wilsonbb Dec 1, 2023
494eefc
Fix linting
wilsonbb Dec 4, 2023
046942b
Remove unsupported type annotations
wilsonbb Dec 4, 2023
dd22e9e
Fix merge error
wilsonbb Dec 4, 2023
3de618f
Use client=False in test_analysis
wilsonbb Dec 5, 2023
d061b3c
Remove '_object' and '_source' fields
wilsonbb Dec 5, 2023
81bd28c
Fix linting errors
wilsonbb Dec 5, 2023
a415078
Address review comments, add tests
wilsonbb Dec 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/tutorials/binning_slowly_changing_sources.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
"outputs": [],
"source": [
"fig, ax = plt.subplots(1, 1)\n",
"ax.hist(ens._source[\"midPointTai\"].compute().tolist(), 500)\n",
"ax.hist(ens.source[\"midPointTai\"].compute().tolist(), 500)\n",
"ax.set_xlabel(\"Time (MJD)\")\n",
"ax.set_ylabel(\"Source Count\")"
]
Expand Down Expand Up @@ -90,7 +90,7 @@
"source": [
"ens.bin_sources(time_window=7.0, offset=0.0)\n",
"fig, ax = plt.subplots(1, 1)\n",
"ax.hist(ens._source[\"midPointTai\"].compute().tolist(), 500)\n",
"ax.hist(ens.source[\"midPointTai\"].compute().tolist(), 500)\n",
"ax.set_xlabel(\"Time (MJD)\")\n",
"ax.set_ylabel(\"Source Count\")"
]
Expand Down Expand Up @@ -120,7 +120,7 @@
"source": [
"ens.bin_sources(time_window=28.0, offset=0.0, custom_aggr={\"midPointTai\": \"min\"})\n",
"fig, ax = plt.subplots(1, 1)\n",
"ax.hist(ens._source[\"midPointTai\"].compute().tolist(), 500)\n",
"ax.hist(ens.source[\"midPointTai\"].compute().tolist(), 500)\n",
"ax.set_xlabel(\"Time (MJD)\")\n",
"ax.set_ylabel(\"Source Count\")"
]
Expand Down Expand Up @@ -150,7 +150,7 @@
"ens.from_source_dict(rows, column_mapper=cmap)\n",
"\n",
"fig, ax = plt.subplots(1, 1)\n",
"ax.hist(ens._source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.hist(ens.source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.set_xlabel(\"Time (MJD)\")\n",
"ax.set_ylabel(\"Source Count\")"
]
Expand Down Expand Up @@ -179,7 +179,7 @@
"ens.bin_sources(time_window=1.0, offset=0.0)\n",
"\n",
"fig, ax = plt.subplots(1, 1)\n",
"ax.hist(ens._source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.hist(ens.source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.set_xlabel(\"Time (MJD)\")\n",
"ax.set_ylabel(\"Source Count\")"
]
Expand Down Expand Up @@ -209,7 +209,7 @@
"ens.bin_sources(time_window=1.0, offset=0.5)\n",
"\n",
"fig, ax = plt.subplots(1, 1)\n",
"ax.hist(ens._source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.hist(ens.source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.set_xlabel(\"Time (MJD)\")\n",
"ax.set_ylabel(\"Source Count\")"
]
Expand Down Expand Up @@ -259,7 +259,7 @@
"ens.bin_sources(time_window=1.0, offset=0.5)\n",
"\n",
"fig, ax = plt.subplots(1, 1)\n",
"ax.hist(ens._source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.hist(ens.source[\"midPointTai\"].compute().tolist(), 60)\n",
"ax.set_xlabel(\"Time (MJD)\")\n",
"ax.set_ylabel(\"Source Count\")"
]
Expand Down Expand Up @@ -290,7 +290,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
"version": "3.10.13"
},
"vscode": {
"interpreter": {
Expand Down
4 changes: 2 additions & 2 deletions docs/tutorials/scaling_to_large_data.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@
"\n",
"print(\"number of lightcurve results in mapres: \", len(mapres))\n",
"print(\"number of lightcurve results in groupres: \", len(groupres))\n",
"print(\"True number of lightcurves in the dataset:\", len(np.unique(ens._source.index)))"
"print(\"True number of lightcurves in the dataset:\", len(np.unique(ens.source.index)))"
]
},
{
Expand Down Expand Up @@ -263,7 +263,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
"version": "3.10.13"
},
"vscode": {
"interpreter": {
Expand Down
4 changes: 2 additions & 2 deletions docs/tutorials/structure_function_showcase.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@
"metadata": {},
"outputs": [],
"source": [
"ens.head(\"object\", 5) \n"
"ens.object.head(5) \n"
]
},
{
Expand All @@ -276,7 +276,7 @@
"metadata": {},
"outputs": [],
"source": [
"ens.head(\"source\", 5) "
"ens.source.head(5) "
]
},
{
Expand Down
8 changes: 4 additions & 4 deletions docs/tutorials/tape_datasets.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
" column_mapper=col_map\n",
" )\n",
"\n",
"ens.head(\"source\") # View the first 5 entries of the source table"
"ens.source.head(5) # View the first 5 entries of the source table"
]
},
{
Expand Down Expand Up @@ -93,7 +93,7 @@
" column_mapper=col_map\n",
" )\n",
"\n",
"ens.head(\"object\") # View the first 5 entries of the object table"
"ens.object.head(5) # View the first 5 entries of the object table"
]
},
{
Expand Down Expand Up @@ -168,7 +168,7 @@
"source": [
"ens.from_dataset(\"s82_rrlyrae\") # Let's grab the Stripe 82 RR Lyrae\n",
"\n",
"ens.head(\"object\", 5)"
"ens.object.head(5)"
]
},
{
Expand Down Expand Up @@ -270,7 +270,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
"version": "3.10.13"
},
"vscode": {
"interpreter": {
Expand Down
8 changes: 4 additions & 4 deletions docs/tutorials/using_ray_with_the_ensemble.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@
"outputs": [],
"source": [
"ens.from_dataset(\"s82_qso\")\n",
"ens._source = ens._source.repartition(npartitions=10)\n",
"ens.source = ens.source.repartition(npartitions=10)\n",
"ens.batch(calc_sf2, use_map=False) # use_map is false as we repartition naively, splitting per-object sources across partitions"
]
},
Expand Down Expand Up @@ -116,7 +116,7 @@
"\n",
"ens=Ensemble(client=False) # Do not use a client\n",
"ens.from_dataset(\"s82_qso\")\n",
"ens._source = ens._source.repartition(npartitions=10)\n",
"ens.source = ens.source.repartition(npartitions=10)\n",
"ens.batch(calc_sf2, use_map=False)"
]
},
Expand Down Expand Up @@ -150,7 +150,7 @@
"\n",
"ens = Ensemble()\n",
"ens.from_dataset(\"s82_qso\")\n",
"ens._source = ens._source.repartition(npartitions=10)\n",
"ens.source = ens.source.repartition(npartitions=10)\n",
"ens.batch(calc_sf2, use_map=False)"
]
}
Expand All @@ -171,7 +171,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
"version": "3.10.13"
},
"vscode": {
"interpreter": {
Expand Down
Loading