Skip to content

Commit

Permalink
Changing the class names of LH5Store and LH5Iterator to Store a…
Browse files Browse the repository at this point in the history
…nd `Iterator`
  • Loading branch information
MoritzNeuberger committed Oct 30, 2023
1 parent 51a34db commit a08c535
Show file tree
Hide file tree
Showing 10 changed files with 85 additions and 85 deletions.
20 changes: 10 additions & 10 deletions docs/source/notebooks/LH5Files.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@
"id": "ef09f43c",
"metadata": {},
"source": [
"The group contains several LGDOs. Let's read them in memory. We start by initializing an `LH5Store` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.LH5Store) object:"
"The group contains several LGDOs. Let's read them in memory. We start by initializing an `Store` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.Store) object:"
]
},
{
Expand All @@ -111,17 +111,17 @@
"metadata": {},
"outputs": [],
"source": [
"from lgdo import LH5Store\n",
"from lgdo import Store\n",
"\n",
"store = LH5Store()"
"store = Store()"
]
},
{
"cell_type": "markdown",
"id": "60ce5b61",
"metadata": {},
"source": [
"`read_object()` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.LH5Store.read_object) reads an LGDO from disk and returns the object in memory together with the number of rows (as a tuple), if an object has such a property. Let's try to read `geds/raw`:"
"`read_object()` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.Store.read_object) reads an LGDO from disk and returns the object in memory together with the number of rows (as a tuple), if an object has such a property. Let's try to read `geds/raw`:"
]
},
{
Expand Down Expand Up @@ -200,7 +200,7 @@
"id": "b3f52d77",
"metadata": {},
"source": [
"As you might have noticed, `read_object()` loads all the requested data in memory at once. This can be a problem when dealing with large datasets. `LH5Iterator` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.LH5Iterator) makes it possible to handle data one chunk at a time (sequentially) to avoid running out of memory:"
"As you might have noticed, `read_object()` loads all the requested data in memory at once. This can be a problem when dealing with large datasets. `Iterator` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.Iterator) makes it possible to handle data one chunk at a time (sequentially) to avoid running out of memory:"
]
},
{
Expand All @@ -210,9 +210,9 @@
"metadata": {},
"outputs": [],
"source": [
"from lgdo import LH5Iterator\n",
"from lgdo import Iterator\n",
"\n",
"for lh5_obj, entry, n_rows in LH5Iterator(lh5_file, \"geds/raw/energy\", buffer_len=20):\n",
"for lh5_obj, entry, n_rows in Iterator(lh5_file, \"geds/raw/energy\", buffer_len=20):\n",
" print(f\"entry {entry}, energy = {lh5_obj} ({n_rows} rows)\")"
]
},
Expand Down Expand Up @@ -248,7 +248,7 @@
"id": "cbfd91c2",
"metadata": {},
"source": [
"The `write_object()` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.LH5Store.write_object) method of `LH5Store` makes it possible to write LGDO objects on disk. Let's start by writing `scalar` with name `message` in a file named `my_data.lh5` in the current directory:"
"The `write_object()` [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.Store.write_object) method of `Store` makes it possible to write LGDO objects on disk. Let's start by writing `scalar` with name `message` in a file named `my_data.lh5` in the current directory:"
]
},
{
Expand All @@ -258,7 +258,7 @@
"metadata": {},
"outputs": [],
"source": [
"store = LH5Store()\n",
"store = Store()\n",
"\n",
"store.write_object(\n",
" scalar, name=\"message\", lh5_file=\"my_objects.lh5\", wo_mode=\"overwrite_file\"\n",
Expand Down Expand Up @@ -316,7 +316,7 @@
"\n",
"<div class=\"alert alert-info\">\n",
"\n",
"**Note:** `write_objects()` allows for more advanced usage, like writing only some rows of the input object or appending to existing array-like structures. Have a look at the [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.LH5Store.write_object) for more information.\n",
"**Note:** `write_objects()` allows for more advanced usage, like writing only some rows of the input object or appending to existing array-like structures. Have a look at the [[docs]](https://legend-pydataobj.readthedocs.io/en/stable/api/lgdo.html#lgdo.lh5_store.Store.write_object) for more information.\n",
"\n",
"</div>"
]
Expand Down
8 changes: 4 additions & 4 deletions src/lgdo/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,13 @@
types with the same length (number of rows)
Currently the primary on-disk format for LGDO object is LEGEND HDF5 (LH5) files. IO
is done via the class :class:`.lh5_store.LH5Store`. LH5 files can also be
is done via the class :class:`.lh5_store.Store`. LH5 files can also be
browsed easily in python like any `HDF5 <https://www.hdfgroup.org>`_ file using
`h5py <https://www.h5py.org>`_.
"""

from ._version import version as __version__
from .lh5 import LH5Iterator, LH5Store, load_dfs, load_nda, ls, show
from .lh5 import Iterator, Store, load_dfs, load_nda, ls, show
from .types import (
LGDO,
Array,
Expand All @@ -66,8 +66,8 @@
"VectorOfVectors",
"VectorOfEncodedVectors",
"WaveformTable",
"LH5Iterator",
"LH5Store",
"Iterator",
"Store",
"load_dfs",
"load_nda",
"ls",
Expand Down
10 changes: 5 additions & 5 deletions src/lgdo/lh5/__init__.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
"""Routines from reading and writing LEGEND Data Objects in HDF5 files.
Currently the primary on-disk format for LGDO object is LEGEND HDF5 (LH5) files. IO
is done via the class :class:`.store.LH5Store`. LH5 files can also be
is done via the class :class:`.store.Store`. LH5 files can also be
browsed easily in python like any `HDF5 <https://www.hdfgroup.org>`_ file using
`h5py <https://www.h5py.org>`_.
"""

from .iterator import LH5Iterator
from .store import LH5Store, load_dfs, load_nda, ls, show
from .iterator import Iterator
from .store import Store, load_dfs, load_nda, ls, show

__all__ = [
"LH5Iterator",
"LH5Store",
"Iterator",
"Store",
"load_dfs",
"load_nda",
"ls",
Expand Down
18 changes: 9 additions & 9 deletions src/lgdo/lh5/iterator.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@
import pandas as pd

from ..types import Array, Scalar, Struct, VectorOfVectors
from .store import LH5Store
from .store import Store
from .utils import expand_path

LGDO = Union[Array, Scalar, Struct, VectorOfVectors]


class LH5Iterator(Iterator):
class Iterator(Iterator):
"""
A class for iterating through one or more LH5 files, one block of entries
at a time. This also accepts an entry list/mask to enable event selection,
Expand All @@ -29,7 +29,7 @@ class LH5Iterator(Iterator):
This can also be used as an iterator:
>>> for lh5_obj, entry, n_rows in LH5Iterator(...):
>>> for lh5_obj, entry, n_rows in Iterator(...):
>>> # do the thing!
This is intended for if you are reading a large quantity of data but
Expand All @@ -48,7 +48,7 @@ def __init__(
entry_mask: list[bool] | list[list[bool]] = None,
field_mask: dict[str, bool] | list[str] | tuple[str] = None,
buffer_len: int = 3200,
friend: LH5Iterator = None,
friend: Iterator = None,
) -> None:
"""
Parameters
Expand All @@ -68,16 +68,16 @@ def __init__(
mask of entries to read. If a list of arrays is provided, expect
one for each file. Ignore if a selection list is provided.
field_mask
mask of which fields to read. See :meth:`LH5Store.read_object` for
mask of which fields to read. See :meth:`Store.read_object` for
more details.
buffer_len
number of entries to read at a time while iterating through files.
friend
a ''friend'' LH5Iterator that will be read in parallel with this.
a ''friend'' Iterator that will be read in parallel with this.
The friend should have the same length and entry list. A single
LH5 table containing columns from both iterators will be returned.
"""
self.lh5_st = LH5Store(base_path=base_path, keep_open=True)
self.lh5_st = Store(base_path=base_path, keep_open=True)

# List of files, with wildcards and env vars expanded
if isinstance(lh5_files, str):
Expand Down Expand Up @@ -162,8 +162,8 @@ def __init__(

# Attach the friend
if friend is not None:
if not isinstance(friend, LH5Iterator):
raise ValueError("Friend must be an LH5Iterator")
if not isinstance(friend, Iterator):
raise ValueError("Friend must be an Iterator")
self.lh5_buffer.join(friend.lh5_buffer)
self.friend = friend

Expand Down
10 changes: 5 additions & 5 deletions src/lgdo/lh5/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,15 @@
DEFAULT_HDF5_COMPRESSION = None


class LH5Store:
class Store:
"""
Class to represent a store of LEGEND HDF5 files. The two main methods
implemented by the class are :meth:`read_object` and :meth:`write_object`.
Examples
--------
>>> from lgdo import LH5Store
>>> store = LH5Store()
>>> from lgdo import Store
>>> store = Store()
>>> obj, _ = store.read_object("/geds/waveform", "file.lh5")
>>> type(obj)
lgdo.waveform_table.WaveformTable
Expand Down Expand Up @@ -1214,7 +1214,7 @@ def ls(lh5_file: str | h5py.Group, lh5_group: str = "") -> list[str]:
+ ("" if lh5_group == "" else f" (and group {lh5_group})")
)

lh5_st = LH5Store()
lh5_st = Store()
# To use recursively, make lh5_file a h5group instead of a string
if isinstance(lh5_file, str):
lh5_file = lh5_st.gimme_file(lh5_file, "r")
Expand Down Expand Up @@ -1378,7 +1378,7 @@ def load_nda(
# Expand wildcards
f_list = [f for f_wc in f_list for f in sorted(glob.glob(os.path.expandvars(f_wc)))]

sto = LH5Store()
sto = Store()
par_data = {par: [] for par in par_list}
for ii, f in enumerate(f_list):
f = sto.gimme_file(f, "r")
Expand Down
2 changes: 1 addition & 1 deletion src/lgdo/lh5/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def parse_datatype(datatype: str) -> tuple[str, tuple[int, ...], str | list[str]
LGDO. Note this is not the same as the NumPy shape of the underlying
data object. See the LGDO specification for more information. Also see
:class:`~.types.ArrayOfEqualSizedArrays` and
:meth:`.lh5_store.LH5Store.read_object` for example code elements for
:meth:`.lh5_store.Store.read_object` for example code elements for
numeric objects, the element type for struct-like objects, the list of
fields in the struct.
"""
Expand Down
4 changes: 2 additions & 2 deletions tests/compression/conftest.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import pytest

from lgdo import LH5Store
from lgdo import Store


@pytest.fixture()
def wftable(lgnd_test_data):
store = LH5Store()
store = Store()
wft, _ = store.read_object(
"/geds/raw/waveform",
lgnd_test_data.get_path("lh5/LDQTA_r117_20200110T105115Z_cal_geds_raw.lh5"),
Expand Down
4 changes: 2 additions & 2 deletions tests/compression/test_radware_sigcompress.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import numpy as np

from lgdo import ArrayOfEncodedEqualSizedArrays, ArrayOfEqualSizedArrays, LH5Store
from lgdo import ArrayOfEncodedEqualSizedArrays, ArrayOfEqualSizedArrays, Store
from lgdo.compression.radware import (
_get_hton_u16,
_radware_sigcompress_decode,
Expand Down Expand Up @@ -168,7 +168,7 @@ def test_aoesa(wftable):


def test_performance(lgnd_test_data):
store = LH5Store()
store = Store()
obj, _ = store.read_object(
"/geds/raw/waveform",
lgnd_test_data.get_path("lh5/LDQTA_r117_20200110T105115Z_cal_geds_raw.lh5"),
Expand Down
14 changes: 7 additions & 7 deletions tests/lh5/test_lh5_iterator.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import pytest

import lgdo
from lgdo.lh5 import LH5Iterator
from lgdo.lh5 import Iterator


@pytest.fixture(scope="module")
Expand All @@ -11,7 +11,7 @@ def lgnd_file(lgnd_test_data):


def test_basics(lgnd_file):
lh5_it = LH5Iterator(
lh5_it = Iterator(
lgnd_file,
"/geds/raw",
entry_list=range(100),
Expand All @@ -35,14 +35,14 @@ def test_basics(lgnd_file):

def test_errors(lgnd_file):
with pytest.raises(RuntimeError):
LH5Iterator("non-existent-file.lh5", "random-group")
Iterator("non-existent-file.lh5", "random-group")

with pytest.raises(ValueError):
LH5Iterator(1, 2)
Iterator(1, 2)


def test_lgnd_waveform_table_fancy_idx(lgnd_file):
lh5_it = LH5Iterator(
lh5_it = Iterator(
lgnd_file,
"geds/raw/waveform",
entry_list=[
Expand Down Expand Up @@ -97,13 +97,13 @@ def more_lgnd_files(lgnd_test_data):


def test_friend(more_lgnd_files):
lh5_raw_it = LH5Iterator(
lh5_raw_it = Iterator(
more_lgnd_files[0],
"ch1084803/raw",
field_mask=["waveform", "baseline"],
buffer_len=5,
)
lh5_it = LH5Iterator(
lh5_it = Iterator(
more_lgnd_files[1],
"ch1084803/hit",
field_mask=["is_valid_0vbb"],
Expand Down
Loading

0 comments on commit a08c535

Please sign in to comment.