Python Dataset Reader #2414
Python Dataset Reader #2414
Conversation
fiedorowicz1
force-pushed
the
distconv-python-data-reader
branch
from
eee429e to
40c516b
Compare
| @@ -0,0 +1,137 @@ | |||
| import os | |||
There was a problem hiding this comment.
Why not use the pytest version?
Would be nice to extend the test infrastructure (maybe at a later PR?)
There was a problem hiding this comment.
Can you clarify what you mean here?
There was a problem hiding this comment.
Certainly. What I meant is that our pytest version of the tests (i.e., @test_util.lbann_test) might have a nicer syntax here. However, it is designed to use the existing Python data reader and may be extended as such to use the dataset reader (for example, with a flag @test_util.lbann_test(dataset=MyObject()) or similar).
There was a problem hiding this comment.
I agree we should change the test framework to use the new python dataset reader, but I think that should be a separate PR.
include/lbann/data_ingestion/readers/data_reader_python_dataset.hpp
Outdated
Show resolved
Hide resolved
fiedorowicz1
force-pushed
the
distconv-python-data-reader
branch
from
40c516b to
861931c
Compare
| send_rank = recv_rank = send_rank_count = recv_rank_count = 0; | ||
| uint64_t send_rank_max_count = | ||
| local_distconv_mb_size + (distconv_extra_samples > 0); | ||
| uint64_t recv_rank_max_count = local_mb_size + (extra_samples > 0); |
There was a problem hiding this comment.
@fiedorowicz1 this looks like an Alltoall collective. Do you think using it would be more effective instead of the loop?
Co-authored-by: Tal Ben-Nun <[email protected]>
fiedorowicz1
force-pushed
the
distconv-python-data-reader
branch
from
e0cf5f2 to
75aac4e
Compare
Implements a new python data reader that is cleaner, more general, and supports DistConv. It also supports parallel pre-fetching of multiple mini-batches.