Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve neighbor allreduce #78

Merged
merged 40 commits into from
Apr 11, 2021
Merged

Improve neighbor allreduce #78

merged 40 commits into from
Apr 11, 2021

Conversation

hanbinhu
Copy link
Collaborator

No description provided.

BichengYing and others added 30 commits February 7, 2021 21:06
@BichengYing
Fixed the self_weight under emtpy receiving case
61b472c
Enable empty send neighbors and fix HalfTensor for recv_size==0
7e8ea66
@BichengYing
Fixed the self_weight under emtpy receiving case
1d43096
Enable empty send neighbors and fix HalfTensor for recv_size==0
c082ce9
Merge branch 'improve_neighbor_allreduce' of https://github.com/ybc19…
50933fb 
…91/bluefog into improve_neighbor_allreduce
Rename neighbor_weights to src_weights, and send_neighbors to dst_wei…
a3365b3 
…ghts for neighbor_allreduce
A script to test existing examples
cf17875
Accept dst_weights as Dict, and reorganize DoNeighborAllreduce
923df09
Reorganize CheckNeighborSendRecvPattern
42d0af7
Fix timeline_ptr for NCCL
3184e17
Fix timeline_ptr for NCCL
c85cbd5
Merge branch 'improve_neighbor_allreduce' of https://github.com/ybc19…
8d9fe31 
…91/bluefog into improve_neighbor_allreduce
Put dst_weights information into TensorTableEntry
7d60585
First Version of neighbor_allreduce dst_weight, existing problem: Fus…
79c57b3 
…ion Not Implemented, CUDA data_weight problem
Add some delay after data_weight as a temporary solution
29a239c
CPU Fusion for dst_weighted added
97517b3
Merge branch 'master' into improve_neighbor_allreduce
364f5fd
Add ReadyEvent for dst_weight for single entry neighbor_allreduce
cc9faf4
Remove const identifier for tensor dtype as it is meaningless
7f47ce4
Add cuda source for scalebuffer
d84138b
Scale buffer to modify itself
06c375d
Add .o file to .gitignore
e4db92f
dst_weight using CUDA for fused entry & compile flow in Python setup.py
ec2a3fc
make clean *.o files generated by nvcc
adaa191
Add fix for NCCL single entry
46e6431
Make setup.py more robust
ae57e75
Add timeout and cuda check
62dc99d
Move test example
a836b02
Fix NCCL side dst_weight fusion bug
e02a3d0
Add agg to make matplotlib more stable
8e42ca2
@hanbinhu hanbinhu added the enhancement New feature or request label Mar 18, 2021
@hanbinhu hanbinhu requested a review from BichengYing March 18, 2021 09:12
@hanbinhu hanbinhu self-assigned this Mar 18, 2021
BichengYing
BichengYing reviewed Mar 21, 2021
View reviewed changes
Copy link
Collaborator

@BichengYing BichengYing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please merge the master to add the ci.yml. And add more test into it?

bluefog/common/common.h Show resolved Hide resolved
bluefog/common/cuda/cuda_kernels.cu Show resolved Hide resolved
bluefog/common/mpi_context.h Show resolved Hide resolved
bluefog/common/mpi_controller.cc Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/torch/mpi_ops.py Show resolved Hide resolved
test/torch_ops_test.py Show resolved Hide resolved
bluefog/torch/mpi_ops.py Outdated Show resolved Hide resolved
bluefog/torch/mpi_ops.py Outdated Show resolved Hide resolved
bluefog/common/nccl_controller.cc Outdated Show resolved Hide resolved
BichengYing
BichengYing reviewed Mar 24, 2021
View reviewed changes
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/common.h Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/nccl_controller.cc Outdated Show resolved Hide resolved
bluefog/common/operations.cc Outdated Show resolved Hide resolved
bluefog/common/mpi_controller.cc Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/nccl_controller.cc Outdated Show resolved Hide resolved
@hanbinhu hanbinhu requested a review from BichengYing March 26, 2021 08:06
BichengYing
BichengYing reviewed Mar 27, 2021
View reviewed changes
bluefog/common/mpi_controller.cc Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/nccl_controller.cc Outdated Show resolved Hide resolved
bluefog/common/mpi_controller.cc Show resolved Hide resolved
bluefog/common/mpi_controller.cc Outdated Show resolved Hide resolved
bluefog/common/tensor_queue.h Outdated Show resolved Hide resolved
bluefog/torch/mpi_ops.cc Outdated Show resolved Hide resolved
bluefog/torch/mpi_ops.cc Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
Hanbin Hu and others added 2 commits March 27, 2021 12:49
Address review comments
96887b5
@BichengYing
Add condition variable to control the loop (#88)
36c073a 
* Add condition variable to control the loop

* Minor update on topology_setting in global_state

* Add missing <condition_variable> header

* Change cv.wait to cv.wait_for 10 seconds

* Address comment and remove adjusting resetVersionWinMem in ibfrun
@hanbinhu hanbinhu merged commit 2f696ed into master Apr 11, 2021
@hanbinhu hanbinhu deleted the improve_neighbor_allreduce branch April 11, 2021 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BichengYing BichengYing BichengYing left review comments

Assignees

@hanbinhu hanbinhu

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hanbinhu @BichengYing