Integrate renumbering and compression to `cugraph-dgl` to accelerate MFG creation #3887

tingyu66 · 2023-09-26T19:11:33Z

Allow cugraph-dgl dataloader to consume sampled outputs from BulkSampler in CSC format.

…ctions.hpp

…ugraph-sample-convert

seunghwak · 2023-09-27T16:52:59Z

I think all the C/C++ updates in this PR is from #3841, this won't require C/C++ review once #3841 gets merged to the main branch.

seunghwak

This has no C/C++ updates and approving without review.

tingyu66 · 2023-10-02T11:20:14Z

Using ogbn-products dataset, the time spent on MFG creation (excluding IO) for each epoch is reduced from 4.34 s to 2.95 s. The script used for the comparison is here. CC: @VibhuJawa

batch_size: 128, batches_per_partition: 50.

COO

CSC

VibhuJawa

Some minor changes but mostly looks great. Thanks again @tingyu66 for the great work here.

VibhuJawa · 2023-10-02T18:12:56Z

python/cugraph-dgl/cugraph_dgl/dataloading/dataloader.py

        batch_size: int = 1024,
        drop_last: bool = False,


@tingyu66 , Can we also change seeds_per_call to 100_000 to make a better default based on your testing ?

Should we change it to the default value of BulkSampler: 200_000? After our call the other day, I tested a wide range of seeds_per_call values and none of the runs threw a OOM error.

Interesting, if it just works we can probably set the default to None and let upstream handle it ? What do you think, any default which is reasonable and just works is fine by me.

I updated the value to 200_000 in 45f93f2 to align with BulkSampler. Did not set to None to avoid the extra step of handling None case.

VibhuJawa · 2023-10-02T18:18:43Z

python/cugraph-dgl/cugraph_dgl/dataloading/utils/sampling_helpers.py

+    major_offsets = df.major_offsets.dropna().values
+    label_hop_offsets = df.label_hop_offsets.dropna().values
+    renumber_map_offsets = df.renumber_map_offsets.dropna().values
+    renumber_map = df.map.dropna().values
+    minors = df.minors.dropna().values


I think this assumes that the length of the renumber_map is smaller than major_offsets. I will check this again if possible.

I don't think we are making any assumptions here. renumber_map and major_offsets are simply two different tensors that happen to be stored in a single df.

Ahh, My bad I just re-read the code below and I think we should be fine.

For each batch:

The length of renumber_map = number of distinct nodes (i.e., the number of sources in hop 0)

The length of major_offsets = sum of number of destination nodes in each hop

alexbarghi-nv

👍

VibhuJawa · 2023-10-02T19:51:59Z

Using ogbn-products dataset, the time spent on MFG creation (excluding IO) for each epoch is reduced from 4.34 s to 2.95 s. The script used for the comparison is here. CC: @VibhuJawa

Curious: What is the batch size being tested here , @tingyu66 ?

tingyu66 · 2023-10-02T20:04:52Z

Using ogbn-products dataset, the time spent on MFG creation (excluding IO) for each epoch is reduced from 4.34 s to 2.95 s. The script used for the comparison is here. CC: @VibhuJawa

Curious: What is the batch size being tested here , @tingyu66 ?

batch_size: 128, batches_per_partition: 50.

VibhuJawa

LGTM

rlratzel · 2023-10-03T17:45:09Z

/merge

seunghwak and others added 30 commits August 21, 2023 18:32

move sampling relatd functions in graph_functions.hpp to sampling_fun…

f0e9f1f

…ctions.hpp

draft sampling post processing function APIs

3b1fd23

API updates

67f4d7b

API updates

8f521d2

deprecate the existing renumber_sampeld_edgelist function

da3da9b

combine renumber & compression/sorting functions

0b87ee1

minor documentation updates

9b5950b

mionr documentation updates

5fbb177

deprecate the existing sampling output renumber function

b9611ab

initial implementation of sampling post processing

c3ee02b

cuda::std::atomic=>cuda::atomic

04c9105

update API documentation

bdc840c

add additional input testing

8c304b3

replace testing for sampling output post processing

b16a071

cosmetic updates

09a38d7

bug fixes

82ad8e4

Merge branch 'fea_mfg' of https://github.com/seunghwak/cugraph into c…

d99b512

…ugraph-sample-convert

the c api

c15d580

fix compile errors

9135629

reformat

dfd1cb7

rename test file from .cu to .cpp

6dfd4fe

bug fixes

7d5821f

add fill wrapper

58189ed

undo adding fill wrapper

39db98a

sampling test from .cpp to .cu

98c8e0a

fix a typo

c151f95

Merge branch 'branch-23.10' of github.com:rapidsai/cugraph into fea_mfg

fc5a4f0

do not return valid nzd vertices if doubly_compress is false

094aaf9

bug fix

cf57a6d

test code

2b48b7e

tingyu66 marked this pull request as draft September 26, 2023 19:11

tingyu66 added 3 commits September 26, 2023 16:13

cast to tensors, create list for minibatches

22217dc

infer n_hops, n_batches from df

7f838ae

enable csc loader

6531e14

tingyu66 self-assigned this Sep 27, 2023

BradReesWork added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Sep 27, 2023

BradReesWork requested review from VibhuJawa, alexbarghi-nv and seunghwak September 27, 2023 12:49

BradReesWork added this to the 23.10 milestone Sep 27, 2023

tingyu66 added 2 commits September 28, 2023 07:41

docstring

3a6b6b9

Merge branch 'branch-23.10' into dgl-mfg-integration

564ddb4

tingyu66 marked this pull request as ready for review September 28, 2023 16:21

seunghwak reviewed Sep 28, 2023

View reviewed changes

seunghwak approved these changes Sep 28, 2023

View reviewed changes

tingyu66 added 2 commits September 28, 2023 09:55

add test using karate dataset

9e73617

improve slicing

e9c8bbb

Merge branch 'branch-23.10' into dgl-mfg-integration

3620321

VibhuJawa suggested changes Oct 2, 2023

View reviewed changes

alexbarghi-nv approved these changes Oct 2, 2023

View reviewed changes

update seeds_per_call default value

45f93f2

VibhuJawa approved these changes Oct 2, 2023

View reviewed changes

rapids-bot bot merged commit a863835 into rapidsai:branch-23.10 Oct 3, 2023
70 checks passed

tingyu66 deleted the dgl-mfg-integration branch October 3, 2023 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate renumbering and compression to `cugraph-dgl` to accelerate MFG creation #3887

Integrate renumbering and compression to `cugraph-dgl` to accelerate MFG creation #3887

tingyu66 commented Sep 26, 2023 •

edited

Loading

seunghwak commented Sep 27, 2023

seunghwak left a comment

tingyu66 commented Oct 2, 2023 •

edited

Loading

VibhuJawa left a comment

VibhuJawa Oct 2, 2023

tingyu66 Oct 2, 2023

VibhuJawa Oct 2, 2023

tingyu66 Oct 2, 2023

VibhuJawa Oct 2, 2023

tingyu66 Oct 2, 2023

VibhuJawa Oct 2, 2023

tingyu66 Oct 2, 2023

alexbarghi-nv left a comment

VibhuJawa commented Oct 2, 2023

tingyu66 commented Oct 2, 2023

VibhuJawa left a comment

rlratzel commented Oct 3, 2023

Integrate renumbering and compression to cugraph-dgl to accelerate MFG creation #3887

Integrate renumbering and compression to cugraph-dgl to accelerate MFG creation #3887

Conversation

tingyu66 commented Sep 26, 2023 • edited Loading

seunghwak commented Sep 27, 2023

seunghwak left a comment

Choose a reason for hiding this comment

tingyu66 commented Oct 2, 2023 • edited Loading

COO

CSC

VibhuJawa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexbarghi-nv left a comment

Choose a reason for hiding this comment

VibhuJawa commented Oct 2, 2023

tingyu66 commented Oct 2, 2023

VibhuJawa left a comment

Choose a reason for hiding this comment

rlratzel commented Oct 3, 2023

Integrate renumbering and compression to `cugraph-dgl` to accelerate MFG creation #3887

Integrate renumbering and compression to `cugraph-dgl` to accelerate MFG creation #3887

tingyu66 commented Sep 26, 2023 •

edited

Loading

tingyu66 commented Oct 2, 2023 •

edited

Loading