[Doc] Add Doc String for graphstorm.dataloading.dataloading #471
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GentleZhu
changed the title
*Description of changes:* Fix an issue in logging and print out more information. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Ubuntu <[email protected]>
*Issue #, if available:* *Description of changes:* This PRs implements writing GNN embeddings from inference in stream. Unit test has been added. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Houyu Zhang <[email protected]> Co-authored-by: Da Zheng <[email protected]>
*Issue #, if available:* when using precision_recall as evaluation metric, we need to set `return_proba=True` because scikit-learn's precision_recall_curve takes probability as input. *Description of changes:* By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: xiang song(charlie.song) <[email protected]>
*Issue #, if available: Feature transformation of inference graphs should follow the same transformation rule as the corresponding training graph. However, for min/max transformation, the rule is not saved. Thus when processing inference graphs, gconstruct can not load the transformation info. *Description of changes:* Fix the issue. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Xiang Song <[email protected]> Co-authored-by: Da Zheng <[email protected]>
*Issue #, if available:* *Description of changes:* Disable storing embeddings when inferring on ec/er task By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: xiang song(charlie.song) <[email protected]>
*Issue #, if available:* *Description of changes:* * Improve the way we handle unknown config versions with more descriptive messages and fall-back parsing. * If a version name is not recognized we'll first try to parse the file as GConstruct. If that fails we assume GSProcessing input. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
zhjwy9343
reviewed
Sep 26, 2023
*Issue #, if available:* *Description of changes:* Updated MAG example with GLEM instructions By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
*Description of changes:* We should allow users to customize the model layers to load in the inference script. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. Co-authored-by: Ubuntu <[email protected]>
*Issue #, if available:*
*Description of changes:*
This PR reorganized GS API documentation in the following perspectives:
- Put all Base classes into their corresponding module;
- Add docstring for the graphstorm module;
- Create individual template.rst for each module so that they only
display their specific APIs;
- Fix the .. code:: directive error in the GSgnnModelBase class.
By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
---------
Co-authored-by: Ubuntu <[email protected]>
Co-authored-by: xiang song(charlie.song) <[email protected]>
*Issue #, if available:* *Description of changes:* By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Xiang Song <[email protected]> Co-authored-by: Israt Nisa <[email protected]>
… sortable. (#479) *Issue #, if available:* The saved embeddings or prediction results are named as emb-0.pt, emb-1.pt, .... emb-10.pt ... When there are more than 10 workers in the graph, the output files can not be easily sorted. *Description of changes:* We pad file index with extra zeros, so that the saved files will be named as emb-00000.pt, emb-00001.pt, etc. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Xiang Song <[email protected]>
… to do validation. (#489) Originally, if a user do not want to do validation, the graph should not contain test_mask, otherwise GraphStorm will crash. This PR will allow this use scenario. In certain cases, GraphStorm will only do inference on the test set. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: Xiang Song <[email protected]>
*Description of changes:* In the GNN inference with BERT models, we first cache the BERT embeddings and then cache the embeddings from the input layer. This causes a lot of unnecessary memory consumption. An easy to avoid this is to remove caching the BERT embeddings in the inference script. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. Co-authored-by: Ubuntu <[email protected]>
*Issue #, if available:* *Description of changes:* * We use `dst:relation-rev:src` for reverse edge names instead of `dst:rev-relation:src` to match GConstruct. *Testing*: * pytest and test job (with re-partition follow-up) succeed By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
`python3 -m pip install --upgrade prospector pip` has a change to install astroid==3.0 (do not know why) which is not compatible with lint. Fix astroid version to 2.15.7 in this PR. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. Co-authored-by: Xiang Song <[email protected]>
Docker file to create docker image which supports NVIDIA's WholeGraph in GraphStorm. This Dockerfile should install EFA and NCCL on top of the other prerequisites required by GraphStorm. --------- Co-authored-by: Israt Nisa <[email protected]> Co-authored-by: xiang song(charlie.song) <[email protected]>
This test is necessary to trigger for each graphstorm commit to avoid any further unexploited problem. Related to #491 By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
*Issue #, if available:* *Description of changes:* * Add documentation for GSProcessing By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. --------- Co-authored-by: xiang song(charlie.song) <[email protected]>
*Issue #, if available:* *Description of changes:* Add Doc String for dataloading.dataset By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
…into dataloader-doc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
Add Doc String for dataloaders in dataloading
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.