Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GSProcessing] Fix bug in repartition on leader, fix mypy errors and latent bugs #915

Merged
merged 2 commits into from
Jul 11, 2024
Merged

[GSProcessing] Fix bug in repartition on leader, fix mypy errors and latent bugs #915

merged 2 commits into from
Jul 11, 2024

Conversation

thvasilo
Copy link
Contributor

Issue #, if available:

Description of changes:

  • During the refactor with [GSProcessing] Add pre-computed categorical transformation loading #870 we moved the loader to be a class var for DistributedExecutor, but because the S3 path is not unit tested we missed on case where the output is on S3 and the user requests repartition on leader.
  • This error was actually picked up by mypy, I fixed some other potential issues and type annotations here.

Testing

Pre-commit, unit tests, and one test SageMaker job all succeed.

The S3 codepath can only be integration tested currently.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@thvasilo thvasilo added ready able to trigger the CI gsprocessing For issues and PRs related the the GSProcessing library 0.3.1 labels Jul 10, 2024
@thvasilo thvasilo added this to the 0.3.1 milestone Jul 10, 2024
@thvasilo thvasilo requested a review from jalencato July 10, 2024 02:02
@thvasilo thvasilo self-assigned this Jul 10, 2024
jalencato
jalencato reviewed Jul 10, 2024
View reviewed changes
Copy link
Collaborator

@jalencato jalencato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small comment

graphstorm-processing/graphstorm_processing/distributed_executor.py Show resolved Hide resolved
graphstorm-processing/graphstorm_processing/repartition_files.py Show resolved Hide resolved
graphstorm-processing/graphstorm_processing/graph_loaders/dist_heterogeneous_loader.py Show resolved Hide resolved
jalencato
jalencato approved these changes Jul 10, 2024
View reviewed changes
Copy link
Collaborator

@jalencato jalencato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thvasilo thvasilo merged commit d5b1e03 into awslabs:main Jul 11, 2024
3 checks passed
@thvasilo thvasilo deleted the gsp-fix-local-repart branch July 11, 2024 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jalencato jalencato jalencato approved these changes

Assignees

@thvasilo thvasilo

Labels
0.3.1 gsprocessing For issues and PRs related the the GSProcessing library ready able to trigger the CI
Projects
None yet
Milestone
0.3.1
Development

Successfully merging this pull request may close these issues.
2 participants
@thvasilo @jalencato