Add the support of using WholeGraph distributed embedding to store/update sparse_emb

[chang-l]

Update on 12/11/2023:

Most added code is covered by unit tests, except the sparse optimizer step function in graphstorm. To compensate it and gain more confident for the integrated feature, I also added an independent unit test test_wg_sparse_opt.py, which has no dependency on graphstorm, to show the integrated wholegraph sparse optimizer can generate close enough embs vs. torch sparse adam optimizer.

Current code should be compatible now, ie., all unit tests should pass (except the fails due to #685). This PR is ready for review.

Updated todos:

• Test/debug for sparse_optimizer.step() to show the updates are consistent between WholeGraph vs. DistDGL vs. PyTorch
• Need to validate/benchmark an end2end test with learnable embeddings using WholeGraph
• Misc. questions/TODOs in the comments

---

Just like using WholeGraph distributed feature store for gathering, this PR tries to add the support of using WholeGraph to manage sparse_embs to unblock learnable embeddings with WholeGraph.

For now, it is still a draft PR. The working test only covers save/load sparse embeddings using WholeGraph.

Todos:

• Test/debug for sparse_optimizer.step() to show the updates are consistent between WholeGraph vs. DistDGL vs. PyTorch
• Still need to work on adding end2end test with learnable embeddings using WholeGraph (any missing pieces?)
• Misc. questions/TODOs in the comments

@zheng-da @classicsong @isratnisa @nvcastet @TristonC

[chang-l]

@classicsong Here is an example of yaml file change, w.r.t this PR, to add wholegraph support for learnable embeddings: 44e4643

Note I only updated file gsgnn_lp.py for link pred tasks.