Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix function name change from nemo #71

Merged
merged 1 commit into from
Jan 9, 2024
Merged

Conversation

gshennvm
Copy link
Collaborator

@gshennvm gshennvm commented Jan 5, 2024

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

change function name because of NeMo PR

@github-actions github-actions bot added the Utils label Jan 5, 2024
@gshennvm gshennvm mentioned this pull request Jan 5, 2024
8 tasks
@gshennvm gshennvm self-assigned this Jan 5, 2024
@gshennvm gshennvm requested a review from odelalleau January 5, 2024 18:48
@gshennvm
Copy link
Collaborator Author

gshennvm commented Jan 5, 2024

potential issue with this PR is that it will make #47 not work unless that container pins nemo main

@odelalleau
Copy link
Collaborator

potential issue with this PR is that it will make #47 not work unless that container pins nemo main

We need to be clear on expectations. Here are my thoughts:

  • Users need to be able to easily setup an official version by following instructions in the main branch's README.md
  • Developers need to be able to easily setup the main branch's latest code following instructions in the main branch's CONTRIBUTING.md

I haven't looked closely at #47 yet and I'm not sure whether it's meant to serve users, developers, or both.

My main concern with merging this PR is it may prevent devs from using the latest main branch if we don't provide setup instructions somewhere. Just grabbing latest NeMo's main may or may not work. I think CONTRIBUTING.md should document the development setup, and we should keep it updated. Do you have other ideas on how to handle this?

@gshennvm
Copy link
Collaborator Author

gshennvm commented Jan 8, 2024

potential issue with this PR is that it will make #47 not work unless that container pins nemo main

We need to be clear on expectations. Here are my thoughts:

  • Users need to be able to easily setup an official version by following instructions in the main branch's README.md
  • Developers need to be able to easily setup the main branch's latest code following instructions in the main branch's CONTRIBUTING.md

I haven't looked closely at #47 yet and I'm not sure whether it's meant to serve users, developers, or both.

My main concern with merging this PR is it may prevent devs from using the latest main branch if we don't provide setup instructions somewhere. Just grabbing latest NeMo's main may or may not work. I think CONTRIBUTING.md should document the development setup, and we should keep it updated. Do you have other ideas on how to handle this?

Agree with the above concerns. Just to summarize the concrete plan(let me know if you think we can do this better!).

code dev flow

  • there will be 2 working branches: main and CI on aligner. Main will pin nemo version and only upgrade when nemo releases something stable which is every 2 months. The CI branch will track nemo main. Before every aligner release we will do a diff on CI branch and main and pick the CI branch(which i suspect have less changes) onto nemo aligner main.
  • we guarantee that users can build their own container using the dockerfile and nemo-aligner main will work with the dockerfile in Public Dockerfile #47. This dockerfile will then be upgraded every other month.

PRs

  • for stability PRs will be submitted to nemo-aligner main that uses a stale nemo version. But it is very important for the CI to be setup on any feature additions because when we cherry pick from CI -> main we need those tests to guard for convergence issues.

documentation

  • agree with your suggestion. README will install stable nemo with our dockerfile. CONTRIBUTING will be dev setup.

How do these sound?

@odelalleau
Copy link
Collaborator

How do these sound?

Sounds good to me! Just a couple of questions to be sure I got the full picture:

  1. I assume bringing PRs from main to ci will be automated, but what will happen in case of conflict?
  2. Where are CI results reported btw? (in other words, if a PR on main breaks the CI, where do we see it?)

@gshennvm
Copy link
Collaborator Author

gshennvm commented Jan 8, 2024

How do these sound?

Sounds good to me! Just a couple of questions to be sure I got the full picture:

  1. I assume bringing PRs from main to ci will be automated, but what will happen in case of conflict?
  2. Where are CI results reported btw? (in other words, if a PR on main breaks the CI, where do we see it?)
  1. In case of conflict I think we need to find the contributor for resolution. Normally NeMo changes all relate to renaming things and changing some implementation of core functions. I don't expect there to be too many conflicts.
  2. I'll send you the info for the CI

@odelalleau
Copy link
Collaborator

  • In case of conflict I think we need to find the contributor for resolution. Normally NeMo changes all relate to renaming things and changing some implementation of core functions. I don't expect there to be too many conflicts.

Yeah me neither, I just wanted to understand the process. I'm not sure how this automation would work, and how someone would fix the conflicts when they occur. As long as you're confident it'll work smoothly that's good for me, it's just a bit of a grey area right now (for me).

@gshennvm
Copy link
Collaborator Author

gshennvm commented Jan 8, 2024

  • In case of conflict I think we need to find the contributor for resolution. Normally NeMo changes all relate to renaming things and changing some implementation of core functions. I don't expect there to be too many conflicts.

Yeah me neither, I just wanted to understand the process. I'm not sure how this automation would work, and how someone would fix the conflicts when they occur. As long as you're confident it'll work smoothly that's good for me, it's just a bit of a grey area right now (for me).

Yeah.. I need to look more into automating this. For the time being I need to do it manually so things don't randomly break.

It's also a bit of a grey area for me... But I guess this allows us to work on a stable nemo version so maybe it's worth seeing how it'll go

@gshennvm gshennvm changed the base branch from main to dev January 8, 2024 22:22
@gshennvm
Copy link
Collaborator Author

gshennvm commented Jan 8, 2024

I've pointed this MR to dev which will be the branch we run nightly CI with nemo main on, please let me know what you think.

Copy link
Collaborator

@odelalleau odelalleau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pointed this MR to dev which will be the branch we run nightly CI with nemo main on, please let me know what you think.

I think this should be good!

@gshennvm gshennvm merged commit 6414131 into dev Jan 9, 2024
5 checks passed
@gshennvm gshennvm deleted the geshen/fix_name_change branch January 9, 2024 06:20
gshennvm added a commit that referenced this pull request Jan 9, 2024
gshennvm added a commit that referenced this pull request Jan 29, 2024
gshennvm added a commit that referenced this pull request Feb 12, 2024
gshennvm added a commit that referenced this pull request Feb 28, 2024
gshennvm added a commit that referenced this pull request Feb 29, 2024
gshennvm added a commit that referenced this pull request Mar 4, 2024
* fix function name change from nemo (#71)

Signed-off-by: Gerald Shen <[email protected]>

* moving _str_to_dtype (#93)

Signed-off-by: Gerald Shen <[email protected]>

* change to call public str to dtype func (#94)

Signed-off-by: Gerald Shen <[email protected]>

* 0.2.0 version

Signed-off-by: Gerald Shen <[email protected]>

* update dockerfile

Signed-off-by: Gerald Shen <[email protected]>

* update comment on dockerfile

Signed-off-by: Gerald Shen <[email protected]>

* update to 0.3.0dev

Signed-off-by: Gerald Shen <[email protected]>

* address CHANGELOG from commit in main

Signed-off-by: Gerald Shen <[email protected]>

---------

Signed-off-by: Gerald Shen <[email protected]>
rohitrango pushed a commit to rohitrango/NeMo-Aligner that referenced this pull request Jun 25, 2024
* fix function name change from nemo (NVIDIA#71)

Signed-off-by: Gerald Shen <[email protected]>

* moving _str_to_dtype (NVIDIA#93)

Signed-off-by: Gerald Shen <[email protected]>

* change to call public str to dtype func (NVIDIA#94)

Signed-off-by: Gerald Shen <[email protected]>

* 0.2.0 version

Signed-off-by: Gerald Shen <[email protected]>

* update dockerfile

Signed-off-by: Gerald Shen <[email protected]>

* update comment on dockerfile

Signed-off-by: Gerald Shen <[email protected]>

* update to 0.3.0dev

Signed-off-by: Gerald Shen <[email protected]>

* address CHANGELOG from commit in main

Signed-off-by: Gerald Shen <[email protected]>

---------

Signed-off-by: Gerald Shen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants