Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Add extra "expt-tracking" that uses mlflow #778

Open
2 tasks
NickleDave opened this issue Sep 28, 2024 · 1 comment
Open
2 tasks

ENH: Add extra "expt-tracking" that uses mlflow #778

NickleDave opened this issue Sep 28, 2024 · 1 comment
Assignees
Labels
ENH: enhancement enhancement; new feature or request

Comments

@NickleDave
Copy link
Collaborator

NickleDave commented Sep 28, 2024

  • pip install vak[expt-tracking] should add this optional dependency and then allow experiment tracking
  • we should include at a bare minimum the path to metadata json files from datasets so that one can programatically relate metadata to experiments / runs

I don't want to invest a lot right now in supporting every feature of mlflow but it's clear to me we need some sort of experiment tracking (as @achabotl wisely pointed out many moons ago).

Prefer going with mlflow for now since it's the one I'm most familiar with and I would rather just adopt something that's widely used in industry, although sth more lightweight and application-agnostic like sacred or sumatra might be worth thinking about

@NickleDave
Copy link
Collaborator Author

NickleDave commented Sep 28, 2024

Just realized lightning has an mlflow logger
https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.mlflow.html#mlflow-logger
so we can perhaps somewhat painlessly integrate that way -- pass in experiment name, run name, etc

edit: looking at this again and related issues, questions in the Discussions, etc., my impression is that integration is not perfect and it's probably less painful in the long run to keep them separated if possible

edit edit: basically I need to implement the bad version of this and see if it can even work

@NickleDave NickleDave self-assigned this Oct 25, 2024
@NickleDave NickleDave added the ENH: enhancement enhancement; new feature or request label Oct 25, 2024
@NickleDave NickleDave moved this to Todo in vak-1.1 Oct 25, 2024
@NickleDave NickleDave changed the title ENH: Add experiment tracking with mlflow as an extra ENH: Add extra "expt-tracking" that uses mlflow Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ENH: enhancement enhancement; new feature or request
Projects
Status: Todo
Development

No branches or pull requests

1 participant