Skip to content

Commit

Permalink
More changes
Browse files Browse the repository at this point in the history
  • Loading branch information
tapadipti committed Sep 14, 2023
1 parent 9631e60 commit 3f869dc
Show file tree
Hide file tree
Showing 3 changed files with 92 additions and 71 deletions.
3 changes: 2 additions & 1 deletion content/docs/studio/experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ terminal, etc. To quickly start tracking your experiments with Iterative Studio:

- Copy your
[DVC Studio token](/doc/studio/user-guice/account-and-billing#studio-access-token)
and configure your model training environment to use the token:
and
[configure your model training environment to use the token](/doc/studio/user-guide/projects-and-experiments/live-metrics-and-plots#set-up-an-access-token):

```cli
dvc config --global studio.token ***
Expand Down
6 changes: 3 additions & 3 deletions content/docs/studio/model-registry.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Manage models

Iterative Studio provides a model registry that you can access by clicking on
the `Models` tab. Iterative Studio uses your project's `dvc.yaml` files to
identify ML models and specially formatted Git tags to identify model versions
and stage assignments.
the `Models` tab. Your project's `dvc.yaml` files are used to identify ML models
and specially formatted Git tags are used to identify model versions and stage
assignments.

To quickly start tracking your models in the Iterative Studio model registry:

Expand Down
154 changes: 87 additions & 67 deletions content/docs/studio/user-guide/experiments/index.md
Original file line number Diff line number Diff line change
@@ -1,72 +1,36 @@
# Experiment tracking and management

You can submit your experiments from your favorite interface - whether it is
Jupyter Notebooks, a code editor or IDE like VS Code, the Python cli, the bash
terminal, etc.
Jupyter Notebooks, a code editor or IDE like [VS Code](/doc/vs-code-extension),
the Python cli, the bash terminal, etc. You can also [submit new experiments
from Iterative Studio][run-experiments].

To track and manage experiments,
[create a project](/doc/studio/uer-guide/experiments/create-a-project) in
Iterative Studio. A project is an interactive representation of the ML datasets,
hyperparameters, models and metrics defined in your Git repositories. These
values are configured in your project's `dvc.yaml` file. Additionally, live
metrics that you [send to Iterative Studio][live-metrics-and-plots] using
[DVCLive] are also included in the project.

Within a project, you can:

- [Explore all the details of the experiments][explore-ml-experiments] that you
have pushed to your Git repository.
- [Visualize the experiments][visualize] using plots and trend charts.
- [Compare experiments][compare].
- [Run new experiments][run-experiments], including on your own cloud instances,
directly from Iterative Studio.
- [Track live metrics and plots][live-metrics-and-plots] by sending them to
Iterative Studio by using [DVCLive].

[project settings]:
/doc/studio/user-guide/projects-and-experiments/configure-a-project
[when do you need project settings]:
/doc/studio/user-guide/projects-and-experiments/configure-a-project#scenarios-where-project-settings-are-required
[create multiple projects from a single git repository]:
/doc/studio/user-guide/projects-and-experiments/create-a-project#create-multiple-projects-from-a-single-git-repository
[explore ml experiments]:
/doc/studio/user-guide/projects-and-experiments/explore-ml-experiments
[create a team]: /doc/studio/user-guide/teams
[sign up for the **basic** or **enterprise** plan]:
/doc/studio/user-guide/change-team-plan-and-size
[make your projects public]:
/doc/studio/user-guide/projects-and-experiments/share-a-project
[train on the cloud, including on your own cloud infrastructure, and submit new experiments]:
/doc/studio/user-guide/projects-and-experiments/run-experiments
[live-metrics-and-plots]:
/doc/studio/user-guide/projects-and-experiments/live-metrics-and-plots
[dvclive]: /doc/dvclive
[monorepo]:
/doc/studio/user-guide/projects-and-experiments/configure-a-project#monorepo
To quickly start tracking your experiments with Iterative Studio, check out the
[experiments quick start guide](/doc/studio/experiments).

## Different ways to track experiments

Iterative Studio offers a few different ways to track your experiments.

We recommend combining all these methods in a single project, so that you can
easily track live and completed experiments.
You can track live as well as completed experiments in Iterative Studio. First,
click on `Add a project` to connect Iterative Studio to your ML project's Git
repository. Then, follow the instructions presented below.

### 1. To track experiments in real-time, use [DVCLive]:

- Install [DVCLive]:
- In your model training environment, install [DVCLive]:

```cli
pip install dvclive
```

- Copy your DVC Studio token and configure your model training environment to
use the token:
- Copy your
[DVC Studio token](/doc/studio/user-guice/account-and-billing#studio-access-token)
and configure your model training environment to use the token:

```cli
dvc config --global studio.token ***
```

- Add DVCLive to your model training code:
- Use the DVCLive log_metric() method in your model training code:

```python
from dvclive import Live
Expand All @@ -85,12 +49,12 @@ easily track live and completed experiments.
python train.py
```

- The metrics and plots will be updated in the project in real-time. [Learn
more][live-metrics-and-plots]
- The metrics and plots will be [tracked live][live-metrics-and-plots] in the
project in Iterative Studio.

### 2. To track outputs of reproducible pipelines, [initialize DVC](https://dvc.org/doc):
### 2. To set up, run and track [reproducible pipelines](/doc/start/experiments/experiment-pipelines), use [DVC](https://dvc.org/doc):

- Install [DVC]:
- In your model training environment, install [DVC]:

```cli
pip install dvc
Expand All @@ -104,31 +68,53 @@ easily track live and completed experiments.

This will create a `dvc.yaml` file in the Git repository.

- When running your experiments, save your metrics, plots, models and other
files to `dvc.yaml`. You can automate this by [setting up DVC pipelines].
- Copy your
[DVC Studio token](/doc/studio/user-guice/account-and-billing#studio-access-token)
and
[configure your model training environment to use the token](/doc/studio/user-guide/projects-and-experiments/live-metrics-and-plots#set-up-an-access-token):

```cli
dvc config --global studio.token ***
```

- [Set up the DVC
pipeline]/doc/start/experiments/experiment-pipelines#creating-the-experiment-pipeline
for model training.

- [Run an experiment](/doc/start/experiments/experiment-pipelines#modifying-parameters):

- After the experiment completes, commit and push the `dvc.yaml` file to your
Git remote.
```cli
dvc exp run
```

Metrics, plots and other details of your experiment get saved in your
project's `dvc.yaml` file.

- Studio reads the values of metrics, plots, models and other details from this
`dvc.yaml`.
- When the experiment completes,
[push the results](/doc/start/experiments/experiment-collaboration#sharing):

```cli
dvc exp push origin <exp_name>
```

### 3. To avoid over-crowding your Git history, use `dvc exp push`
- Iterative Studio reads the values of metrics, plots and other details from
your project's `dvc.yaml` file, and displays them in an experiment row in the
project table. The experiment row will appear nested within the parent Git
commit row.

- Once you have initialized DVC, you can also use the `dvc exp push` command to
**push experiments without creating separate Git commits** for them.
- From here, you can
[persist the experiments](/doc/start/experiments/experiment-collaboration#persisting)
as Git commits and PRs as well as
[remove unnecessary experiments](/doc/start/experiments/experiment-collaboration#removing).

- To notify Iterative Studio when you push experiments using `dvc exp push`,
[configure the `DVC_STUDIO_TOKEN` environment variable](/doc/studio/user-guide/projects-and-experiments/live-metrics-and-plots#set-up-an-access-token).
<admon type="tip">

If you are working with a **non-DVC repository**, you can
[indicate which files contain metrics and hyperparameters](/doc/studio/user-guide/projects-and-experiments/configure-a-project#custom-metrics-and-parameters)
that Iterative Studio should display in the project. However, we strongly
recommend using DVC to avail of all the features of Iterative Studio.

To **run new experiments** from Iterative Studio, add credentials with
sufficient access to your cloud. Optionally, set up Continuous Integration (CI)
to run experiments automatically on CI triggers. [Learn more][run-experiments]
</admon>

[on project settings]:
/doc/studio/user-guide/projects-and-experiments/configure-a-project#non-dvc-repositories
Expand All @@ -150,3 +136,37 @@ to run experiments automatically on CI triggers. [Learn more][run-experiments]
[create data registries]: /doc/use-cases/data-registry
[create data pipelines]: /doc/start/data-management/data-pipelines
[ci/cd in machine learning]: /doc/use-cases/ci-cd-for-machine-learning

## Visualize, compare and run experiments

Within a project, you can:

- [Explore all the details of the experiments][explore-ml-experiments] that you
have pushed to your Git repository.
- [Visualize the experiments][visualize] using plots and trend charts.
- [Compare experiments][compare].
- [Run new experiments][run-experiments], including on your own cloud instances,
directly from Iterative Studio.
- [Track live metrics and plots][live-metrics-and-plots] by sending them to
Iterative Studio by using [DVCLive].

[project settings]:
/doc/studio/user-guide/projects-and-experiments/configure-a-project
[when do you need project settings]:
/doc/studio/user-guide/projects-and-experiments/configure-a-project#scenarios-where-project-settings-are-required
[create multiple projects from a single git repository]:
/doc/studio/user-guide/projects-and-experiments/create-a-project#create-multiple-projects-from-a-single-git-repository
[explore ml experiments]:
/doc/studio/user-guide/projects-and-experiments/explore-ml-experiments
[create a team]: /doc/studio/user-guide/teams
[sign up for the **basic** or **enterprise** plan]:
/doc/studio/user-guide/change-team-plan-and-size
[make your projects public]:
/doc/studio/user-guide/projects-and-experiments/share-a-project
[train on the cloud, including on your own cloud infrastructure, and submit new experiments]:
/doc/studio/user-guide/projects-and-experiments/run-experiments
[live-metrics-and-plots]:
/doc/studio/user-guide/projects-and-experiments/live-metrics-and-plots
[dvclive]: /doc/dvclive
[monorepo]:
/doc/studio/user-guide/projects-and-experiments/configure-a-project#monorepo

0 comments on commit 3f869dc

Please sign in to comment.