How to tag a new version of the dataset?

[msis]

Currently, when uploading a dataset, one would call:

dataset.push_to_hub(
  "user_name/dataset_name",
)

So I thought one could create a new version with:

dataset.push_to_hub(
  "user_name/dataset_name",
  branch="v1",
)

But then I get the following error:

Revision Not Found for url: https://huggingface.co/api/datasets/tarteel-ai/tldn_hf/revision/v1.                                                                                                            
Invalid rev id: v1  

How to then create a new version?

I've tried cloning, tagging the last commit, pushing it, then calling push_to_hub again.

However, I seed now duplicates of shard, see below:

How do I then get only one version in the commit of v1?

[mariosasko]

Hi! You can use Git tags to mark the releases (versions) of a dataset:

import huggingface_hub

# version 1.0
dataset.push_to_hub(
  "user_name/dataset_name",
)
huggingface_hub.create_tag("user_name/dataset_name", tag="1.0", repo_type="dataset")

# version 2.0
dataset.push_to_hub(
  "user_name/dataset_name",
)
huggingface_hub.create_tag("user_name/dataset_name", tag="2.0", repo_type="dataset")

And then reference them when loading as follows:

# version 1.0
load_dataset("user_name/dataset_name", revision="1.0")
# version 2.0
load_dataset("user_name/dataset_name", revision="2.0")

PS: push_to_hub can only push to an existing branch. So if branch points to a non-existing branch, call huggingface_hub.create_branch("user_name/dataset_name", branch=branch, repo_type="dataset") before running push_to_hub.