The NAVSIM framework utilizes several dataset splits for standardized training and evaluating agents.
All of them use the OpenScene dataset that is divided into the dataset splits mini,trainval,test,private_test_e2e, which can all be downloaded separately.
It is possible to run trainings and evaluations directly on these sets (see Standard in table below).
Alternatively, you can run trainings and evaluations on training and validation splits that were filtered for challenging scenarios (see NAVSIM in table below), which is the recommended option for producing comparable and competitive results efficiently.
In contrast to the dataset splits which refer to a downloadable set of logs, the training / test splits are implemented as scene filters, which define how scenes are extracted from these logs.
The NAVSIM training / test splits subsample the OpenScene dataset splits.
Moreover, the NAVSIM splits include overlapping scenes, while the Standard splits are non-overlapping.
Specifically, navtrain is based on the trainval data and navtest on the test data.
As the trainval sensor data is very large, we provide a separate download link, which loads only the frames needed for navtrain.
This eases access for users that only want to run the navtrain split and not the trainval split. If you already downloaded the full trainval sensor data, it is not necessary to download the navtrain frames as well.
The logs are always the complete dataset split.
The Table belows offers an overview on the training and test splits supported by NAVSIM. It also shows which config parameters have to be used to set the dataset split (split) and training/test split (scene-filter).
| Name | Description | Logs | Sensors | Config parameters | |
|---|---|---|---|---|---|
| Standard | trainval | Large split for training and validating agents with regular driving recordings. Corresponds to nuPlan and downsampled to 2HZ. | 14GB | >2000GB |
split=trainval scene_filter=all_scenes |
| test | Small split for testing agents with regular driving recordings. Corresponds to nuPlan and downsampled to 2HZ. | 1GB | 217GB |
split=test scene_filter=all_scenes |
|
| mini | Demo split for with regular driving recordings. Corresponds to nuPlan and downsampled to 2HZ. | 1GB | 151GB |
split=mini scene_filter=all_scenes |
|
| NAVSIM | navtrain | Standard split for training agents in NAVSIM with non-trivial driving scenes. Sensors available separately in download_navtrain.sh. | - | 445GB* |
split=trainval scene_filter=navtrain |
| navtest | Standard split for testing agents in NAVSIM with non-trivial driving scenes. Available as a filter for test split. | - | - |
split=test scene_filter=navtest |
|
| Competition | warmup_test_e2e | Warmup test split to validate submission on hugging face. Available as a filter for mini split. | - | - |
split=mini scene_filter=warmup_test_e2e |
| private_test_e2e | Private test split for the challenge leaderboard on hugging face. | <1GB | 25GB |
split=private_test_e2e scene_filter=private_test_e2e |
(*300GB without history)
The standard splits trainval, test, and mini are from the OpenScene dataset. Note that the data corresponds to the nuPlan dataset with a lower frequency of 2Hz. You can download all standard splits over Hugging Face with the bash scripts in download
NAVSIM provides a subset and filter of the trainval split, called navtrain. The navtrain split facilitates a standardized training scheme and requires significantly less sensor data storage than travel (445GB vs. 2100GB). If your agents don't need historical sensor inputs, you can download navtrain without history, which requires 300GB of storage. Note that navtrain can be downloaded separately via download_navtrain.sh but still requires access to the trainval logs. Similarly, the navtest split enables a standardized set for testing agents with a provided scene filter. Both navtrain and navtest are filtered to increase interesting samples in the sets.
For the challenge on Hugging Face, we provide the warmup_test_e2e and private_test_e2e for the warm-up and challenge track, respectively. Note that private_test_e2e requires you to download the data, while warmup_test_e2e is a scene filter for the mini split.