devkit-v1.1

.flake8

@@ -0,0 +1,19 @@
+[flake8]
+ignore =
+    # Whitespace before ':'
+    E203,
+    # Module level import not at top of file
+    E402,
+    # Line break occurred before a binary operator
+    W503,
+    # Line break occurred after a binary operator
+    W504
+    # line break before binary operator
+    E203
+    # line too long
+    E501
+    # No lambdas — too strict
+    E731
+    # Local variable name is assigned to but never used
+    F841
+max-line-length = 120

README.md

@@ -3,7 +3,7 @@
 <p align="center">
   <img src="assets/navsim_transparent.png" width="600">
   <h2 align="center">Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking</h1>
-  <h3 align="center"><a href="https://arxiv.org/abs/2406.15349">Paper</a> | Supplementary | Talk | <a href="https://opendrivelab.com/challenge2024/#end_to_end_driving_at_scale">2024 Challenge</a> </h3>
+  <h3 align="center"><a href="https://arxiv.org/abs/2406.15349">Paper</a> | Supplementary | <a href="https://youtu.be/QKy2ZDCGbqE?si=l9ghG3pbiFgl8bmx&t=2345">Talk</a> | <a href="https://opendrivelab.com/challenge2024/#end_to_end_driving_at_scale">2024 Challenge</a> | <a href="https://huggingface.co/spaces/AGC2024-P/e2e-driving-navsim">Leaderboard</a>  </h3>
 </p>
 
 </div>
@@ -51,6 +51,10 @@
 
 
 ## Changelog <a name="changelog"></a>
+- **`[2024/09/03]`** NAVSIM v1.1 release
+  - Leaderboard for `navtest` on [Hugging Face](https://huggingface.co/spaces/AGC2024-P/e2e-driving-navsim)
+  - Updated docs for [submission](docs/submission.md) and [paper](https://arxiv.org/abs/2406.15349)
+  - Code refactoring, formatting, minor fixes
 - **`[2024/04/21]`** NAVSIM v1.0 release (official devkit version for [AGC 2024](https://opendrivelab.com/challenge2024/#end_to_end_driving_at_scale))
   - Parallelization of metric caching / evaluation
   - Adds [Transfuser](https://arxiv.org/abs/2205.15997) baseline (see [agents](docs/agents.md#Baselines))

docs/metrics.md

@@ -1,17 +1,20 @@
-# Understanding the PDM Score
+# The Predictive Driver Model Score (PDMS)
 
-Fair comparisons are challenging in the open-loop planning literature, due to metrics of narrow scope or inconsistent definitions between different projects. The PDM Score is a combination of six sub-metrics, which provides a comprehensive analysis of different aspects of driving performance. Five of these sub-metrics are discrete-valued, and one is continuous. All metrics are computed after a 4-second non-reactive simulation of the planner output: background actors follow their recorded future trajectories, and the ego vehicle moves based on an LQR controller. The full composition of the PDM score is detailed below:
+Fair comparisons are challenging in the open-loop planning literature, due to metrics of narrow scope or inconsistent definitions between different projects. The PDM Score is a combination of five sub-metrics, which provides a comprehensive analysis of different aspects of driving performance. Five of these sub-metrics are discrete-valued, and one is continuous. All metrics are computed after a 4-second non-reactive simulation of the planner output: background actors follow their recorded future trajectories, and the ego vehicle moves based on an LQR controller. The full composition of the PDM Ì is detailed below:
 
 Metric | Weight | Range |
 |---|---|---|
 No at-fault Collisions (NC) | multiplier | {0, 1/2, 1} |
 Drivable Area Compliance (DAC) | multiplier | {0, 1} |
-Driving Direction Compliance (DDC) | multiplier | {0, 1/2, 1} |
 Time to Collision (TTC) within bound | 5 | {0, 1} |
-Comfort (C) | 2 | {0, 1} |
 Ego Progress (EP) | 5 | [0, 1] |
+Comfort (C) | 2 | {0, 1} |
+Driving Direction Compliance (DDC) | 0 | {0, 1/2, 1} |
+
+**Note:** The Driving Direction Compliance (DDC) metric is ignored in PDMS (with zero weight).
+
+i.e., `PDMS = NC * DAC * (5*TTC + 5*EP + 2*C + 0*DDC) / 12`
 
-i.e., `PDM Score = NC * DAC * DDC * (5*TTC + 2*C + 5*EP) / 12`
 
 To evaluate the PDM score for an agent you can run:
 ```

docs/splits.md

@@ -16,7 +16,8 @@ This eases access for users that only want to run the `navtrain` split and not t
 The logs are always the complete dataset split.
 
 ## Overview
-The Table belows offers an overview on the training and test splits supported by NAVSIM. It also shows which config parameters have to be used to set the dataset split (`split`) and training/test split (`scene-filter`).
+The Table belows offers an overview on the training and test splits supported by NAVSIM. 
+In Navsim-v1.1, the training/test split can bet set with a single config parameter given in the table.
 
 <table border="0">
     <tr>
@@ -34,8 +35,7 @@ The Table belows offers an overview on the training and test splits supported by
         <td>14GB</td>
         <td>&gt;2000GB</td>
         <td>
-        split=trainval<br>
-        scene_filter=all_scenes
+        train_test_split=trainval
         </td>
     </tr>
     <tr>
@@ -44,8 +44,7 @@ The Table belows offers an overview on the training and test splits supported by
         <td>1GB</td>
         <td>217GB</td>
         <td>
-        split=test<br>
-        scene_filter=all_scenes
+        train_test_split=test
         </td>
     </tr>
     <tr>
@@ -54,8 +53,7 @@ The Table belows offers an overview on the training and test splits supported by
         <td>1GB</td>
         <td>151GB</td>
         <td>
-        split=mini<br>
-        scene_filter=all_scenes
+        train_test_split=mini
         </td>
     </tr>
     <tr>
@@ -65,8 +63,7 @@ The Table belows offers an overview on the training and test splits supported by
         <td>-</td>
         <td>445GB*</td>
         <td>
-        split=trainval<br>
-        scene_filter=navtrain
+        train_test_split=navtrain
         </td>
     </tr>
     <tr>
@@ -75,8 +72,7 @@ The Table belows offers an overview on the training and test splits supported by
         <td>-</td>
         <td>-</td>
         <td>
-        split=test<br>
-        scene_filter=navtest
+        train_test_split=navtest
         </td>
     </tr>
     <tr>
@@ -86,8 +82,7 @@ The Table belows offers an overview on the training and test splits supported by
         <td>-</td>
         <td>-</td>
         <td>
-        split=mini<br>
-        scene_filter=warmup_test_e2e
+        train_test_split=warmup_test_e2e
         </td>
     </tr>
     <tr>
@@ -96,8 +91,7 @@ The Table belows offers an overview on the training and test splits supported by
         <td>&lt;1GB</td>
         <td>25GB</td>
         <td>
-        split=private_test_e2e<br>
-        scene_filter=private_test_e2e
+        train_test_split=private_test_e2e
         </td>
     </tr>
 </table>

docs/submission.md

@@ -1,21 +1,29 @@
-# Submitting to the Leaderboard
+# Submitting to the leaderboard
 
-NAVSIM comes with official leaderboards on HuggingFace. The leaderboards prevent ambiguity in metric definitions between different projects, as all evaluation is performed on the server with the official evaluation script.
+NAVSIM comes with an [official leaderboard](https://huggingface.co/spaces/AGC2024-P/e2e-driving-navsim) on HuggingFace. The leaderboard prevents ambiguity in metric definitions between different projects, as all evaluation is performed on the server with the official evaluation script.
+
+After the NAVSIM challenge 2024, we have re-opened the leaderboard with the `navtest` split. With 12k samples, the `navtest` split contains a larger set for comprehensive evaluations. In this guide, we describe how to create a valid submission and what rules apply for the new leaderboard.
+
+### Rules
+- **Technical Reports**:
+  - We will periodically be removing all entries on the leaderboard which **do not provide an associated technical report**.
+  - The technical report can be in any format (e.g. ArXiv paper, github readme, privately hosted pdf file, etc), as long as it clearly describes the methodology used in the submission, enabling reimplementation.
+  - Technical reports must be provided by setting the `TEAM_NAME` variable of the submission file with `"<a href=Link/to/tech/report.pdf>Method name</a>"`. Note that this can also be edited on the leaderboard for an existing submission, if the report is created (or updated) after the initial submission.
+- **Multi-seed submissions**:
+  - Driving policies often differ significantly in performance when re-trained with different network initialization seeds.
+  - Therefore, the leaderboard now supports (1) regular single-seed submissions and (2) multi-seed submission, which we **strongly encourage** (with a minimum of 3 training seeds).
+  - The maximum, mean and standard deviations of our evaluation metrics will be displayed for multi-seed submissions.
+
+### Regular navtest submission
 
 To submit to a leaderboard you need to create a pickle file that contains a trajectory for each test scenario. NAVSIM provides a script to create such a pickle file. 
 
-Have a look at `run_create_submission_pickle.sh`: this file creates the pickle file for the ConstantVelocity agent. You can run it for your own agent by replacing the `agent` override.
-Follow the [submission instructions on huggingface](https://huggingface.co/spaces/AGC2024-P/e2e-driving-2024) to upload your submission.
+Have a look at `run_create_submission_pickle.sh`: this file creates the pickle file for the ConstantVelocity agent. You can run it for your own agent by replacing the `agent` override. Follow the [submission instructions on huggingface](https://huggingface.co/spaces/AGC2024-P/e2e-driving-navsim) to upload your submission.
 **Note that you have to set the variables `TEAM_NAME`, `AUTHORS`, `EMAIL`, `INSTITUTION`, and `COUNTRY` in `run_create_submission_pickle.sh` to generate a valid submission file**
 
-### Warm-up track
-The warm-up track evaluates your submission on a [warm-up leaderboard](https://huggingface.co/spaces/AGC2024-P/e2e-driving-warmup) based on the `mini` split. This allows you to test your method and get familiar with the devkit and the submission procedure, with a less restrictive submission budget (up to 5 submissions daily). Instructions on making a submission on HuggingFace are available in the HuggingFace space. Performance on the warm-up leaderboard is not taken into consideration for determining your team's ranking for the 2024 Autonomous Grand Challenge.
-Use the script `run_create_submission_pickle_warmup.sh` which already contains the overrides `scene_filter=warmup_test_e2e` and `split=mini` to generate the submission file for the warmup track.
+You should be able to obtain the same evaluation results as on the server by running the evaluation locally.
+To do so, use the override `train_test_split=navtest` when executing the script to run the PDM scoring.
 
-You should be able to obtain the same evaluation results as on the server, by running the evaluation locally.
-To do so, use the overrides `scene_filter=warmup_test_e2e` when executing the script to run the PDM scoring (e.g.,  `run_cv_pdm_score_evaluation.sh` for the constant-velocity agent).
+### Multi-seed navtest submission
 
-### Formal track
-This is the [official challenge leaderboard](https://huggingface.co/spaces/AGC2024-P/e2e-driving-2024), based on secret held-out test frames (see submission_test split on the install page). 
-Use the script `run_create_submission_pickle.sh`. It will by default run with `scene_filter=private_test_e2e` and `split=private_test_e2e`.
-You only need to set your own agent with the `agent` override.
+For a multi-seed submission, you first have to create individual agents, i.e. trained on different seeds. Consequently, you can merge your entries to a single submission file with the `run_merge_submission_pickles.sh` bash script. Please set the override `train_test_split=navtest` to ensure all individual entries contain trajectories for the evaluation.

download/download_navtrain.sh

@@ -11,7 +11,7 @@ for split in {1..4}; do
     tar -xzf navtrain_current_${split}.tgz
     rm navtrain_current_${split}.tgz
 
-    mv -v current_split_${split}/* trainval_sensor_blobs/trainval
+    rsync -v current_split_${split}/* trainval_sensor_blobs/trainval
     rm -r current_split_${split}
 done
 
@@ -21,6 +21,6 @@ for split in {1..4}; do
     tar -xzf navtrain_history_${split}.tgz
     rm navtrain_history_${split}.tgz
 
-    mv -v history_split_${split}/* trainval_sensor_blobs/trainval
+    rsync -v history_split_${split}/* trainval_sensor_blobs/trainval
     rm -r history_split_${split}
 done

navsim/agents/abstract_agent.py

@@ -8,6 +8,8 @@
 
 
 class AbstractAgent(torch.nn.Module, ABC):
+    """Interface for an agent in NAVSIM."""
+
     def __init__(
         self,
         requires_scene: bool = False,
@@ -21,7 +23,7 @@ def name(self) -> str:
         :return: string describing name of this agent.
         """
         pass
-    
+
     @abstractmethod
     def get_sensor_config(self) -> SensorConfig:
         """
@@ -64,7 +66,7 @@ def compute_trajectory(self, agent_input: AgentInput) -> Trajectory:
         :return: Trajectory representing the predicted ego's position in future
         """
         self.eval()
-        features : Dict[str, torch.Tensor] = {}
+        features: Dict[str, torch.Tensor] = {}
         # build features
         for builder in self.get_feature_builders():
             features.update(builder.compute_features(agent_input))
@@ -79,7 +81,7 @@ def compute_trajectory(self, agent_input: AgentInput) -> Trajectory:
 
         # extract trajectory
         return Trajectory(poses)
-    
+
     def compute_loss(
         self,
         features: Dict[str, torch.Tensor],
@@ -90,27 +92,19 @@ def compute_loss(
         Computes the loss used for backpropagation based on the features, targets and model predictions.
         """
         raise NotImplementedError("No loss. Agent does not support training.")
-    
+
     def get_optimizers(
-        self
-    ) -> Union[
-        torch.optim.Optimizer,
-        Dict[str, Union[
-            torch.optim.Optimizer,
-            torch.optim.lr_scheduler.LRScheduler]
-        ]
-    ]:
+        self,
+    ) -> Union[torch.optim.Optimizer, Dict[str, Union[torch.optim.Optimizer, torch.optim.lr_scheduler.LRScheduler]]]:
         """
         Returns the optimizers that are used by thy pytorch-lightning trainer.
         Has to be either a single optimizer or a dict of optimizer and lr scheduler.
         """
         raise NotImplementedError("No optimizers. Agent does not support training.")
-
-    def get_training_callbacks(
-        self
-    ) -> List[pl.Callback]:
+
+    def get_training_callbacks(self) -> List[pl.Callback]:
         """
         Returns a list of pytorch-lightning callbacks that are used during training.
         See navsim.planning.training.callbacks for examples.
         """
-        return []
+        return []

navsim/agents/constant_velocity_agent.py

@@ -1,20 +1,18 @@
-from typing import List
+import numpy as np
+
 from nuplan.planning.simulation.trajectory.trajectory_sampling import TrajectorySampling
 from navsim.agents.abstract_agent import AbstractAgent
 from navsim.common.dataclasses import AgentInput, Trajectory, SensorConfig
 
-import numpy as np
-
 
 class ConstantVelocityAgent(AbstractAgent):
+    """Constant velocity baseline agent."""
 
     requires_scene = False
 
     def __init__(
         self,
-        trajectory_sampling: TrajectorySampling = TrajectorySampling(
-            time_horizon=4, interval_length=0.5
-        ),
+        trajectory_sampling: TrajectorySampling = TrajectorySampling(time_horizon=4, interval_length=0.5),
     ):
         self._trajectory_sampling = trajectory_sampling
 
@@ -32,11 +30,8 @@ def get_sensor_config(self) -> SensorConfig:
         return SensorConfig.build_no_sensors()
 
     def compute_trajectory(self, agent_input: AgentInput) -> Trajectory:
-        """
-        Computes the ego vehicle trajectory.
-        :param current_input: Dataclass with agent inputs.
-        :return: Trajectory representing the predicted ego's position in future
-        """
+        """Inherited, see superclass."""
+
         ego_velocity_2d = agent_input.ego_statuses[-1].ego_velocity
         ego_speed = (ego_velocity_2d**2).sum(-1) ** 0.5