Fair comparisons are challenging in the open-loop planning literature, due to metrics of narrow scope or inconsistent definitions between different projects. The PDM Score is a combination of five sub-metrics, which provides a comprehensive analysis of different aspects of driving performance. Five of these sub-metrics are discrete-valued, and one is continuous. All metrics are computed after a 4-second non-reactive simulation of the planner output: background actors follow their recorded future trajectories, and the ego vehicle moves based on an LQR controller. The full composition of the PDM Ì is detailed below:
| Metric | Weight | Range |
|---|---|---|
| No at-fault Collisions (NC) | multiplier | {0, 1/2, 1} |
| Drivable Area Compliance (DAC) | multiplier | {0, 1} |
| Time to Collision (TTC) within bound | 5 | {0, 1} |
| Ego Progress (EP) | 5 | [0, 1] |
| Comfort (C) | 2 | {0, 1} |
| Driving Direction Compliance (DDC) | 0 | {0, 1/2, 1} |
Note: The Driving Direction Compliance (DDC) metric is ignored in PDMS (with zero weight).
i.e., PDMS = NC * DAC * (5*TTC + 5*EP + 2*C + 0*DDC) / 12
To evaluate the PDM score for an agent you can run:
cd $NAVSIM_DEVKIT_ROOT/scripts/
./run_cv_pdm_score_evaluation.shBy default, this will generate an evaluation csv for a simple constant velocity planning baseline. You can modify the script to evaluate your own planning agent.
For instance, you can add a new config for your agent under $NAVSIM_DEVKIT_ROOT/navsim/navsim/planning/script/config/pdm_scoring/agent/my_new_agent.yaml.
Then, running your own agent is as simple as adding an override agent=my_new_agent to the script.
You can find an example in run_human_agent_pdm_score_evaluation.sh