SLEDGE v0.1 release

.flake8

@@ -0,0 +1,17 @@
+[flake8]
+ignore =
+    # Whitespace before ':'
+    E203,
+    # Module level import not at top of file
+    E402,
+    # Line break occurred before a binary operator
+    W503,
+    # Line break occurred after a binary operator
+    W504
+    # line break before binary operator
+    E203
+    # line too long
+    E501
+    # No lambdas — too strict
+    E731
+max-line-length = 120

.gitignore

@@ -0,0 +1,20 @@
+# python
+*.pyc
+**/__pycache__/
+.pytest_cache/*
+.pydevproject
+
+# IDE
+.vscode/*
+
+# Pip
+*.egg-info
+
+# Log files
+*.out
+*.err
+*.gz
+
+# Custom
+*.ckpt
+*.zip

README.md

@@ -1,15 +1,17 @@
 
 <p align="center">
     <img alt="sledge" src="assets/sledge_logo_transparent.png" width="500">
-    <h1 align="center">Generative Simulation for Vehicle Motion Planning</h1>
+    <h1 align="center">Generative Simulation for Vehicle Motion Planning </h1>
     <h3 align="center"><a href="https://arxiv.org/abs/2403.17933">Paper</a> | <a href="https://danieldauner.github.io/assets/pdf/Chitta2024ARXIV_supplementary.pdf">Supplementary</a> | Video  </h3>
 </p>
 
 <br/>
 
-> [**SLEDGE: Synthesizing Simulation Environments for Driving Agents with Generative Models**](https://arxiv.org/abs/2403.17933) <br>
+> [**SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Traffic**](https://arxiv.org/abs/2403.17933) <br>
 > [Kashyap Chitta](https://kashyap7x.github.io/), [Daniel Dauner](https://danieldauner.github.io/), and [Andreas Geiger](https://www.cvlibs.net/) <br>
 > University of Tübingen, Tübingen AI Center
+>
+> European Conference on Computer Vision (ECCV), 2024
 > <br>
 >
 
@@ -19,48 +21,62 @@ This repo contains SLEDGE, the first generative simulator for vehicle motion pla
 
 https://github.com/autonomousvision/sledge/assets/50077664/1c653fda-6e44-4018-ae98-2ab3d0439cad
 
-## News
-* **`27 Mar, 2024`:** We released our paper on [arXiv](https://arxiv.org/abs/2403.17933). Code and models are coming soon, please stay tuned!
+## News &#128240; 
+* **`18 Aug, 2024`:** We released v0.1 of the SLEDGE code!
+* **`01 Jul, 2024`:** Our paper was accepted at [ECCV 2024](https://eccv.ecva.net/) &#127470;&#127481;
+* **`27 Mar, 2024`:** We released our paper on [arXiv](https://arxiv.org/abs/2403.17933)!
 
-<br/>
 
-## Overview
+## Table of Contents &#128220;
+1. [Overview](#overview)
+2. [Getting started](#gettingstarted)
+3. [Changelog](#changelog)
+4. [Contact](#contact)
+5. [Citation](#citation)
+6. [Other resources](#otherresources)
 
-- SLEDGE is a generative simulator, able to synthesize agent bounding boxes and lane graphs, which serve as an initial state for traffic simulation.
 
-- The unique properties of the entities to be generated for SLEDGE, such as their connectivity and variable count per scene, render the naive application of most modern generative models to this task non-trivial.
+## Getting started &#128640; <a name="gettingstarted"></a>
 
-- Therefore, we introduce a novel raster-to-vector autoencoder (RVAE). It encodes agents and the lane graph into distinct channels in a rasterized latent map. This facilitates both lane-conditioned agent generation and
-combined generation of lanes and agents with a Diffusion Transformer.
+- [Installation and download](docs/installation.md)
+- [Running the autoencoder](docs/autoencoder.md) 
+- [Running the diffusion model](docs/diffusion.md)
+- [Simulation and visualization](docs/simulation.md)
 
-- Using generated entities in SLEDGE enables greater control over the simulation, e.g. long routes, upsampling turns, or increasing traffic density. SLEDGE presents new challenges for planning algorithms, evidenced by failure rates of over 40% for PDM, when tested on hard routes and dense generated traffic.
 
-- Compared to nuPlan, SLEDGE requires 500× less storage to set up (<4GB), making it a more accessible option and helping with democratizing future research in this field.
+## Changelog &#128759; <a name="changelog"></a>
+- **`[2024/08/18]`** SLEDGE v0.1 release
+  - Scripts for pre-processing and downloads
+  - Raster-vector autoencoder (training & latent caching)
+  - Latent diffusion models (training & scenario generation)
+  - Simple simulations
+  - SledgeBoard
 
-<br/>
 
-## TODO
-- [ ] Code release 
-- [ ] Supplementary material & video
+## TODO &#128203;<a name="todo"></a>
+- [ ] Add videos and talks
+- [ ] Release checkpoints
+- [ ] Metrics & complete simulation code
+- [X] SLEDGE v0.1 & camera ready release
 - [x] Initial repository & preprint release
 
-## Contact
+
+## Contact &#9993; <a name="contact"></a>
 If you have any questions or suggestions, please feel free to open an issue or contact us ([email protected]).
 
-## Citation
+## Citation &#128206; <a name="citation"></a>
 If you find SLEDGE useful, please consider giving us a star &#127775; and citing our paper with the following BibTeX entry.
 
 ```BibTeX
-@InProceedings{Chitta2024ARXIV,
-  title={SLEDGE: Synthesizing Simulation Environments for Driving Agents with Generative Models},
-  author={Chitta, Kashyap and Dauner, Daniel and Geiger, Andreas},
-  journal={arXiv},
-  volume={2403.17933},
-  year={2024}
+@InProceedings{Chitta2024ECCV, 
+	title = {SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Traffic}, 
+	author = {Kashyap Chitta and Daniel Dauner and Andreas Geiger}, 
+	booktitle = {European Conference on Computer Vision (ECCV)}, 
+	year = {2024}, 
 }
 ```
 
-## Other resources <a name="otherresources"></a>
+## Other resources &#128161; <a name="otherresources"></a>
 
 <a href="https://twitter.com/AutoVisionGroup" target="_blank">
     <img alt="Twitter Follow" src="https://img.shields.io/twitter/follow/Awesome Vision Group?style=social&color=brightgreen&logo=twitter" />
@@ -72,7 +88,7 @@ If you find SLEDGE useful, please consider giving us a star &#127775; and citing
     <img alt="Twitter Follow" src="https://img.shields.io/twitter/follow/Daniel Dauner?style=social&color=brightgreen&logo=twitter" />
   </a>
 
-- [NAVSIM](https://github.com/autonomousvision/navsim) | [tuPlan garage](https://github.com/autonomousvision/tuplan_garage) | [CARLA garage](https://github.com/autonomousvision/carla_garage) | [Survey on E2EAD](https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving)
+- [NAVSIM](https://github.com/autonomousvision/navsim) | [tuPlan garage](https://github.com/autonomousvision/sledge) | [CARLA garage](https://github.com/autonomousvision/carla_garage) | [Survey on E2EAD](https://github.com/OpenDriveLab/End-to-end-Autonomous-Driving)
 - [PlanT](https://github.com/autonomousvision/plant) | [KING](https://github.com/autonomousvision/king) | [TransFuser](https://github.com/autonomousvision/transfuser) | [NEAT](https://github.com/autonomousvision/neat)
 
 <p align="right">(<a href="#top">back to top</a>)</p>

docs/autoencoder.md

@@ -0,0 +1,31 @@
+# Autoencoder
+
+This guide provides instructions on how to run an autoencoder in SLEDGE. 
+The tutorial below shows the key functionalities of the raster-to-vector autoencoder (RVAE).
+
+### 1. Feature Caching
+Similar to the [nuplan-devkit](https://github.com/motional/nuplan-devkit),
+ pre-processing of the training data is recommended. 
+The cache for the RVAE can be created by running:
+```bash
+cd $SLEDGE_DEVKIT_ROOT/scripts/autoencoder/rvae/
+bash feature_caching_rvae.sh
+```
+This script pre-processes the vector features of several maps sequentially. The cached features only store the local map and agents in a general vector format. The features are further processed and rasterized on the fly during training. This two-step processing enables fast access to training data and allows data augmentation (e.g. random rotation and translation) for RVAE training. The feature cache is compatible with other autoencoders.
+
+### 2. Training Autoencoder
+After creating or downloading the autoencoder cache, you can start the training. We provide an example script in the same folder.
+```bash
+bash training_rvae.sh
+```
+You can find the experiment folder of training in `$SLEDGE_EXP_ROOT/exp` and monitor the run with tensorboard.
+
+### 3. Latent Caching
+You must first cache the latent variables to run a latent diffusion model with the trained autoencoder. In SLEDGE, we cache the latent variables into the autoencoder cache directory (i.e. `$SLEDGE_EXP_ROOT/caches/autoencoder_cache`). The bash script is provided in the RVAE folder.
+```bash
+bash latent_caching_rvae.sh
+```
+Importantly, data augmentation is disabled for latent caching. We also only cache the samples from the training split.
+
+### 4. Evaluating Autoencoder
+Coming soon!

docs/diffusion.md

@@ -0,0 +1,23 @@
+# Diffusion
+
+This section provides instructions on how to utilize diffusion models within the SLEDGE framework. 
+
+### 1. Training Diffusion
+Before training a diffusion model, make sure you have a trained autoencoder checkpoint and latent cache as described in `docs/autoencoder.md`.
+You can start a training experiment by running the script:
+```bash
+cd $SLEDGE_DEVKIT_ROOT/scripts/diffusion/
+bash training_diffusion.sh
+``` 
+Please make sure you added the autoencoder checkpoint path to the bash script. Before training starts, the latent variables will be stored in a Hugging Face dataset format and saved to `$SLEDGE_EXP_ROOT/caches/diffusion_cache`. This format is compatible with the [`accelerate`](https://github.com/huggingface/accelerate) framework and has performance advantages. Read more [here](https://huggingface.co/docs/datasets/about_arrow) if you are interested. Our training pipeline supports [diffusion transformers (DiT)](https://arxiv.org/abs/2212.09748) in four sizes (S, B, L, XL). You can find the experiment folder and checkpoints in `$SLEDGE_EXP_ROOT/exp`. You can also monitor the training with tensorboard. 
+
+### 2. Scenario Synthesis
+Given the trained diffusion model, you can generate a set of samples used for driving simulation or the generative metrics. You can set the diffuser checkpoint path and run the following:
+```bash
+bash scenario_caching_diffusion.sh
+```
+The samples are stored in `$SLEDGE_EXP_ROOT/caches/scenario_cache` by default. These samples can be simulated in the v0.1 release.
+Additional options for route extrapolation by inpainting will be added in a future update.
+
+### 3. Evaluating Diffusion
+Coming soon!

docs/installation.md

@@ -0,0 +1,81 @@
+# Installation and download
+
+To get started with SLEDGE, we provide the main instructions on how to install the devkit and download the necessary data for training and simulation.
+
+### 1. Clone the sledge-devkit
+Begin by cloning the SLEDGE repository and navigating to the repository directory:
+```bash
+git clone https://github.com/autonomousvision/sledge.git
+cd sledge
+```
+
+### 2. Download the data
+**Important:** Before downloading any data, please ensure you have read the [nuPlan license](https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/LICENSE).
+
+You can find the bash scripts to download the datasplits in `scripts/download/`. For complete usage in SLEDGE, you need to download the nuPlan dataset (without sensor data) with the `download_nuplan.sh` script (~2TiB). Each part of nuPlan has different roles in the SLEDGE framework:
+- `train` and `val` contain the training and validation logs, which primarily serve for autoencoder feature caching. The autoencoder and diffusion training pipeline will operate on this pre-processed cache.
+- `test` contains the test logs that are used in the metrics (reconstruction / generative) and for "Lane->Agent" simulation. 
+- `mini` has a few log files (~15GB) and can be used for simple experiments and demos.
+- `maps` contains the maps of the four cities in nuPlan. These maps are necessary for autoencoder feature caching, the metrics, and "Lane->Agent" simulation.
+
+If you don't want to change the autoencoder features, you can directly download the pre-processed training cache with the `download_cache.sh` script (~15GiB). As such, the [license and conditions](https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/LICENSE) of the nuPlan dataset apply. 
+
+After downloading, organize the decompressed files in the following directory structure:
+```angular2html
+~/sledge_workspace
+├── sledge (devkit)
+├── exp
+│   ├── caches
+│   │   ├── autoencoder_cache
+│   │   │   └── <cached logs>
+│   │   ├── diffusion_cache
+│   │   │   └── <hugging face dataset>
+│   │   └── scenario_cache
+│   │       └── <generated logs>
+│   └── exp
+│       └── <sledge_experiments>
+└── dataset
+    ├── maps
+    │   ├── nuplan-maps-v1.0.json
+    │   ├── sg-one-north
+    │   │   └── ...
+    │   ├── us-ma-boston
+    │   │   └── ...
+    │   ├── us-nv-las-vegas-strip
+    │   │   └── ...
+    │   └── us-pa-pittsburgh-hazelwood
+    │       └── ...
+    └── nuplan-v1.1
+         ├── splits 
+         │     ├── mini 
+         │     │    ├── 2021.05.12.22.00.38_veh-35_01008_01518.db
+         │     │    ├── ...
+         │     │    └── 2021.10.11.08.31.07_veh-50_01750_01948.db
+         │     ├── test 
+         │     │    ├── 2021.05.25.12.30.39_veh-25_00005_00215.db
+         │     │    ├── ...
+         │     │    └── 2021.10.06.08.34.20_veh-53_01089_01868.db
+         │     └── trainval
+         │          ├── 2021.05.12.19.36.12_veh-35_00005_00204.db
+         │          ├── ...
+         │          └── 2021.10.22.18.45.52_veh-28_01175_01298.db
+         └── sensor_blobs (empty)
+```
+Several environment variables need to be added next to your `~/.bashrc` file. 
+For the above, the environment variables are defined as:
+```bash
+export NUPLAN_DATA_ROOT="$HOME/sledge_workspace/dataset"
+export NUPLAN_MAPS_ROOT="$HOME/sledge_workspace/dataset/maps"
+
+export SLEDGE_EXP_ROOT="$HOME/sledge_workspace/exp"
+export SLEDGE_DEVKIT_ROOT="$HOME/sledge_workspace/sledge"
+```
+
+### 3. Install the sledge-devkit
+To install SLEDGE, create a new conda environment and install the necessary dependencies as follows:
+```bash
+conda env create --name sledge -f environment.yml
+conda activate sledge
+pip install -e .
+```
+With these steps completed, SLEDGE should be ready to use.

docs/simulation.md

@@ -0,0 +1,21 @@
+# Simulation
+
+This section provides instructions for simulating scenarios in SLEDGE.
+
+### Simple Simulation
+
+For the v0.1 release, you can simulate simple 64mx64m patches. You first need to train a diffusion model and run scenario caching, as described in `docs/diffusion.md`.
+Consequently, you can simulate the scenarios by running.
+```bash
+cd $SLEDGE_DEVKIT_ROOT/scripts/simulation/
+bash simple_simulation.sh
+``` 
+By default, we simulate the [PDM-Closed](https://arxiv.org/abs/2306.07962) planner for 15 seconds. The experiment folder can be found in `$SLEDGE_EXP_ROOT/exp`. Further simulation modes and configurations will follow in future updates.
+
+### Visualization
+
+The simulated scenarios can be visualized with SledgeBoard. Simply run:
+```bash 
+python $SLEDGE_DEVKIT_ROOT/sledge/script/run_sledgeboard.py
+```
+Open the `.nuboard` file in the experiment folder, view simulations, and render videos of scenarios. Note that SledgeBoard is primarily of a skin of nuBoard, with small adaptations to view synthetic scenarios in SLEDGE.

environment.yml

@@ -0,0 +1,9 @@
+name: sledge
+channels:
+  - conda-forge
+dependencies:
+  - python=3.9
+  - pip=23.3.1
+  - nb_conda_kernels
+  - pip:
+    - -r requirements.txt

requirements.txt

@@ -0,0 +1,54 @@
+nuplan-devkit @ git+https://github.com/motional/nuplan-devkit/@nuplan-devkit-v1.2
+
+# nuplan requirements
+aioboto3
+aiofiles
+bokeh==2.4.3 
+casadi
+control==0.9.1 
+Fiona 
+geopandas>=0.12.1 
+guppy3==3.1.2
+hydra-core==1.1.0rc1  
+joblib
+matplotlib  
+nest_asyncio  
+numpy==1.23.4
+opencv-python==4.9.0.80  
+pandas    
+Pillow   
+psutil  
+pyarrow 
+pyinstrument 
+pyogrio 
+pyquaternion>=0.9.5 
+pytest    
+rasterio  
+ray 
+retry
+rtree 
+scipy  
+selenium 
+setuptools==65.5.1
+Shapely>=2.0.0
+SQLAlchemy==1.4.27  
+sympy  
+tornado 
+tqdm  
+ujson  
+notebook==7.2.1
+
+# torch & lighting
+torch==2.0.1
+torchvision==0.15.2
+pytorch-lightning==2.2.1
+tensorboard
+protobuf==4.25.3
+
+# hugging face
+accelerate==0.33.0
+datasets==2.20.0
+transformers==4.43.3
+dataset==1.6.2
+diffusers==0.29.2
+timm==1.0.8