This repo contains the work for the work I have done in preparation for the Spotify Wrapped 2022 event in Berlin.
In the event, a stage is setup with different props, lights, and funny elements. People can adjust the setting as they like, and choose between very classical hits like Yellow Submarine, or modern songs like As it was. When all is ready and the mood is set, a photographer takes a picture which is automatically uploaded to an S3 bucket, and sent as REST request alongside the chosen song to an endpoint exposed by the FastAPI server provided in this repo.
The server runs a custom made, optimized version of Stable Diffusion with Adabins and MiDaS to create a morphing 3D animation of the picture into a world of colors and shapes. For example (sound on! 🔊):
As_it_was_-_Harry_Styles.mp4
The repo is based on the amazing work of Deforum's Stable Diffusion. I optimized, cleaned, refactored, and organized the code, as well as extending it with the music video generation feature, preset system, REST server and more.
1- Install dev dependencies pip install -r requirements/dev.txt
2- Download models in pretrained by running python download_models.py
3- Start API server by running python inference_server/app.py
A docker file is also provided for deployment in a EC2 instance.