This is the code for IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. To show the efficacy of the model, we explore several new IMU-based applications that IMU2CLIP enables, such as motion-based media retrieval and natural language reasoning tasks with motion data. In addition, we show that IMU2CLIP can significantly improve the downstream performance when fine-tuned for each application (e.g. activity recognition), demonstrating the universal usage of IMU2CLIP as a new pre-trained resource.
conda create -n imu2clip python=3.8
conda activate imu2clip
pip install pytorch_lightning
pip install torchaudio
pip install torchvision
pip install git+https://github.com/openai/CLIP.git
pip install opencv-python
pip install matplotlib
pip install ffmpeg-python
pip install pandas
After installing all the library, check the in dataset/ego4d/README.md for instruction on how to preprocess the ego4d data.
To run an example train loop
python pretraining.py
To run a pretrained model in downstream task
python downstream.py
In the config folder, you can find details hyperparamters for training IMU2CLIP with different contrastive losses.
@article{moon2022imu2clip,
title={IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text},
author={Moon, Seungwhan and Madotto, Andrea and Lin, Zhaojiang and Dirafzoon, Alireza and Saraf, Aparajita and Bearman, Amy and Damavandi, Babak},
journal={arXiv preprint arXiv:2210.14395},
year={2022}
}
The majority of IMU2CLIP is licensed under CC-BY-NC, however portions of the project are available under separate license terms: PyTorchLigtning is licensed under the Apache 2.0 license and CLIP is licensed under the MIT License.
See LICENSE for details.