Moto: Latent Motion Token as the Bridging Language for Robot Manipulation
Paper: https://huggingface.co./papers/2412.04445
🚀Introduction
Recent developments in Large Language Models (LLMs) pre-trained on extensive corpora have shown significant success in various natural language processing (NLP) tasks with minimal fine-tuning. This success offers new promise for robotics, which has long been constrained by the high cost of action-labeled data. We ask: given the abundant video data containing interaction-related knowledge available as a rich "corpus", can a similar generative pre-training approach be effectively applied to enhance robot learning? The key challenge is to identify an effective representation for autoregressive pre-training that benefits robot manipulation tasks. Inspired by the way humans learn new skills through observing dynamic environments, we propose that effective robotic learning should emphasize motion-related knowledge, which is closely tied to low-level actions and is hardware-agnostic, facilitating the transfer of learned motions to actual robot actions.
To this end, we introduce Moto, which converts video content into latent Motion Token sequences by a Latent Motion Tokenizer, learning a bridging "language" of motion from videos in an unsupervised manner. We pre-train Moto-GPT through motion token autoregression, enabling it to capture diverse visual motion knowledge. After pre-training, Moto-GPT demonstrates the promising ability to produce semantically interpretable motion tokens, predict plausible motion trajectories, and assess trajectory rationality through output likelihood. To transfer learned motion priors to real robot actions, we implement a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control. Extensive experiments show that the fine-tuned Moto-GPT exhibits superior robustness and efficiency on robot manipulation benchmarks, underscoring its effectiveness in transferring knowledge from video data to downstream visual manipulations.
⚙️Quick Start
Installation
Clone the repo:
git clone https://github.com/TencentARC/Moto.git
Install minimal requirements for Moto training and inference:
conda create -n moto python=3.8
conda activate moto
cd Moto
pip install -r requirements.txt
cd ..
[Optional] Setup the conda environment for evaluating Moto-GPT on the CALVIN benchmark:
conda create -n moto_for_calvin python=3.8
conda activate moto_for_calvin
git clone --recurse-submodules https://github.com/mees/calvin.git
pip install setuptools==57.5.0
cd calvin
cd calvin_env; git checkout main
cd ../calvin_models
sed -i 's/pytorch-lightning==1.8.6/pytorch-lightning/g' requirements.txt
sed -i 's/torch==1.13.1/torch/g' requirements.txt
cd ..
sh ./install.sh
cd ..
sudo apt-get install -y libegl1-mesa libegl1
sudo apt-get install -y libgl1
sudo apt-get install -y libosmesa6-dev
sudo apt-get install -y patchelf
cd Moto
pip install -r requirements.txt
cd ..
[Optional] Setup the conda environment for evaluating Moto-GPT on the SIMPLER benchmark:
source /data/miniconda3/bin/activate
conda create -n moto_for_simpler python=3.10 -y
conda activate moto_for_simpler
git clone https://github.com/simpler-env/SimplerEnv --recurse-submodules
pip install numpy==1.24.4
cd SimplerEnv/ManiSkill2_real2sim
pip install -e .
cd SimplerEnv
pip install -e .
sudo apt install ffmpeg
pip install setuptools==58.2.0
pip install tensorflow==2.15.0
pip install -r requirements_full_install.txt
pip install tensorflow[and-cuda]==2.15.1
pip install git+https://github.com/nathanrooy/simulated-annealing
cd ..
cd Moto
pip install -r requirements.txt
cd ..
Model Weights
We release the Latent Motion Tokenizer, the pre-traiend Moto-GPT, and the fine-tuned Moto-GPT in Moto Hugging Face.
You can download them separately and save them in corresponding directories (latent_motion_tokenizer/checkpoints/
and moto_gpt/checkpoints/
).
💻Inference
Latent trajectory inference with the pre-trained Moto-GPT and the Latent Motion Tokenizer
conda activate moto
export PROJECT_ROOT=[your path to Moto project]
cd ${PROJECT_ROOT}/scripts
nohup bash run_latent_motion_generation.sh > run_latent_motion_generation.log 2>&1 &
tail -f run_latent_motion_generation.log
Evaluating the fine-tuned Moto-GPT on robot manipulation benchmarks
Evaluation on CALVIN
conda activate moto_for_calvin
export PROJECT_ROOT=[your path to Moto project]
cd ${PROJECT_ROOT}/scripts
nohup bash evaluate_moto_gpt_in_calvin.sh > evaluate_moto_gpt_in_calvin.log 2>&1 &
tail -f evaluate_moto_gpt_in_calvin.log
Evaluation on SIMPLER
conda activate moto_for_simpler
export PROJECT_ROOT=[your path to Moto project]
cd ${PROJECT_ROOT}/scripts
nohup bash evaluate_moto_gpt_in_simpler.sh > evaluate_moto_gpt_in_simpler.log 2>&1 &
tail -f evaluate_moto_gpt_in_simpler.log
📝To Do
- Release the Latent Motion Tokenizer
- Release the pre-trained and fine-tuned Moto-GPT
- Release the inference code
- Release the trainig code
🙌Acknowledgement
This repo benefits from Taming Transformers, Phenaki-Pytorch, GR-1, GR1-Training. Thanks for their wonderful works!