What is Google DeepMind RT-2 (Robotics Transformer 2)?

Robot learning is a research field at the intersection of machine learning and robotics. It studies how a robot can acquire new skills or adapt to its environment through learning, which is hard partly because a robot's physical body brings difficulties such as real-time constraints for collecting data. RT-2 (Robotic Transformer 2) is a model that takes camera images and a written instruction and turns them into robot actions. Its makers describe it as a vision-language-action model (one that links images, language and robot actions) that learns from both web and robotics data, and it writes robot actions as text-like tokens so they can be trained alongside web text and images, and the aim is to let a robot interpret commands that were not in its robot training data. Google DeepMind introduced it in July 2023.

Written from primary sources and reference pages, and checked against them by an independent AI fact-checker. Drafted with AI assistance.