A team of researchers at MIT has developed a new approach to training robots to handle household tasks with more flexibility. Rather than relying on traditional imitation learning, where robots mimic human motions, the researchers have integrated large language models (LLMs) to provide robots with common sense knowledge. By connecting robot motion data with logical subtasks generated by LLMs, the robots can better handle disruptions within a task and self-correct errors along the way. This approach allows robots to adapt to unexpected situations without the need for engineers to program explicit fixes for every potential failure.
In a study presented at the International Conference on Learning Representations, the researchers demonstrated their approach using a simple task of scooping marbles from one bowl and pouring them into another. Typically, engineers would train a robot by physically guiding it through the motions of scooping and pouring multiple times. However, the team found that breaking down the task into subtasks and connecting them to an LLM’s natural language labels allowed the robot to better understand the sequence of actions involved in the task. This enabled the robot to self-correct errors and continue the task without starting from the beginning.
The researchers developed an algorithm to automatically connect the LLM’s subtask labels with the robot’s physical coordinates or image view, a process known as “grounding.” By learning a grounding classifier, the robot can determine which subtask it is in based on its physical state, such as reaching or scooping. This dialogue between the robot’s physical actions and the LLM’s knowledge of subtasks allows the robot to replan and recover from errors in real-time. In experiments with a robotic arm trained on the marble-scooping task, the robot successfully completed the task even when pushed off course or disrupted by the experimenters.
By leveraging the capabilities of LLMs and developing a grounding classifier algorithm, the researchers have created a method for training household robots that can handle complex tasks and recover from errors without human intervention. This approach has the potential to revolutionize the training of robots for household chores, enabling them to adapt to dynamic environments and unforeseen disruptions. With further development and integration, this method could lead to more robust and reliable household robots that can assist with a wide range of tasks, from cleaning to cooking and beyond.
