Computer scientists at MIT’s Computer Science and Artificial Intelligence Laboratory and the University of Toronto have created a Sims-inspired “VirtualHome” system that can simulate detailed household tasks. The idea is to allow “artificial agents” to execute tasks, opening up the possibility of one day teaching robots to do such tasks.
Using crowdsourcing, the researchers created videos that simulate detailed household activities and sub-tasks in eight different scenes, including a living room, kitchen, dining room, bedroom, and home office. A simple model can generate a program from either a video or a textual description, allowing robots to be programmed by naive users, either via natural language or video demonstration.
The researchers have trained the system using nearly 3,000 programs for various activities, which are further broken down into subtasks for the computer to understand. A simple task like “making coffee,” for example, would also include the necessary step, “grabbing a cup.”
The end result is a large database of household tasks described using natural language. Companies like Amazon that are working to develop Alexa-like robotic systems at home could in the future use such data to train their models to do more complex tasks.
Robots could eventually be trained to anticipate personalized wants and needs, which could be especially helpful as assistive technology for the elderly, or for those with limited mobility.
The team hopes to train the robots using actual videos instead of Sims-style simulation videos. That would enable a robot to learn directly by simply watching a YouTube video. The team is also working on a reward-learning system in which the robot gets positive feedback when it does tasks correctly.