Learning Trajectory Preferences

High degree-of-freedom manipulators such as PR2 and Baxter can accomplish a task in many ways, e.g. many possible trajectories for moving an egg container. However, given the task specific contextual information very few ways are desirable to the end user. In this work, we learn such context-driven user preferences via eliciting online feedback from the user which does not need to be an optimal demonstration.

Selected Papers: NIPS'13, ISRR'13
Students: Ashesh Jain, Shikhar Sharma.
Research/Code/Data: project webpage
Popular Press: IEEE Spectrum, ACM TechNews, Daily Mail (UK), Techcrunch, FOX News, Kurzweil AI, CBS News, CNET, NBC News, Huffington Post (UK), Gizmodo, PopSci, Slashdot Front page, French Tribune, Discovery Channel Daily Planet.

Learning Trajectory Preferences


Anticipating Human Activities for Reactive Responses

Anticipating which activities will a human do next (and how) can enable an assistive robot to plan ahead for reactive responses in human environments. We propose a graphical model that captures the rich context of activities and object affordances, and obtain the distribution over a large space of future human activities. Tested on robots performing reactive tasks based on anticipations.

Selected Papers: ICML'13, RSS'13 (best student paper).
Students: Hema Koppula.
Research/Code/Data: project webpage
Popular Press: Kurzweil AI, Wired, The Verge, Time Magazine, LA Times, CBS News, NBC News, Discovery News, National Geographic, FOX News (Studio B), Daily Show.

Anticipating Human Activities


Hallucinating Humans

We show that modeling human-object relationships (i.e., object affordances) gives us a more compact way of modeling the contextual relationships (as compared to modeling the object-object relationships). One key aspect of our work is that the humans may not be even seen by our algorithm! Applications to vision/robotics: 3D perception and arranging a disorganized room.

Selected Papers: ICML'12, ISER'12, CVPR'13 (oral), RSS'13.
Students: Yun Jiang, Marcus Lim.
Research/Code/Data: project webpage
Popular Press: ACM Technews, Scientific Computing, Kurzweil AI, Ubergizmo, Gizmodo, Smart Planet, AzoRobotics, IEEE Spectrum.

Reasoning through Hallucinated Humans


Deep Learning for Grasping

Being able to grasp and pick up objects is critical for a robot to interact with human environments in useful ways. Although a robot should be able to reason about how to grasp any object, even one it has not seen before, it can be difficult to design good features which allow it to do so. In this work, we use a deep neural network to learn these features instead, both avoiding the need to hand-engineer them, and improving the performance of our grasp detection system.

Selected Papers: RSS'13.
Students: Ian Lenz, Shikhar Sharma
Research/Code/Data: project webpage.
Popular Press: MarketWatch.

Detecting Robotic Grasps with Deep Networks


Learning Human Activities from RGBD Videos

Being able to detect human activities is important for making personal assistant robots useful in performing assistive tasks. Our CAD dataset comprises twelve different activities (composed of several sub-activities) performed by four people in different environments, such as a kitchen, a living room, and office, etc. Tested on robots reactively responding to the detected activities. (Code + CAD dataset available.)

Selected Papers: AAAI PAIR'11, ICRA'12, GECCO'12, IJRR'13, ICML'13, RSS'13.
Students: Hema Koppula, Jaeyong Sung, Rudhir Gupta.
Research/Code/Data: project webpage, with CAD-60 and CAD-120 dataset and code.
Popular Press: E&T Magazine, R&D Magazine, Gizmag, GizmoWatch, myScience, WonderHowTo, Geekosystem.

Human activity detection


Learning to Place Objects

The ability to place objects in the environment is an important skill for a personal robot. An object should not only be placed stably, but should also be placed in its preferred location/orientation. For instance, a plate is preferred to be inserted vertically into the slot of a dish-rack as compared to be placed horizontally. In this work, we propose a supervised learning algorithm for finding good placements given the point-clouds of the object and the placing area.

Selected Papers: IJRR'12, ICRA'12, ISER'12.
Students: Yun Jiang, Changxi Zheng, Marcus Lim.
Research/Code/Data: project webpage
Popular Press: Newswise, News Tonight, ACM Technews, Innovation News, LiveScience, UPI, NDTV, CBS WBNG Action News, The Engineer UK, MSNBC Future of tech, Discovery News Nuggets, Kurzweil AI.

Arranging Disorganized Rooms


Detecting Objects (and Attributes) in 3D Scenes

Scene understanding is essential for a robot to perform various tasks. We reason about important geometric properties in addition to the local shape and appearance of an object. We propose a graphical model that captures various features and contextual relations, including the local visual appearance and shape cues, object co-occurence relationships and geometric relationships. Tested on several robots on the task of searching objects. (Code and RGBD dataset available.)

Selected Papers: NIPS'11, IJRR'12, CVPR'13a (oral), CVPR'13b (oral).
Students: Hema Koppula, Abhishek Anand, Zhaoyin Jia, Gaurab Basu.
Research/Code/Data: project webpage
Popular Press: New Scientist, ACM Technews, Newswise, Zee News, News Tonight, Azo Robotics, VoiCE, iNewsOne.

3D Scene Understanding


Learning to Grasp Novel Objects

We consider the problem of grasping novel objects, specifically ones that are being seen for the first time through vision. We present a learning algorithm that predicts, directly as a function of the RGB/RGBD image, a point at which to grasp the object. We apply this algorithm to unload items from a dishwasher, and also to novel grippers such as jamming gripper. (Code available.)

Selected Papers: NIPS'06, ISER'06, IJRR'08, AAAI'08a, AAAI'08b, ICRA'11, ICRA'12, RSS '13.
Students: Ian Lenz, Yun Jiang, Stephen Moseson, Marcus Lim, Justin Driemeyer, Justin Kearns.
Research/Code/Data: project webpage
Popular Press: New York Times (frontpage), Wired Magazine, NBC, ABC, BBC, CBS WBNG Action News, Discovery Channel, the Sunday Times (UK), Mercury News, Apple News (Hong Kong), International Herald Tribune.

Grasping novel objects


Learning to Open New Doors

As robots enter novel, uncertain home and office environments, they are able to navigate these environments successfully. However, to be practically deployed, robots should be able to manipulate their environment to gain access to new spaces, such as by opening a door and by operating an elevator. This, however, remains a challenging problem because a robot will likely encounter doors it has not seen before.

Selected Papers: RSS Manipulation workshop'08, IROS'10.
Research/Code/Data: project webpage

Opening new doors


Holistic Scene Understanding for Robots

One of the original goals of computer vision was to fully understand a natural scene. This requires solving several sub-tasks simultaneously, including object detection, labeling of meaningful regions, and 3D reconstruction. In the past, researchers have developed great classifiers for tackling each of these sub-tasks in isolation. In our work, we have developed machine learning techniques that combine the sub-tasks, without needing to know the inner workings of each classifier. I.e., our method only considers each vision module as a "black-box", allowing us to use very sophisticated, state-of-the-art classifiers without having to look under the hood.

Selected Papers: NIPS'08, NIPS'10, NIPS'11, TPAMI'12.
Students: Congcong Li, Adarsh Kowdle.
Research/Data: project webpage.
Popular Press: New Scientist.

Cascaded Classification Models for Robots.


Manipulating Articulated Objects: Boxes

Given a point-cloud of a scene, we present a method for extracting an articulated 3D model that represents the kinematic structure of an object. We apply this to enable a robot to autonomously close boxes of several shapes and sizes. Such ability is of interest to a personal assistant robot, as well as to commercial robots in applications such as packaging and shipping.

Selected Papers: RSS Workshop on Mobile Manipulation, 2011.
Students: Heran (Paul) Yang, Tiffany Low, Matthew Cong.
Research/Code/Data: project webpage

Manipulating articulated objects, e.g., boxes