Learning to Grasp Novel Objects

We consider the problem of grasping novel objects, specifically ones that are being seen for the first time through vision. Supervised learning techniques have been utilized to train a robot to grasp both previously seen and novel objects. Here we present learning algorithms that given the image of an object, predict the gripper configuration for grasping. The configuration can be represented as a single point,

or as an oriented rectangle,

Recently, we used deep learning to learn the features below.


Data/Code

Download the data and code for grasping rectangle and for grasping point.


Videos

These videos show the robotic arm picking up previously unknown objects using only vision completely autonomously:

  • Learning to grasp using grasping rectangles:mp4
  • 1 minute video (3-x speed; Jan 31, 2006): mp4, avi, wmv.
  • Unload a dishwasher (2-x speed; Nov 30, 2006): wmv.
  • Fetching a stapler in response to a verbal command (integrated with navigation, object recognition and speech, with Morgan Quigley; May 2, 2007): mov


Publications


  1. Deep Learning for Detecting Robotic Grasps, Ian Lenz, Honglak Lee, Ashutosh Saxena. To appear in Robotics: Science and Systems (RSS), 2013. [PDF coming soon, arXiv]
  2. Learning Hardware Agnostic Grasps for a Universal Jamming Gripper, Yun Jiang, John Amend, Hod Lipson, Ashutosh Saxena. In International Conference on Robotics and Automation (ICRA), 2012. [PDF, video]
  3. Efficient Grasping from RGBD images: Learning using a new Rectangle Representation, Yun Jiang, Moseson Stephen, Ashutosh Saxena. In International Conference on Robotics and Automation (ICRA), 2011. [PDF]
  4. Learning grasp strategies with partial shape information, Ashutosh Saxena, Lawson Wong, and Andrew Y. Ng. In AAAI, 2008. [PDF]
  5. Learning to Open New Doors, Ellen Klingbeil, Ashutosh Saxena, Andrew Y. Ng. In RSS Workshop on Robot Manipulation, 2008. [PDF, More]
  6. Robotic Grasping of Novel Objects using Vision, Ashutosh Saxena, Justin Driemeyer, and Andrew Y. Ng. International Journal of Robotics Research (IJRR), vol. 27, no. 2, pp. 157-173, Feb 2008. [PDF]
  7. A Vision-based System for Grasping Novel Objects in Cluttered Environments, Ashutosh Saxena, Lawson Wong, Morgan Quigley, and Andrew Y. Ng. In International Symposium of Robotics Research (ISRR), 2007. [PDF]
  8. Robotic Grasping of Novel Objects, Ashutosh Saxena, Justin Driemeyer, Justin Kearns, and Andrew Y. Ng. Neural Information Processing Systems (NIPS 19), 2006. [PDF, More]
  9. Learning to grasp novel objects using vision, Ashutosh Saxena, Justin Driemeyer, Justin Kearns, Chioma Osondu, and Andrew Y. Ng. International Symposium on Experimental Robotics (ISER), 2006. [PDF]
  10. Shorter version appeared as:
  11. Learning to grasp novel objects using vision, Ashutosh Saxena, Justin Driemeyer, Justin Kearns and Andrew Y. Ng. Presented in RSS workshop on Robotic Manipulation, 2006.

People

Faculty

Ashutosh Saxenaasaxena at cs.cornell.edu

Students

Ian Lenzianlenz at cs.cornell.edu
Yun Jiangyunjiang at cs.cornell.edu
Stephen Moseson
Marcus Lim
Justin Kerekes
Katie Lee Meusling