Scene Understanding for Personal Robots

See project description
Corresponding Author: Hema Koppula (


The code has two main parts:
1. Processing Point Clouds: Segmentation, Stitching, Feature Generation, etc.
2. Learning and Inference
For now, we only support live classification. Check the README inside for detailed instructions on running the code.

Download Code (v1.0)

(v1.0 released ... with much better precision and faster running time )

Additional code: Scene Parsing: obj-to-pcd and Model RANSACer.

If you use the data or code, please cite the following:

Contextually Guided Semantic Labeling and Search for 3D Point Clouds, Abhishek Anand, Hema S. Koppula, Thorsten Joachims, Ashutosh Saxena. In IJRR, 2012. [PDF]

Semantic Labeling of 3D Point Clouds for Indoor Scenes, Hema S. Koppula, Abhishek Anand, Thorsten Joachims, Ashutosh Saxena. In NIPS, 2011. [PDF]

Data: Cornell-RGBD-Dataset

This data set has 24 labeled office scene point clouds and 28 labeled home scene point clouds. The files are in pcd format with the following fields: x, y, z, rgb, cameraIndex, distance_from_camera, segment_number and label_number. The label string corresponding to the label_number can be obtained from the file scene_labelling/scene_processing/labels.txt (for example, 1 is 'wall', 2 is 'floor', etc.).

Scene Pointcloud Scene Depth Map Sampled Images

Sample scenes from our dataset: First three rows show office scenes and last three rows show home scenes.

Stitched Pointclouds:
Office Data: Original Binary Format(compatible with ROS Diamondback)       Ascii Format       Ply Format
Home Data: Original Binary Format(compatible with ROS Diamondback)       Ascii Format       Ply Format

Single Frame Pointclouds
Due to it's large size, we have not made all the single frames available here. Instead, we subsampled them to ensure minimum overlap. The filename format is scene[scene_number]_[frame_number].pcd . Scene numbers are consistent with the stitched pointclouds
Office Data
Home Data

Single Frame Images
for each image file, there exists a text file with same name containg the matrix of labels in CSV(tab-separated values) format.
Office Data
Home Data

Extracted Features and Labels:
This data has the extracted features along with the lables for the above mentioned home and office scenes : Feature Data.
Check the README for more details.

Bag files containing original frames:
These bag files contain the recorded output of RGBDSLAM for each scene. (For some scenes where RGBDSLAM's output was too off, we manually stitched then converted to same format). Although RGBDSLAM subsamples some frames, points inside frames have not been subsampled. For every message in topic /rgbdslam/my_clouds containing a frame, there is a message in topic /tf with exactly same timestamp containing the transform-matrix for that frame w.r.t the first frame(the frames are already transformed using this matrix). Note that frame corresponding to cameraIndex=c in stitched pointcoud is the frame with 0-based index of (c-1)*5 in the corresponding bag-file uploaded here.
Office Data
Home Data