Scene semantics from long-term observation of people

People

• Vincent Delaitre
• David F. Fouhey
• Ivan Laptev
• Josef Sivic
• Abhinav Gupta
• Alexei A. Efros

Abstract

Our everyday objects support various tasks and can be used by people for dirent purposes. While object classification is a widely studied topic in computer vision, recognition of object function, i.e., what people can do with an object and how they do it, is rarely addressed. In this paper we construct a functional object description with the aim to recognize objects by the way people interact with them. We describe scene objects (sofas, tables, chairs) by associated human poses and object appearance. Our model is learned discriminatively from automatically estimated body poses in many realistic scenes. In particular, we make use of time-lapse videos from YouTube providing a rich source of common human-object interactions and minimizing the ert of manual object annotation. We show how the models learned from human observations significantly improve object recognition and enable prediction of characteristic human poses in new scenes. Results are shown on a dataset of more than 400,000 frames obtained from 146 time-lapse videos of challenging and realistic indoor scenes.

Paper

ECCV Paper (pdf) Citation   Vincent Delaitre, David F. Fouhey, Ivan Laptev, Josef Sivic, Abhinav Gupta, Alexei Efros. Scene semantics from long-term observation of people. In Proc. 12th European Conference on Computer Vision. 2012. BibTeX @inproceedings{delaitre2012, author = {Delaitre, V. and Fouhey, D. and Laptev, I. and Sivic, J. and Gupta, A. and Efros, A.}, title = {Scene semantics from long-term observation of people}, booktitle = {Proc. 12th European Conference on Computer Vision}, year = {2012}, }

Data

• Database (2.3Mb ZIP file) : code for downloading the 146 videos dataset.


• Pose detectors (167Mb ZIP file) : code and data for training the pose detectors.


• Ground truth pose annotations (400Kb ZIP file) : over 1500 annotated poses (at least 10 per video).

Code

• Scene semantics source code (Coming Soon!) : code used to compute results reported in the paper.


• Annotation tools (Coming Soon!) : tools used to create video annotations.

Extended Results

• Segmentation results for different method over all videos.
  
  
  
  
• Spatial locations of objects relative to the 6 most contributing poses for all labels:
  
  

  Sofa / Armchair
  Table
  Cupboard

Related Works

D. Fouhey, V. Delaitre, A. Gupta, A. Efros, I. Laptev and J. Sivic
People Watching: Human Actions as a Cue for Single-View Geometry.
In Proc. ECCV 2012.

Funding

This research is supported by:

• NSF Graduate Research Fellowship for David Fouhey
• ONR-MURI Grant N000141010934
• Qauero
• OSEO
• MSR-INRIA
• EIT-ICT
• ERC Grant Videoworld

Copyright Notice

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright.