Finding Actors and Actions in Movies

People

Abstract

We address the problem of learning a joint model of actors and actions in movies using weak supervision provided by scripts. Specifically, we extract actor/action pairs from the script and use them as constraints in a discriminative clustering framework. The corresponding optimization problem is formulated as a quadratic program under linear constraints. People in video are represented by automatically extracted and tracked faces together with corresponding motion features. First, we apply the proposed framework to the task of learning names of characters in the movie and demonstrate significant improvements over previous methods used for this task. Second, we explore the joint actor/action constraint and show its advantage for weakly-supervised action learning. We validate our method in the challenging setting of localizing and recognizing characters and their actions in the feature length movie Casablanca.

Paper

[ICCV 2013 Paper] [Poster]

BibTeX

@InProceedings{Bojanows13finding,
	author      = "Bojanowski, Piotr and Bach, Francis and Laptev, Ivan and Ponce, Jean and Schmid, Cordelia and Sivic, Josef",
	title       = "Finding Actors and Actions in Movies",
	booktitle   = "Proc. ICCV",
	year        = "2013"
}

Face Pipeline Code

GitHub project page : face-pipeline
Packaged Code
Video frames dump

Weakly Supervised Person and Action Identification Code

GitHub project page : actor-action
Packaged Code
Pre-computed data

Acknowledgements

This research was supported in part by the ERC advanced grant VideoWorld, the ERC starting grant Activia, the ERC advanced grant Allegro, Quaero project, the MSR-INRIA laboratory, the European integrated project AXES, Google, the EIT ICT Labs and the Institut Universitaire de France.

Copyright Notice

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright.