There is more to images than their objective physical content: for example, advertisements are created to persuade a viewer to take a certain action. We propose the novel problem of automatic advertisement understanding. To enable research on this problem, we create two datasets: an image dataset of 64,832 image ads, and a video dataset of 3,477 ads. Our data contains rich annotations encompassing the topic and sentiment of the ads, questions and answers describing what actions the viewer is prompted to take and the reasoning that the ad presents to persuade the viewer ("What should I do according to this ad, and why should I do it?"), and symbolic references ads make (e.g. a dove symbolizes peace). We also analyze the most common persuasive strategies ads use, and the capabilities that computer vision systems should have to understand these strategies. We present baseline classification results for several prediction tasks, including automatically answering questions about the messages of the ads.



Automatic Understanding of Image and Video Advertisements. Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, Adriana Kovashka. To appear, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. (Spotlight)     [pdf]     [supplementary]     [poster]     [spotlight]



NSF CISE CRII: RI: Automatically Understanding the Messages and Goals of Visual Media (Award #1566270)
      Google Faculty Research Award

This material is based upon work supported by the National Science Foundation under Grant Number 1566270. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.


Image Data and Annotations

Sentiment102,340 Cheerful
Q+A202,090 I should bike because it’s healthy.
Symbol64,131 Danger (+ bounding box)
Strategy20,000 Contrast
Slogan11,130 Save the planet... save you.

Video Data and Annotations

Type Count Example
Topic 17,345 Cars and automobiles, Safety
Sentiment 17,345 Cheerful, Amazed
Action/Reason 17,345 I should buy this car because it is pet-friendly.
Funny? 17,374 Yes/No
Exciting? 17,374 Yes/No
English? 15,380 Yes/No/Does not matter
Effective? 16,721 Not/.../Extremely Effective


For any questions, issues, concerns, and comments, please email Adriana Kovashka at kovashka AT cs DOT pitt DOT edu