About
Google Scholar Profile

I am a fifth-year Ph.D. student in the Department of Computer Science of the University of Pittsburgh. Before studying at Pitt, I worked as a software engineer at Baidu Inc. for five years.

I am now advised by Dr. Kovashka, working on the thesis study "Multimodal knowledge integration for object detection and visual reasoning". My research interests lie broadly in the combination of computer vision and natural language processing, including multi-modal learning, knowledge representation, weakly supervised object detection and scene parsing. I had also worked on mobile object detection and face thumbnailing studies.

I got both of my bachelor's and master's degrees (2004-2011) from Beihang University (previously known as Beijing Univeristy of Aeronautics and Astronautics). My advisor in the graduate school of Beihang was Dr. Jiang.


Experience
(2021.9 - ?) Senior Applied Research Scientist at Cruise, San Francisco, United States
(2019.6 - 2019.8) Ph.D. Intern at Mobile Vision Team, Google, Paris, France
(2018.6 - 2018.8) Ph.D. Intern at Video Content Analysis Team, Google, Zurich, Switzerland
(2017.5 - 2017.8) Ph.D. Intern at Mobile Vision Team, Google, Los Angeles, United States
(2016.5 - 2016.8) Ph.D. Intern at Mobile Vision Team, Google, Los Angeles, United States
(2013.9 - 2015.8) Senior Software Engineer at Data and Recommendation Team, Baidu, Beijing, China
(2011.3 - 2013.8) Senior Software Engineer at Peer-to-Peer Team, Baidu, Beijing, China

Publications

Linguistic Structures as Weak Supervision for Visual Scene Graph Generation.
Keren Ye, Adriana Kovashka.
To appear, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021. [pdf] [project] [poster] [video]

A Case Study of the Shortcut Effects in Visual Commonsense Reasoning.
Keren Ye, Adriana Kovashka.
Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), February 2021. [pdf] [poster] [project]

Breaking Shortcuts by Masking for Robust Visual Reasoning.
Keren Ye, Mingda Zhang, Adriana Kovashka.
Proceedings of the Winter Conference on Applications of Computer Vision (WACV), Janurary 2021. [pdf] [supp]

SpotPatch: Parameter-Efficient Transfer Learning for Mobile Object Detection.
Keren Ye, Adriana Kovashka, Mark Sandler, Menglong Zhu, Andrew Howard, Marco Fornoni.
Proceedings of the Asian Conference on Computer Vision (ACCV), November 2020. (Oral) [pdf]

SpotPatch: Parameter-Efficient Transfer Learning for Mobile Object Detection (Extended Abstract).
Keren Ye, Adriana Kovashka, Mark Sandler, Menglong Zhu, Andrew Howard, Marco Fornoni.
Transferring and Adapting Source Knowledge in computer Vision, hold in conjunction with ECCV 2020 (ECCV Workshop), August 2020. [pdf]

Story Completion with Explicit Modeling of Commonsense Knowledge (Extended Abstract).
Mingda Zhang, Keren Ye, Rebecca Hwa, Adriana Kovashka.
Minds vs. Machines: How far are we from the common sense of a toddler?, held in conjunction with IEEE Conference on Computer Vision and Pattern Recognition (CVPR Workshop), June 2020. [pdf] [video]

Interpreting the Rhetoric of Visual Advertisements.
Keren Ye, Narges Honarvar Nazari, James Hahn, Zaeem Hussain, Mingda Zhang, Adriana Kovashka.
Transactions of Pattern Analysis and Machine Intelligence (TPAMI), 2019. [pdf]

Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detection.
Keren Ye, Mingda Zhang, Adriana Kovashka, Wei Li, Danfeng Qin, Jesse Berent.
Proceedings of the International Conference on Computer Vision (ICCV), October 2019. [pdf] [supp] [poster] [project]

ADVISE: Symbolism and External Knowledge for Decoding Advertisements.
Keren Ye, Adriana Kovashka.
Proceedings of the European Conference on Computer Vision (ECCV), September 2018. [pdf] [supp] [poster] [project]

Story Understanding in Video Advertisements.
Keren Ye, Kyle Buettner, Adriana Kovashka.
Proceedings of the British Machine Vision Conference (BMVC), September 2018. [pdf] [poster] [project]

Automatic Understanding of Image and Video Advertisements.
Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, Adriana Kovashka.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. (Spotlight) [pdf] [supp] [poster] [project]

Teaching
(2018.9 - 2018.12) CS1501 Algorithm Implementation
(2018.1 - 2018.4) CS1501 Algorithm Implementation
(2017.9 - 2017.12) CS1501 Algorithm Implementation
(2017.1 - 2017.4) CS2770 Computer Vision
(2016.1 - 2016.4) CS1501 Algorithm Implementation
(2015.9 - 2015.12) CS1501 Algorithm Implementation

Contact
Email: yekeren.cn@gmail.com
Phone: +1 (412) 999-3248
Office: 5404 Sennott Square
210 South Bouquet Street,
Pittsburgh, PA 15213, USA