Fusing video and text data by integrating appearance and behavior similarity

Georgiy Levchuk; Charlotte Shabarekh

doi:10.1117/12.2014878

28 May 2013 Fusing video and text data by integrating appearance and behavior similarity

Georgiy Levchuk, Charlotte Shabarekh

Proceedings Volume 8751, Machine Intelligence and Bio-inspired Computation: Theory and Applications VII; 875107 (2013) https://doi.org/10.1117/12.2014878
Event: SPIE Defense, Security, and Sensing, 2013, Baltimore, Maryland, United States

Abstract

In this paper, we describe an algorithm for multi-modal entity co-reference resolution and present experimental results using text and motion imagery data sources. Our model generates probabilistic association between entities mentioned in text and detected in video data by jointly optimizing the measure of appearance and behavior similarity. Appearance similarity is calculated as a match between propositionderived entity attributes mentioned in text, and the object appearance classification from video sources. The behavior similarity is calculated based on the semantic information about entity movements, actions, and interactions with other entities mentioned in text and detected in video sources. Our model achieved 79% Fscore for text-to-video entity co-reference resolution; we show that entity interactions present unique features for resolving variability present in text data and ambiguity of visual appearance of entities.

Citation Download Citation

Georgiy Levchuk and Charlotte Shabarekh "Fusing video and text data by integrating appearance and behavior similarity", Proc. SPIE 8751, Machine Intelligence and Bio-inspired Computation: Theory and Applications VII, 875107 (28 May 2013); https://doi.org/10.1117/12.2014878

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available