Textual Inference for Retrieving Labeled Object Descriptions

This thesis presents a knowledge-based solution for retrieving English descriptions for objects, such as images, in a collection. Based on detailed analysis of the errors made by a baseline system relying on surface-level features (i.e. term frequency), we infer that an ideal solution to this problem should use deeper representations of the meaning encoded in textual descriptions.

Applied Textual Inference (ATI) as used in this thesis refers to the class of generic task-based evaluations that address this need. ATI tasks are challenge problems. Because they are intended to drive research on text understanding, the problems are designed to be hard enough to require reasoning. However in order to support cross-site comparisons of results, the problems are evaluated at the surface level. Examples include recognizing textual entailment (RTE), paraphrasing, summarization, word-replacement, and some types of question answering (QA).

This thesis frames the problem of image description retrieval as an in stance of ATI, and demonstrates how an inference engine and a set of symbolic knowledge resources in the form of ontologies can improve performance on this task, as measured by Mean Reciprocal Rank. In the process, we describe the results of several sub-tasks: Introduce an image retrieval task supported by a data set containing over 50,000 images, hand-labeled with multiple descriptions; present a series of parameterizations for calculating the similarity between two descriptions; identify classes of error in a keyword-driven baseline system and use these classes to inform a set of knowledge-based improvements; implement and evaluate the knowledge based approach.

The success of shared tasks for ATI in the last decade indicates growth in the field of Natural Language Understanding, and in particular a grow ing interest in deep text representations that can be leveraged by modern machine learning frameworks. The work of this thesis contributes to better understanding of why deep representations are necessary, and how they may be effectively applied.