Research

Dissertation Proposal: Intention Recognition of Grouped Bar Charts in Multimodal Documents

Abstract: Information graphics in popular media are usually utilized to communicate a high-level contextual message. This message is often not repeated in the accompanying text or caption of the graphic; thus, it is necessary to consider information graphics to completely understand a multimodal document. A methodology is presented for automatically recognizing high-level intentions conveyed by grouped bar charts, a type of information graphic. Communicative signals include the salient coloring of bars, their ordering and positioning, evidence from the caption of the graphic, and an estimate of the relative cognitive effort required to recognize a possible message. Evidence is entered into a trained Bayesian network which automatically hypothesizes a graphic's intention.

Applications:

SIGHT (Summarizaing Information GrapHics Textually)
indexing and retrieving information graphics from a digital library using their high-level content
summarization of multimodal documents