Departmental Papers (CIS)

Document Type

Conference Paper

Date of this Version

8-2009

Comments

Louis, A. & Nenkova, A., Automatically Evaluating Content Selection in Summarization Without Human Models, Conference on Empirical Methods in Natural Language Processing, Aug. 2009, doi: anthology/D09-1032

Abstract

We present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, Jensen- Shannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations.

Share

COinS
 

Date Posted: 30 July 2012