Performance Confidence Estimation for Automatic Summarization

Loading...
Thumbnail Image
Penn collection
Departmental Papers (CIS)
Degree type
Discipline
Subject
Computer Sciences
Funder
Grant number
License
Copyright date
Distributor
Related resources
Author
Louis, Annie
Contributor
Abstract

We address the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either single- or multi-document inputs. Our labelled corpus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We examine different formulations of the classification task which considerably influence performance. The best results are 84% prediction accuracy for single- and 74% for multi-document summarization.

Advisor
Date of presentation
2009-03-01
Conference name
Departmental Papers (CIS)
Conference dates
2023-05-17T07:17:06.000
Conference location
Date Range for Data Collection (Start Date)
Date Range for Data Collection (End Date)
Digital Object Identifier
Series name and number
Volume number
Issue number
Publisher
Publisher DOI
Journal Issue
Comments
Louis, A. & Nenkova, A., Performance Confidence Estimation for Automatic Summarization, 12th Conference of the European Chapter of the Association for the Computational Linguistics, March-April 2009, doi: http://www.aclweb.org/anthology/E09-1062
Recommended citation
Collection