Integrated receiver operating characteristic and confidence error metrics for assessing the performance of automatic target recognition systems

David R Parker; Steven C Gustafson; Timothy D. Ross

doi:10.1117/12.542118

2 September 2004 Integrated receiver operating characteristic and confidence error metrics for assessing the performance of automatic target recognition systems

David R Parker, Steven C Gustafson, Timothy D. Ross

Author Affiliations +

Proceedings Volume 5427, Algorithms for Synthetic Aperture Radar Imagery XI; (2004) https://doi.org/10.1117/12.542118
Event: Defense and Security, 2004, Orlando, Florida, United States

Abstract

The ability of certain performance metrics to quantify how well target recognition systems under test (SUT) can correctly identify targets and non-targets is investigated. The SUT assigns a score between zero and one which indicates the predicted probability of a target. Sampled target and non-target SUT score outputs are generated using representative sets of Beta probability densities. Two performance metrics, Area under the Receiver Operating Characteristic (AURC) and Confidence Error (CE) are analyzed. AURC quantifies how well the target and non-target distributions are separated, and CE quantifies the statistical accuracy of each assigned score. CE and AURC are generated for many representative sets of beta-distributed scores, and the metrics are calculated and compared using continuous methods as well as discrete (sampling) methods. Close agreement in results with these methods for AURC is shown. Also shown are differences between calculating CE using sampled data and calculating CE using continuous distributions. These differences are due to the collection of similar sampled scores in bins, which results in CE weighting proportional to the sum of target and non-target scores in each bin. A method for an alternative weighted CE calculation using maximum likelihood estimation of density parameters is identified. This method enables sampled data to be processed using continuous methods.

Citation Download Citation

David R Parker, Steven C Gustafson, and Timothy D. Ross "Integrated receiver operating characteristic and confidence error metrics for assessing the performance of automatic target recognition systems", Proc. SPIE 5427, Algorithms for Synthetic Aperture Radar Imagery XI, (2 September 2004); https://doi.org/10.1117/12.542118

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
12 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Error analysis

Statistical analysis

Automatic target recognition

Receivers

Target recognition

Analytical research

Target detection

Show All Keywords

Keywords/Phrases

Search In:

Publication Years