Contributions to ROC methodology from the Department of Radiology at The University of Chicago

1970s

  • Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of signal detection theory. Radiology 105: 199-200, 1972.
  • Goodenough DJ, Metz CE, Lusted LB. Caveat on use of the parameter d’ for evaluation of observer performance. Radiology 106: 565-566, 1973.
  • Metz CE, Goodenough DJ. On failure to improve observer performance with scan smoothing: a rebuttal (letter to the editor). J. Nucl. Med. 14: 873-876, 1973.
  • Metz CE, Goodenough DJ, Rossmann K. Evaluation of receiver operating characteristic curve data in terms of information theory, with applications in radiography. Radiology 109: 297-303, 1973.
  • Goodenough DJ, Metz CE. Effect of listening interval on auditory detection performance. J. Acoust. Soc. Am. 55: 111-116, 1974.
  • Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology 110: 89-95, 1974.
  • Goodenough DJ, Metz CE. Implications of a “noisy” observer to data processing techniques. In: Information Processing in Scintigraphy (C Raynaud and AE Todd-Pokropek, eds.). Orsay, France: Commissariat à l’Energie Atomique, Département de Biologie, Service Hospitalier Frédéric Joliot, 1975, pp. 400-419.
  • Metz CE, Goodenough DJ. Quantitative evaluation of human visual detection performance using empirical receiver operating characteristic curves. In: Information Processing in Scintigraphy (CE Metz, SM Pizer, and GL Brownell, eds.). Oak Ridge, Tennessee: USERDA Tech. Info. Center (Publication CONF-73-0687), 1975, pp. 140-152.
  • Metz CE, Starr SJ, Lusted LB, Rossmann K. Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Information Processing in Scintigraphy (C Raynaud and AE Todd-Pokropek, eds.). Orsay, France: Commissariat à l’Energie Atomique, Département de Biologie, Service Hospitalier Frédéric Joliot, 1975, pp. 420-439.
  • Starr SJ, Metz CE, Lusted LB, Goodenough DJ. Visual detection and localization of radiographic images. Radiology 116: 533-538, 1975.
  • Metz CE, Starr SJ, Lusted LB. Observer performance in detecting multiple radiographic signals: prediction and analysis using a generalized ROC approach. Radiology 121: 337-347, 1976.
  • Metz CE. An overview of some measures of image quality. Proc. SPIE 127: 4-5, 1977.
  • Metz CE. Empirical evaluation of diagnostic medical images using Receiver Operating Characteristic (ROC) analyses. In: Advances in the Psychophysical and Visual Aspects of Image Evaluation (RP Dooley, ed.). Washington, D.C.: SPSE, 1977, pp. 111-114.
  • Metz CE, Starr SJ, Lusted LB. Quantitative evaluation of medical imaging. In: Medical Radionuclide Imaging, Volume 1. Vienna: International Atomic Energy Agency, 1977, pp. 491-504.
  • Metz CE, Starr SJ, Lusted LB. Quantitative evaluation of visual detection performance in medicine: ROC analysis and determination of diagnostic benefit. In: Medical Images: Formation, Perception and Measurement (GA Hay, ed.). London: John Wiley & Sons, 1977, pp. 220-241.
  • Starr SJ, Metz CE, Lusted LB. Comments on generalization of Receiver Operating Characteristic analysis to detection and localization tasks (letter to the editor). Phys. Med. Biol. 22: 376-379, 1977.
  • Metz CE. Basic principles of ROC analysis. Sem. Nucl. Med. 8: 283-298, 1978.
  • Metz CE. Applications of ROC analysis in diagnostic image evaluation. In: The Physics of Medical Imaging: Recording System Measurements and Techniques (AG Haus, ed.). New York: American Institute of Physics, 1979, pp. 546-572.
  • Metz CE. Evaluation of medical imaging in terms of receiver operating characteristic (ROC) curves and decision analysis. Postepy Fizyki Medyczney 14: 47-58, 1979.

1980s

  • Metz CE. Introduction to observer performance tests. In: Optimization of Chest Radiography. Rockville, Maryland: Bureau of Radiological Health (HHS Publication [FDA] 80-8124), 1980, pp. 8-15.
  • Metz CE, Kronman HB. A test for the statistical significance of differences between ROC curves. In: Information Processing in Medical Imaging (R DiPaola and E Kahn, eds.). Paris: INSERM (Vol. 88), 1980, pp. 647-660.
  • Metz CE, Kronman HB. Statistical significance tests for binormal ROC curves.  J. Math. Psychol. 22: 218-243, 1980.
  • Loo L-N, Doi K, Metz CE. A comparison of image quality indices and observer performance in the radiographic detection of nylon beads. Phys. Med. Biol. 29: 837-856, 1984.
  • Metz CE, Wang P-L, Kronman HB. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Information Processing in Medical Imaging (F Deconinck, ed.). The Hague: Nijhoff, 1984, pp. 432-445.
  • Metz CE. Evaluation of image quality by ROC analysis: concepts, techniques, and future possibilities. Jpn. J. Radiol. Technol. 4: 990-1002, 1985.
  • Loo L-N, Doi K, Metz CE. Reply to “Importance of internal noise in models of observer performance” (letter to the editor). Phys. Med. Biol. 30: 267-268, 1985.
  • Wagner RF, Metz CE, Brown DG. Signal detection theory and medical image assessment. In: Recent Developments in Digital Imaging (K Doi, L Lanzl, and P-J Lin, eds). New York: American Institute of Physics, 1985, pp. 39-59.
  • Metz CE. ROC methodology in radiologic imaging. Invest. Radiol. 21: 720-733, 1986.
  • Metz CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Multiple Regression Analysis: Applications in the Health Sciences (D Herbert and R Myers, eds.). New York: American Institute of Physics, 1986, pp. 365-384.
  • Metz CE. Current problems in ROC analysis. In: Proceedings of the Chest Imaging Conference 1987 (WW Peppler and AA Alter, eds). Madison, Wisconsin: Department of Medical Physics, University of Wisconsin-Madison, 1988, pp. 315-336.
  • Metz CE. Quality of the observed image. ICRU News 2: 20-23, 1989.
  • Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest. Radiol. 24: 234-245, 1989.

1990s

  • Begg CB, Metz CE. Consensus diagnoses and “gold standards” (letter to the editor). Med. Decis. Making 10: 29-30, 1990. [Erratum: Med. Decis. Making 10: 149, 1990.]
  • Gur D, Rockette HE, Good W, Slasky BS, Cooperstein LA, Straub WH, Obuchowski NA, Metz CE. Effect of observer instruction on ROC study of chest images. Invest. Radiol. 25: 230-234, 1990.
  • Metz CE. Some applications of ROC analysis in digital radiology. In: Jpn. J. Med. Imaging and Info. Sci. 7: 14-16, 1990.
  • Rockette HE, Gur D, Cooperstein LA, Obuchowski NA, King JL, Fuhrman CR, Tabor EK, Metz CE. Effect of two rating formats in multi-disease ROC study of chest images. Invest. Radiol. 25: 225-229, 1990.
  • Rockette HE, Obuchowski N, Metz CE, Gur D. Statistical issues in ROC curve analysis. Proc. SPIE 1234: 111-119, 1990.
  • Straub WH, Rockette HE, King JL, Obuchowski NA, Good WA, Feist JH, Good BC, Metz CE. Training observers for receiver operating characteristic (ROC) studies. Proc. SPIE 1234: 126-130, 1990.
  • King JL, Gur D, Rockette HE, Curtin HD, Obuchowski NA, Thaete FL, Britton CA, Metz CE. Radiologists’ confidence in detecting abnormalities on chest images and their subjective judgments of image quality. Proc. SPIE 1446: 268-275, 1991.
  • Metz CE. ROC methodology for the evaluation of PACS. In: A New Horizon in Medical Physics and Biomedical Engineering (H. Abe, K. Atsumi, T. Iinuma, M. Saito, and M. Inoue, eds.). Amsterdam: Elsevier Science Publishers, 1991, pp. 89-97
  • Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest. Radiol. 27: 723-731, 1992.
  • Metz CE. Evaluation of medical images. In: The Formation, Handling, and Evaluation of Medical Images (AE Todd-Pokropek and MA Viergever, eds.). Berlin: Springer-Verlag, 1992, pp. 277-302.
  • Metz CE, Shen J-H. Gains in accuracy from replicated readings of diagnostic images: prediction and assessment in terms of ROC analysis. Med. Decis. Making 12: 60-75, 1992.
  • Rockette HE, Gur D, Metz CE. The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Invest. Radiol. 27: 169-172, 1992.
  • Metz CE. Quantification of failure to demonstrate statistical significance: the usefulness of confidence intervals. Invest. Radiol. 28: 59-63, 1993.
  • Metz CE. Foreword. In: Fundamentals and Applications of ROC Analysis. Kyoto: Japanese Society of Radiological Technology, 1994.
  • Metz CE. Recent progress in ROC analysis. Jpn. J. Radiol. Technol. 50: 1915-1922, 1994.
  • Nishikawa RM, Giger ML, Doi K, Metz CE, Yin FF, Vyborny CJ, Schmidt RA. Effect of case selection on the performance of computer-aided detection schemes. Med. Phys. 21: 265-269, 1994.
  • Dorfman DD, Metz CE. Multi-reader multi-case ROC analysis: comments on Begg’s commentary. Acad. Radiol. 2 (Supplement 1): S76-S78, 1995.
  • Metz CE. Evaluation of radiologic imaging systems by ROC analysis. Med. Imag. Inform. Sci. 12: 113-121, 1995.
  • Metz CE, Wagner RF, Doi K, Brown DG, Nishikawa RN, Myers KJ. Toward consensus on quantitative assessment of medical imaging systems. Med. Phys. 22: 1057-1061, 1995.
  • Halpern EJ, Alpert M, Krieger AM, Metz CE, Maidment AD. Comparisons of ROC curves on the basis of optimal operating points. Acad. Radiol. 3: 245-253, 1996.
  • Jiang Y, Metz CE, Nishikawa RM. A receiver operating characterisitc partial area index for highly sensitive diagnostic tests. Radiology 201: 745-750, 1996.
  • Metz CE. Evaluation of digital mammography by ROC analysis. In: Digital Mammography ‘96 (K Doi, ML Giger, RM Nishikawa, RA Schmidt, eds.). Amsterdam: Elsevier Science (Excerpta Medica International Congress Series, No. 1119), 1996, pp. 61-68.
  • Dorfman DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA. Proper ROC analysis: the bigamma model. Acad. Radiol. 4: 138-149, 1997.
  • Metz CE. Receiver operating characteristic (ROC) analysis in medical imaging. ICRU News (June 1997 issue; no vol. #): 7-16, 1997.
  • Pan X, Metz CE. Non-iterative methods and their noise characteristics in 2D SPECT image reconstruction. IEEE Trans. Nucl. Sci. 44: 1388-1397, 1997.
  • Pan X, Metz CE. The “proper” binormal model: parametric ROC curve estimation with degenerate data. Acad. Radiol. 4: 380-389, 1997.
  • Roe CA, Metz CE. The Dorfman-Berbaum-Metz method for statistical analysis of multi-reader, multi-modality ROC data: validation by computer simulation. Acad. Radiol. 4: 298-303, 1997.
  • Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets. Med. Decis. Making 18: 110-121, 1998.
  • Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Statist. Med. 17: 1033-1053, 1998.
  • Roe CA, Metz CE. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad. Radiol. 4: 587-600, 1997.
  • Engelmann R, MacMahon H, Metz CE, Hoffmann K, Doi K. Implementation of a technique for performing real-time ROC observer studies to examine the efects of CAD schemes on the performance of radiologists. In: Computer-Aided Diagnosis in Medical Imaging (K Doi, H MacMahon, ML Giger and KR Hoffmann, eds.). Amsterdam: Elsevier Science (Excerpta Medica International Congress Series, Vol. 1182), pp. 71-76, 1999.
  • Metz CE. Evaluation of CAD methods. In: Computer-Aided Diagnosis in Medical Imaging (K Doi, H MacMahon, ML Giger and KR Hoffmann, eds.). Amsterdam: Elsevier Science (Excerpta Medica International Congress Series, Vol. 1182), pp. 543-554, 1999.
  • Metz CE, Pan X. “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J. Math. Psych. 43: 1-33, 1999.

2000s

  • Metz CE. Fundamental ROC analysis. In: Handbook of Medical Imaging, Vol. 1: Physics and Psychophysics (J Beutel, H Kundel and R Van Metter, eds.). Bellingham, WA; SPIE Press, 2000, pp. 751-769.
  • Metz CE. Software for ROC Analysis. Comm. Imaging Jap. Soc. Radiol. Technol. 23: 2-9, 2000.
  • Beiden SV, Wagner RF, Campbell G, Metz CE, Chan H-P, Nishikawa RM, Schnall MD, Jiang Y. Analysis of variance components in multi-reader studies of computer-aided diagnosis with different tasks. Proc. SPIE 4324: 167-176, 2001.
  • Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y. Components-of-variance models for random-effects ROC analysis: The case of unequal variance structures across modalities. Academic Radiol. 8: 605-615, 2001.
  • Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y, Chan H-P. Multiple-reader studies, digital mammography, computer-aided diagnosis – and the Holy Grail of imaging physics (II). Proc. SPIE 4320: 619-626, 2001.
  • Jiang Y, Metz CE. An optimal method for combining two correlated diagnostic assessments with application to computer-aided diagnosis. Proc. SPIE 4324: 177-183, 2001.
  • Wagner RF, Beiden SV, Metz CE. Continuous vs. categorical data for ROC analysis: Some quantitative considerations. Academic Radiol. 8: 328-334, 2001.
  • Edwards DC, Kupinski MA, Metz CE, Nishikawa RN. Maximum-likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med. Phys. 29: 2861-2870, 2002.
  • Wagner RF, Beiden SV, Campbell G, Metz CE, Sachs WM. Assessment of medical imaging and computer-assist systems: Lessons from recent experience. Academic Radiol. 8: 1264-1277, 2002.
  • Edwards DC, Lan L, Metz CE, Giger ML, Nishikawa RM. Bayesian ANN estimates of three-class ideal observer decision variables for classification of mammographic masses. Proc. SPIE 5034: 474-482, 2003.
  • Edwards DC, Lan L, Metz CE, Giger ML, Nishikawa RM. Estimating three-class ideal observer decision variables for computerized detection and classification of mammographic mass lesions. Med. Phys. 31: 81-90, 2004.
  • Edwards DC, Metz CE, Kupinski MA. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans. Med. Imaging 23: 891-895, 2004.
  • Edwards DC, Metz CE, Nishikawa RM, Hypervolume under the ROC hypersurface of a “near-guessing” ideal observer in a three-class classification task. Proc. SPIE 5372: 128-137, 2004.
  • Edwards DC, Metz CE.  Review of several proposed three-class classification decision rules and their relation to the ideal observer decision rule.  Proc. SPIE 5749: 128-137, 2005.
  • Edwards DC, Metz CE.  Evaluating Bayesian ANN estimates of ideal observer decision variables by comparison with identity functions.  Proc. SPIE 5749: 174-182, 2005.
  • Jiang Y, Sacks W, Metz CE. Effect of observer inattention in a detection task on ROC analysis. Proc SPIE 5749:114-117, 2005.
  • Edwards DC, Metz CE. Restrictions on the three-class ideal observer’s decision boundary lines.  IEEE Trans. Med. Imag. 24:1566-1573, 2005.
  • Edwards DC, Metz CE.  Optimization of an ROC hypersurface constructed only from an observer’s within-class sensitivities.  Proc. SPIE 6146: 61460A1-61460A7, 2006.
  • Edwards DC, Metz CE. Analysis of proposed three-class classification decision rules in terms of the ideal observer decision rule. J. Math. Psychol. 50: 478-487, 2006.
  • He X, Metz CE, Tsui BMW, Links JM, Frey EC.Three-class ROC analysis – A decision theoretic approach under the ideal observer framework. IEEE Trans. Med. Imag. 25 (5): 571-581, 2006.
  • Metz CE. Receiver operating characteristic (ROC) analysis: a tool for quantitative evaluation of observer performance and imaging systems. JACR 3: 413-422, 2006.
  • Edwards DC, Metz CE. A utility-based performance metric for ROC analysis of N-class classification tasks. Proc. SPIE 6515: 6515031–65150310, 2007.
  • Edwards DC, Metz CE.  Optimization of restricted ROC surfaces in three-class classification tasks.  IEEE Trans. Med. Imag. 26: 1345-1356, 2007.
  • Jiang Y, Miglioretti D, Metz CE, Schmidt RA. Designing imaging trials to demonstrate improvements in breast cancer detection rate. Radiology 243: 360-367, 2007.
  • Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Acad. Radiol. 14: 814–829, 2007.
  • Wagner RF, Metz CE, Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad. Radiol. 14: 723–748, 2007.
  • Edwards DC, Metz CE. Optimality of a utility-based performance metric for ROC analysis. Proc. SPIE 6917: 69170F1–68170F6, 2008.
  • Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad. Radiol. 15: 647-661, 2008.
  • Horsch KJ, Giger ML, Metz CE. Potential effect of different radiologist reporting methods on studies showing benefit of CAD. Acad. Radiol. 15: 139-152, 2008.
  • Horsch K, Giger ML, Metz CE. Prevalence scaling: applications to an intelligent workstation for the diagnosis of breast cancer. Acad. Radiol. 15: 1446-1457, 2008.
  • Metz CE. ROC analysis in medical imaging: a tutorial review of the literature. Radiol. Phys. & Technol. 1: 2-12, 2008.
  • Edwards DC, Metz CE. Comparing the performance of two observers using a novel utility-based performance metric for ROC analysis. Proc. SPIE 7263: 72630W1–72630W7, 2009.
  • Shiraishi J, Pesce LL, Metz CE, Doi K. On experimental design and data analysis in receiver operating characteristic (ROC) studies: lessons learned from papers published in RADIOLOGY from 1997 to 2006. Radiology 253: 822-830, 2009.

2010s

  • Nishikawa RM, Jiang Y, Metz CE.  Rating scales for observer performance studies. Proc. SPIE 7627: 7627031-7627037, 2010.
  • Edwards DC, Metz CE. Behavior of the decision variables of the three-class ideal observer for univariate trinormal data. Proc. SPIE 7627: 76270Y1-76270Y11, 2010.
  • Pesce LL, Metz CE, Berbaum KS. On the convexity of ROC curves estimated from radiological test results. Acad. Radiol.17: 960-968, 2010.
  • Shiraishi J, Pesce LL, Metz CE, Doi K. On Experimental Design and Data Analysis in Receiver Operating Characteristic (ROC) Studies: Lessons Learned from Papers Published in RADIOLOGY from 1997 to 2006. Radiology; 253: 822-830, 2009