Through collaboration with academic and government institutes, NIST is critically evaluating and attempting to improve measurement quality in clinical mass spectrometric proteomic investigations. This research aims to improve the repeatability and reproducibility of data for both qualitative proteomics for clinical biomarker discovery as well as targeted quantitative proteomics for biomarker verification. NIST is developing data quality assessment tools and certified reference materials for the proteomics community.
The field of mass spectrometry-based proteomics has expanded rapidly in the last decade. This rapid growth has occurred in the absence of any community standards on measurement quality in qualitative and quantitative proteomics or a thorough assessment of measurement quality across all mass spectrometry instrument platforms used in proteomics. In the limited interlaboratory assessments of proteomics that have been published in recent years, a rather bleak picture of poor repeatability within labs and even worse reproducibility between labs for proteomic investigations on common samples has emerged. For the field of clinical proteomics, which focuses on the discovery and verifications on new protein biomarkers for disease diagnosis and prognosis, poor measurement performance has been particularly troubling. In light of the extensive amount of research dollars spent and the high hopes that proteomics will deliver new clinical biomarkers, growing evidence of poor proteomics measurement performance has greatly reduced confidence in the field. Additionally, poor proteomic data quality leads to high false positive and false negative rates in protein biomarker discovery. A high false positive discovery rate means that much time and money will be spend verifying biomarkers that should have been eliminated from consideration during the discovery phase. A high false negative rate reduces the likelihood that the proteomic investigation will yield its desired outcome, namely, new clinical biomarkers.
In 2006 the National Cancer Institute (NCI) established the 5-year Clinical Proteomics Technology Assessment for Cancer (CPTAC) program which aims to assess and improve measurement quality in clinical proteomics for cancer biomarker discovery and verification. NIST has been collaborating with the CPTAC program (including NCI as well as the 5 academic centers funded through CPTAC) in proteomic interlaboratory studies. Through these interlaboratory studies, NIST is identifying the major sources of variability in qualitative and quantitative proteomics. With these sources identified, NIST has developed computational tools to evaluate proteomic instrument performance and is also developing reference materials that can be used in concert with computational tools. Through this effort, NIST will deliver to the proteomics community methods and materials that can be used to evaluate measurement performance and proteomic data quality. The impact of this will be high quality, more confident proteomic data yielding, for clinical proteomics, more candidate biomarkers for clinical validation.
Additional Technical Details:
The first technical challenge of the CPTAC interlaboratory program was to be able to process and manage extremely large data files from a variety of mass spectrometry instrument manufacturers and platform designs at different geographic locations. NIST developed methods for curating and archiving proteomic data and a data analysis pipeline that could process divergent data. Achieving this technical milestone made it possible to pursue a variety of interlaboratory studies within the CPTAC program; in the first two years of the program NIST helped to administer eight CPTAC studies. In addition to supporting the proteomic data, NIST has produced the materials for CPTAC interlaboratory studies as well as provided technical input on study design. The materials that have been produced include human plasma and yeast lysates spiked quantitatively with several target proteins either to be “discovered” or quantified.
Participation in the CPTAC program and evaluation of the data from CPTAC interlaboratory studies has lead to the development at NIST of both proteomic data quality assessment tools and proteomic reference materials. The data assessment tools are based on the identification of more than 40 chromatographic and mass spectrometric instrument performance metrics. These metrics and their corresponding data tools are being used in the design of a certified reference material for “shotgun” proteomics.
- The development of data analysis and data management tools capable of evaluating data from a diverse field of mass spectrometric proteomic instrument platforms.
- Evaluation of repeatability and reproducibility in clinical “shotgun” mass spectrometry-based proteomics through a multi-site, multi-platform interlaboratory comparison study
- Evaluation of repeatability and reproducibility in clinical quantitative mass spectrometry-based proteomics through a multi-site, multi-platform interlaboratory comparison study.
- Development of computational tools and certified reference materials for the assessment of instrument performance and the evaluation of technical variability in “shotgun” mass spectrometry-based proteomics.
October 1, 2006
Lead Organizational Unit:
National Cancer Institute, Bethesda, MD
Broad Institute of MIT and Harvard, Cambridge, MA
Vanderbilt University School of Medicine, Nashville, TN
University of California-San Francisco, San Francisco, CA
Purdue University, West Lafayette, IN
Memorial Sloan-Kettering Cancer Center, New York, NY
Fred Hutchinson Cancer Research Center, Seattle WA
University of Victoria, Victoria BC, Canada
Plasma Proteome Institute, Washington, DC
Skirball Institute of New York University, New York, NY
Monarch Life Sciences, West Lafayette, IN
Buck Institute for Age Research, Novato, CA
Massachusetts General Hospital, Boston, MA
University of Utah, Salt Lake City, UT
Texas A & M University, College Station, TX
National Institute of Statistical Sciences, Research Triangle Park, NC
David M. Bunk
Stephen E. Stein
Karen W. Phinney
Nathan G. Dodder
Paul A. Rudnick
Lisa E. Kilpatrick
Johanna E. Camara
Related Programs and Projects:
Rudnick, P.A., Clauser, K.R., Kilpatrick, L.E., Tchekhovskoi, D.V., Neta, P., BillHeimer, D.D., Blackman, R.K., Bunk, D.M., Cardasis, H.L., Carr, S.A., Ham, A-J.L., Kinsinger, C.R., Neubert, T.A., Paulovich, A.G., Schilling, B., Spiegelman, C., Tabb, D.L., Tegler, T.J., Wang, M., Wang, P., Whiteaker, J.R., Zimmerman, L.J., Liebler, D.C., Stein, S.E., “Performance Metrics for Evaluating Liquid-Chromatography Tandem Mass Spectrometry Systems in Shotgun Proteomics” submitted to Nature Meth. (December 2008).
Tabb., D.L., Vega-Montoto, L., Variyath, A.M., Ham, A-J.L., Bunk, D.M., Kilpatrick, L.E., Rudnick, P.A., Cardasis, H., Clauser, K.R., Jaffe, J., Kowalski, K.A., Neubert, T.A., Tegeler, T.J., Wang, M., Whiteaker, J., Paulovich, A.G., Liebler, D.C., Spiegelman, C., “Reproducibility and Repeatability in CPTAC Consortium Proteomic Analyses by Liquid Chromatography-Tandem Mass Spectrometry” submitted to Nature Meth. (December 2008).
Paulovich, A.G., Billheimer, D., Ham, A-J. L., Vega-Montoto, L.J., Rudnick, P.A., Tabb, D.L., Wang, P., Blackman, R.K., Bunk, D.M., Cardasis, H., Clauser, K.R., Kinsinger, C.R., Schilling, B., Tegeler, T.J., Variyath, A.M., Wang, M., Whiteaker, J.R., Zimmerman, L.J., Carr, S.A., Fisher, S.J., Gibson, B.W., Mesri, M., Neubert, T.A., Regnier, F.E., Rodriguez, H., Spiegelman, C., Stein, S.E., Tempst, P., Liebler, D.C., “A Reference Yeast Proteome for Benchmarking LC-MS Platform Performance” submitted to Nature Meth. (December 2008).
Addona, T., Abbatiello, S.E., Schilling, B., Skates, S.J., Mani, D.R., Bunk, D. M., Spiegelman, C.H., Zimmerman, L.J., Ham, A-J.L., Keshishian, H., Hall, S.C., Allen, S., Blackman, R.K., Borchers, C.H., Buck, C., Cardasis, H.L., Cusack, M.P., Dodder, N.G., Gibson, B.W., Held, J.M., Hiltke, T., Jackson, A., Johansen, E.B., Kinsinger, C.R., Li, J., Mesri, M., Neubert, T.A., Niles, R.K., Pulsipher, T.C., Rodriguez, H., Rudnick, P.A., Smith, D., Tabb, D.L., Tegeler, T.J., Variyath, A.M., Vega-Montoto, L.J., Wahlander, A., Waldemarson, S., Wang, M., Whiteaker, J.R.; Zhao, L., Anderson, N.L., Fisher, S.J., Liebler, D.C., Paulovich, A.G., Regneir, F.E., Tempst, P., Carr, S.A., “A Multi-site Assessment of Precision and Reproducibility of Multiple Reaction Monintoring-based Measurements:
Toward Quantitative Protein Biomarker Verification in Human Plasma” submitted to Nature Biotech. (January 2009).