BIOCHANGE 2008 PILOT
Pilot Study of
Change Measurement Algorithms & CAD Tools
Organized by the
Report Pilot Results by
16 July 16, 2008
With regard to benchmarking, NIST’s Information Technology Laboratory has experience with performance evaluation of software and algorithms for text retrieval, biometrics, motion image quality, face recognition, speech, and text retrieval. The evaluation of biomedical change analysis algorithms presents very real challenges in developing suitable assessment methods. We expect the use of benchmarking to contribute to algorithm improvement.
In comparison with previous benchmarking work at NIST, the distinctive element of the current study is the lack of available ground truth for much of the clinical imagery data. In the present case, lung CT derived from the medical diagnosis of lung cancer and the assessment of the response to therapy is available, but in many cases the size and the change in size of lung nodules and lesions are not reliably known. The Reference Image Database to Evaluate Response (or RIDER) White Paper discusses approaches to estimating the uncertainty associated with the measurement of change. In the Pilot study we will examine such methods and their applicability to the algorithms and CAD tools submitted to Biochange 2008.
While the Pilot study contains a small number of CT pairs, it is our expectation that even a small data set will provide insight into measuring performance of multiple algorithms and CAD tools for volume change in lung CT. Because we lack access to reliable ground truth, the study will consider the effectiveness of various performance measures and methodologies near the zero-change point. The study will include cases with no clinical change, in order to estimate the local variance and bias of each change detector. The Pilot should provide guidance in conducting future larger studies.
Develop methodology & performance measures to validate change analysis using publicly available lung CT data.
· Participants provide algorithms or CAD tools that measure change in lung nodules.
· NIST gathers pairs of CT scans from various sources: RIDER (Reference Image Database to Evaluate Response) clinical data from NCI, phantom collections from FDA, and possibly other clinical data. The Pilot includes pairs of scans with “No Change” conditions.
· With industry participation, NIST defines evaluation methodology and provides code samples for data access. The design is intended to assure low barriers to entry. The Experiment Description is available in XML format and also as a spreadsheet. The evaluation has two stages, Training and Validation
How to Participate in Biochange 2008 Pilot
· Download the CT scan data from the NIST FTP site and from the National Cancer Institute Archive (NCIA).
· Run your change analysis algorithm or commercial CAD tool in your lab on the Validation data set.
change results in one of the required formats and provide the signed
Participation Agreement to NIST by
May 16 July 16, 2008. Optionally,
Participants may submit their Training stage data by April 21 for preliminary
analysis and discussion at the Lung Cancer Workshop.
· NIST will analyze the results as described in the Biochange2008 Project Plan. NIST will provide Participants with individual analysis of their results. We will publish the results of the evaluation, without identifying individual scores by Participant.
Dr. Charles Fenimore, National Institute of Standards and Technology (NIST)
firstname.lastname@example.org, phone 301-975-2428.
NIST is an agency of the
U.S. Department of Commerce
Site created on March 21, 2008
Last updated: Friday, May 1, 2008