Ground Truth for a Breast Cancer Detection AI

Challenge

Intrinsic Imaging supported this study’s imaging activities, addressing the scientific and operational considerations that inform the work of an imaging CRO and imaging core lab. An AI developer required a large, independent test dataset to support a standalone performance study of a breast cancer detection algorithm. The evidentiary requirements for such studies are exacting: the reference standard must be established independently of the algorithm, truth determinations must be defensible at the lesion level, and the dataset must remain sequestered from any use in algorithm training. Assembling thousands of cases to these criteria demanded both scale and methodological discipline.

The cases themselves required careful curation. The test set needed to reflect the intended-use population across breast densities, lesion types, imaging systems and cancer-negative examinations. Each positive case required reliable pathology linkage, negative cases needed adequate follow-up support, and annotation had to be precise enough to support lesion-level rather than merely examination-level analysis. Eligibility, annotation, pathology correlation and consensus decisions each needed documented provenance, since weakness in any one could compromise the interpretation of standalone performance even where aggregate results appeared favorable. Examination-level accuracy can conceal differences in lesion localization, so a defensible lesion-level reference standard requires more than a binary cancer label. Maintaining the link between an annotated finding and its supporting clinical evidence preserves the meaning of the standalone performance comparison.

Intrinsic Imaging’s Solution

Intrinsic Imaging worked with the developer to define case selection criteria, annotation conventions and reference-standard rules before annotation began. Cases were reviewed for completeness, image quality and supporting clinical information, and only eligible cases entered the sequestered dataset, which was maintained under strict separation from the algorithm's development throughout the project.

Fellowship-trained breast radiologists performed lesion-level annotation of digital mammography and tomosynthesis examinations within our Part 11 compliant reading environment. Radiologists were trained on the annotation tools and lesion definitions using cases outside the test set. Reference-standard determinations incorporated pathology correlation for positive cases and supporting follow-up for negative cases; expert consensus, under procedures documented in advance, resolved disagreements, with each decision recorded in a complete audit trail from source case to final determination.

Quality reviews were conducted throughout annotation, including checks for missing marks, inconsistent lesion boundaries and discrepancies between imaging findings and pathology records. Issues were resolved under the documented consensus procedures, so that the final reference standard was both complete and internally consistent across the entire dataset. Project managers monitored progress and quality throughout, and the documented methodology was structured so the developer could reuse it to assemble future test sets as the algorithm and its intended uses evolve.

Intrinsic Imaging’s Results

• 5,000+ cases with lesion-level reference standards and complete audit trails, traceable from source examination through pathology correlation to final consensus determination
• 100% of the test set sequestered from algorithm training, preserving its independence as a credible benchmark for standalone performance and subsequent regulatory review
• Standalone performance study completed on schedule, supported by pathology-confirmed truth for every positive case and follow-up-supported determinations for cancer-negative examinations
• Documented curation, annotation and consensus methodology retained by the developer as a reusable template for future test sets as the algorithm and its intended uses evolve