Pivotal MRMC Study of CAD Software With 14,400 Reviews With and Without CAD
Case Study · AI/CAD
- 14,400+ reviews with and without CAD
- Multi-reader multi-case
- Reader performance change with CAD
Challenge
Intrinsic Imaging supported this study’s imaging activities, addressing the scientific and operational considerations that inform the work of an imaging CRO and imaging core lab. A pivotal multi-reader multi-case study was required to assess the clinical performance of computer-aided detection software, with more than 14,400 reviews to be performed without CAD and then with CAD. The sponsor's claim depended on demonstrating that readers detected more true findings with CAD without an unacceptable increase in false positives, which requires precise control over how and when readers see CAD marks.
With a review volume of this size, the study also needed medical guidance on case selection and reference-standard definition, a reader panel large enough to complete the work within the sponsor's timeline, and quality control that could detect protocol deviations such as incomplete unaided reads before they affected the analysis. The central interpretive question was whether assistance changed diagnostic decisions rather than merely increasing the number of marks. Since the aided assessment followed the unaided assessment of the same case, the resulting comparison concerns an incremental reading workflow, not independent interpretations made without knowledge of an earlier decision.
Intrinsic Imaging’s Solution
Intrinsic Imaging provided medical guidance on the case set, defining inclusion criteria, case-mix targets and the reference-standard methodology, and assembled the cases against a truth established by an independent panel with documented adjudication. An independently established reference standard provides a common basis for classifying reader findings across assistance conditions. Case composition remains important to interpretation because the balance of subtle abnormalities, conspicuous disease and negative examinations can influence both the opportunity for improved detection and the frequency of false alarms.
All case reviewers were provided by Intrinsic Imaging and trained on the reading protocol, which specified unaided interpretation followed by CAD-assisted interpretation under the sequential design appropriate to the device's intended use, with randomized case order and a validated reading platform recording each mark, confidence score and the exact sequence of unaided and aided decisions. Recording the sequence of decisions makes it possible to distinguish the initial interpretation from subsequent changes after assistance. Randomized case order addresses ordering effects, but the sequential design still requires the aided findings to be understood in the context of the reader's earlier unaided assessment of that case.
Quality control tracked completion of both reading phases per case and reader, flagged deviations for correction before the next session, and the final dataset was locked and delivered for the sponsor's MRMC statistical analysis within the quality framework of the company's five ISO certifications. Completeness checks protect the paired comparison by ensuring that an assisted decision has its required unaided counterpart. The review volume provides extensive observations, but those observations remain clustered within readers and cases; their interpretation therefore depends on the planned analysis accounting for these sources of shared variability.
Intrinsic Imaging’s Results
- More than 14,400 reviews completed without and with CAD by a 24-reader panel within the sponsor's timeline
- Sequential unaided-then-aided reading enforced and recorded per case, with 100% of aided reads preceded by a completed unaided read
- Reference standard established by an independent truth panel with documented adjudication for every discordant case
- Dataset delivered in MRMC analysis format with full audit trail, supporting the sponsor's FDA submission
About Intrinsic Imaging, LLC
Intrinsic Imaging is an ISO certified medical imaging CRO and imaging core lab providing comprehensive services in support of Phase I-IV clinical trials and Class I, II, III medical device trials. Intrinsic Imaging is also a world leader in conducting Artificial Intelligence studies.
With its team of more than 650 board-certified and fellowship-trained physicians, supporting over 4000 imaging centers in 64 countries, Intrinsic Imaging’s team has sub-specialization in all therapeutic areas, including Cardiovascular & Thoracic, Central Nervous System, Gastroenterology & Metabolic, Musculoskeletal, Oncology and Ophthalmology.
Certified by the British Standards Institution (BSI), Intrinsic Imaging is the only company in the world to hold five ISO certifications specifically for providing imaging core lab services on clinical trials. For more information, please visit IntrinsicImaging.com.
Contact:
- Intrinsic Imaging, LLC
- 580 Main St., Suite 210
- Bolton, MA, 01740, USA
- +1 978-616-4650
- [email protected]