Intrinsic Imaging
Intrinsic Imaging

Pivotal Device Trial of AI Software for Screening Clinical Images

Case Study · AI/CAD

  • Standalone and MRMC performance
  • Enriched screening case set
  • Sensitivity, specificity and reader AUC
Download the PDF

Challenge

Intrinsic Imaging supported this study’s imaging activities, addressing the scientific and operational considerations that inform the work of an imaging CRO and imaging core lab. An emerging AI company required a pivotal medical device trial to assess the clinical performance of software that screens clinical images for findings requiring attention. Screening populations have low disease prevalence, so a case set drawn at random would contain too few positives to estimate sensitivity with any precision, while an over-enriched set would misrepresent the specificity readers achieve in practice.

The sponsor also needed evidence acceptable to both FDA and European regulators, which meant standalone algorithm performance and reader performance with and without the software had to be measured in the same study, under a design and documentation standard suitable for two submissions. Enrichment creates a necessary distinction between estimating discrimination in a selected study population and predicting performance in routine screening. The apparent burden of positive findings can affect reader expectations, while predictive values depend on disease prevalence and cannot be transferred directly from an enriched sample to everyday clinical practice.

Intrinsic Imaging’s Solution

Intrinsic Imaging designed the case set with a prevalence-enrichment plan that was documented and justified in the protocol, drawing positive and negative cases from multiple institutions to represent scanner, demographic and acquisition variability. The reference standard combined pathology, follow-up imaging and expert panel consensus according to a predefined hierarchy. A hierarchical reference standard clarifies which evidence governs classification when several sources are available. Nevertheless, pathology, longitudinal imaging and expert consensus do not provide identical forms of verification, so interpretation should consider whether positive and negative cases were supported by different kinds of evidence within the documented hierarchy.

Standalone performance was assessed by running the software on the locked case set and comparing output to the reference standard. Reader performance was assessed in a fully crossed MRMC design, with readers sourced from the company's 200-plus reviewer network and trained to use the software output as a second read, as intended in the device labeling. Standalone assessment addresses the algorithm's output in isolation, whereas assisted reading addresses the interaction between that output and clinical judgment. These are related but distinct questions: accurate standalone classification does not by itself establish that readers use the information effectively or that their diagnostic performance improves.

The reading environment captured per-case decisions, confidence scores and reading times, and the datasets for standalone and MRMC analyses were delivered together with the full reference-standard documentation for inclusion in both regulatory submissions. Confidence and reading time help describe how readers interact with assistance alongside their final decisions. Session measurements remain specific to the study environment, however, and should not be interpreted as direct estimates of clinical throughput, downstream workload or patient benefit without evidence addressing those broader settings.

Intrinsic Imaging’s Results

  • Enriched case set of approximately 1,200 cases from 6 institutions assembled against a hierarchical reference standard
  • Standalone and MRMC reader performance measured in a single study, providing one evidence package for FDA and European submissions
  • 18 readers completed roughly 8,600 reviews with and without AI assistance in a fully crossed randomized design
  • Reading-time data captured per case, allowing the sponsor to characterize workflow impact alongside diagnostic accuracy
View all case studies

About Intrinsic Imaging, LLC

Intrinsic Imaging is an ISO certified medical imaging CRO and imaging core lab providing comprehensive services in support of Phase I-IV clinical trials and Class I, II, III medical device trials. Intrinsic Imaging is also a world leader in conducting Artificial Intelligence studies.

With its team of more than 650 board-certified and fellowship-trained physicians, supporting over 4000 imaging centers in 64 countries, Intrinsic Imaging’s team has sub-specialization in all therapeutic areas, including Cardiovascular & Thoracic, Central Nervous System, Gastroenterology & Metabolic, Musculoskeletal, Oncology and Ophthalmology.

Certified by the British Standards Institution (BSI), Intrinsic Imaging is the only company in the world to hold five ISO certifications specifically for providing imaging core lab services on clinical trials. For more information, please visit IntrinsicImaging.com.

Contact: