Pivotal Device Trial of AI Software for Screening Clinical Images

Challenge

An emerging AI company required a pivotal medical device trial to assess the clinical performance of software that screens clinical images for findings requiring attention. Screening populations have low disease prevalence, so a case set drawn at random would contain too few positives to estimate sensitivity with any precision, while an over-enriched set would misrepresent the specificity readers achieve in practice.

The sponsor also needed evidence acceptable to both FDA and European regulators, which meant standalone algorithm performance and reader performance with and without the software had to be measured in the same study, under a design and documentation standard suitable for two submissions.

Intrinsic Imaging’s Solution

Intrinsic Imaging designed the case set with a prevalence-enrichment plan that was documented and justified in the protocol, drawing positive and negative cases from multiple institutions to represent scanner, demographic and acquisition variability. The reference standard combined pathology, follow-up imaging and expert panel consensus according to a predefined hierarchy.

Standalone performance was assessed by running the software on the locked case set and comparing output to the reference standard. Reader performance was assessed in a fully crossed MRMC design, with readers sourced from the company's 200-plus reviewer network and trained to use the software output as a second read, as intended in the device labeling.

The reading environment captured per-case decisions, confidence scores and reading times, and the datasets for standalone and MRMC analyses were delivered together with the full reference-standard documentation for inclusion in both regulatory submissions.

Intrinsic Imaging’s Results

• Enriched case set of approximately 1,200 cases from 6 institutions assembled against a hierarchical reference standard
• Standalone and MRMC reader performance measured in a single study, providing one evidence package for FDA and European submissions
• 18 readers completed roughly 8,600 reviews with and without AI assistance in a fully crossed randomized design
• Reading-time data captured per case, allowing the sponsor to characterize workflow impact alongside diagnostic accuracy