Pivotal MRMC Trial of AI Detection Software With 14,000 Radiology Reviews

Challenge

A leading artificial intelligence company needed a pivotal multi-reader multi-case study to support its FDA submission for software that detects pathology on medical images. The study design called for more than 14,000 individual radiology reviews, each reader interpreting every case both with and without AI assistance, within a timeline set by the sponsor's submission plan.

At that scale the risks are operational as much as scientific: recruiting and qualifying enough readers, preventing recall bias between reading sessions, keeping the reading environment compliant with 21 CFR Part 11, and delivering a dataset in which every one of 14,000 interpretations is traceable to a reader, a case, a session and a timestamp. A single inconsistency in session logistics can compromise the statistical model that FDA expects for MRMC analysis.

Intrinsic Imaging’s Solution

Intrinsic Imaging managed all aspects of the trial. The case set was assembled against a reference standard established by an independent truth panel, with case-mix targets for positive and negative cases, pathology subtypes and difficulty, and the reading protocol specified a fully crossed design with randomized case order and a washout period between unaided and aided sessions.

Readers were sourced from the company's network of more than 200 qualified reviewers, screened for board certification, subspecialty and absence of prior exposure to the software, and trained on the reading platform and the case report form before the first session. Reading took place in a validated, Part 11 compliant environment that captured each mark, confidence score and timestamp.

Quality control monitored session completion, case-order compliance and washout intervals in real time, and project managers scheduled readers in parallel cohorts so the full review volume was completed within the sponsor's window. The locked dataset was delivered in the format required for the sponsor's MRMC statistical analysis.

Intrinsic Imaging’s Results

• More than 14,000 reader reviews completed in a fully crossed, randomized MRMC design within the sponsor's submission timeline
• 24 readers sourced, qualified and trained from a 200-plus reviewer network with zero reader dropouts after enrollment
• Every interpretation captured in a Part 11 compliant reading environment with a complete reader, case, session and timestamp audit trail
• Reference standard established by an independent truth panel with documented adjudication of every discordant truth determination