MRMC Reader Study for AI 510(k) Submission

Challenge

Intrinsic Imaging supported this study’s imaging activities, addressing the scientific and operational considerations that inform the work of an imaging CRO and imaging core lab. An AI developer preparing a 510(k) submission required a pivotal multi-reader multi-case study to establish whether algorithmic assistance improved reader performance. The study's credibility depended on its reference standard: every case needed a systematically established truth determination supported by documentation suitable for FDA review. The design also had to control recall bias and ordering effects across reading sessions while documenting every step to the standard of a regulatory inspection.

The developer faced practical constraints as well. Cases had to represent the intended-use population across a meaningful range of findings, difficulty and imaging conditions so that conclusions would generalize to clinical practice. Readers had to be recruited and trained without exposure to information that could bias their interpretations, and the reading software had to present aided and unaided cases in a manner faithful to the device's intended use. Each assessment needed to remain traceable to its case, reading condition and reference determination, permitting regulators to evaluate not only the evidence but the procedures through which it was obtained. The reference standard and reading design address different sources of uncertainty: the former defines the comparison target, while the latter governs how assistance is evaluated against that target.

Intrinsic Imaging’s Solution

Intrinsic Imaging collaborated with the developer on the imaging and reader-operation components of the protocol, including case selection criteria, reader qualifications, reading-session structure and documentation requirements. Each case was reviewed for technical adequacy and eligibility before entering the study, and the case set was locked before reading began to preserve its independence.

A consensus reference-standard panel of fellowship-trained subspecialists established truth for each case. Panelists recorded independent determinations before convening to resolve disagreements, and their initial determinations were preserved alongside the final consensus, so that reviewers could see not only the reference standard but how and why each disputed case had been resolved. Reading sessions were randomized and counterbalanced across readers with defined washout periods between aided and unaided interpretations.

Readers completed standardized training on the reading software and scoring definitions using cases outside the study set. Throughout the reading periods, our project team monitored session completion, enforced washout intervals and documented every deviation, so that the final dataset reflected the protocol as written. The entire study was conducted within our 21 CFR Part 11-compliant reading environment, producing a complete, audit-ready record of every assessment, annotation and consensus decision, with the case set, reading sequence and reference standard each traceable to its documented origin.

Intrinsic Imaging’s Results

• 510(k) clearance received on the strength of a reader study whose reference standard, reading design and documentation were constructed to regulatory expectations from the outset
• Zero FDA questions on the reader study design, reflecting prospective control of recall bias, ordering effects and reference-standard independence rather than correction during analysis
• Audit-ready reference-standard datasets delivered, with each panelist's independent determination preserved alongside the final consensus decision for every case in the locked set
• Documented methodology retained by the developer as the template for subsequent product updates and reader studies throughout the device's ongoing regulatory lifecycle