Chest CT Lung Nodule MRMC Reader Study

Challenge

Intrinsic Imaging supported this study’s imaging activities, addressing the scientific and operational considerations that inform the work of an imaging CRO and imaging core lab. Evaluating the effect of a lung nodule detection tool on reader performance required a multi-reader multi-case study comparing interpretation of low-dose chest CT with and without algorithmic assistance. The design demanded readers spanning a range of experience levels to support generalizable conclusions, together with a reading workflow that would prevent recall of prior interpretations from contaminating the comparison between aided and unaided sessions. Every element had to satisfy the evidentiary expectations of a 510(k) submission.

Case composition and reading conditions required equally careful control. Cases had to include a range of nodule sizes, types and locations alongside negative examinations, so that performance could be evaluated across clinically relevant scenarios; the software had to present aided and unaided cases consistently; and readers needed sufficient training to use the tool without prior exposure to study cases. Coordinating eighteen readers across two reading periods under a fixed submission schedule meant that scheduling, documented training and washout management had to be managed so that incomplete sessions or procedural deviations could not compromise interpretation of the comparison.

Intrinsic Imaging’s Solution

Intrinsic Imaging collaborated with the developer to finalize the case set, reader qualifications, reading sequence and documentation requirements. Cases were reviewed for eligibility and technical adequacy before the dataset was locked, protecting the integrity of the study design from the outset.

We recruited and trained a reader group of 18 radiologists spanning the required experience spectrum. Readers completed standardized training on the software and scoring conventions using non-study cases, then read under randomized sequences separated by a four-week washout period to reduce the potential for recall bias between sessions. Throughout both reading periods our project team monitored completion, enforced the washout interval and documented every deviation, so that the final dataset reflected the protocol as written and the operational record could be examined alongside the results.

A reference-standard panel of thoracic radiologists established truth for every case under a documented process in which panelists first recorded independent determinations before resolving disagreements. Nodule locations, characteristics and final truth decisions were captured in detail, allowing reader marks to be matched reliably against the reference standard during analysis. All reading activity took place within a controlled, fully documented environment suitable for regulatory review, giving the developer not only a defensible comparison of aided and unaided performance but a methodology it could apply to future product updates and reader studies.

Intrinsic Imaging’s Results

• 10 weeks to complete all reader assessments across 18 radiologists of varying experience and two randomized reading periods separated by a four-week washout interval
• Improved reader sensitivity demonstrated with algorithmic assistance for lung nodule detection, under counterbalanced reading conditions designed prospectively to control recall bias and ordering effects
• Study data accepted in the developer's 510(k) submission, supported by an independently established thoracic reference standard and complete operational records for every session
• Every procedural deviation documented during both reading periods, giving the developer a defensible record and a methodology reusable for future product updates