Quantitative and Qualitative Assessment of an AI Model for Chest Radiograph Report Generation

Purpose To evaluate an AI model for chest radiograph report generation using real-world clinical data through quantitative analysis across diverse findings and qualitative assessment of report quality.
Materials and Methods This single-center retrospective study included chest radiographs collected from a tertiary hospital database under IRB approval, representing real-world clinical imaging data rather than curated public datasets. A set of 500 images was randomly selected to ensure a balanced distribution across diverse findings. Quantitative evaluation compared AI-generated reports with radiologist reports for diverse findings (e.g., pneumothorax, pleural effusion, cardiomegaly, and others), using sensitivity and specificity. Qualitative evaluation was conducted by two board-certified radiologists who independently assessed the AI-generated reports according to the RadPeer scoring system.
Results The model demonstrated variable performance across different findings. Sensitivity and specificity for pleural effusion were 0.835 and 0.906, respectively; for lung opacity, 0.704 and 0.935; and for pneumoperitoneum, 0.667 and 0.998. Subcutaneous emphysema detection achieved 100% sensitivity and specificity. In qualitative evaluation, the acceptance rate (RadPeer scores of 1 or 2) was 94.4% for Evaluator 1 and 89.8% for Evaluator 2, indicating general concordance between AI-generated and expert reports.
Conclusion The chest radiograph report generation AI model, evaluated on real-world clinical data, demonstrated reasonable performance across multiple findings and was generally acceptable to radiologists in preliminary assessments. Further prospective studies are warranted to investigate whether using AI-generated reports as preliminary drafts can improve diagnostic accuracy and workflow efficiency for clinicians.
Clinical Relevance Statement This AI model may support radiologists by providing preliminary chest radiograph reports across a broad spectrum of findings.