Knowledge Check
In a study evaluating ChatGPT's responses to colon cancer screening questions, how did gastroenterology experts rate the accuracy of ChatGPT's answers on a 6-point scale?
- 3.5
- 4.8
- 5.6
- 0
ChatGPT Answering Questions About Colorectal Cancer Screening
Douglas K. Rex, MD, MASGE, reviewing Maida M, et al. Endoscopy 2024 Aug 14.
European and American experts designed 15 questions on colorectal cancer (CRC) screening, including follow-up of positive tests, bowel preparation, and treatment of CRC, and posed them to ChatGPT4. The answers were rated by 20 gastroenterology experts and 20 non-gastroenterology physicians for accuracy, completeness, and comprehensibility and by 100 Italian patients for completeness, comprehensibility, and trustability.
On a 6-point scale, expert and nonexpert physicians rated accuracy with mean scores of 4.8 and 5.6, respectively (P < .001). On a 3-point scale, experts and nonexperts rated the completeness of answers 2.1 and 2.7 (P < .001), whereas their mean ratings for comprehensibility were similar at 2.7 and 2.8.
Patient scores for completeness, comprehensibility, and trustworthiness ranged from 97% to 100% in all cases.
COMMENT
It looks like ChatGPT generally does a good job answering questions about CRC screening and follow-up, with some reduction in accuracy and completeness when answers are scored by experts.
Note to readers: At the time we reviewed this paper, its publisher noted that it was not in final form and that subsequent changes might be made.
Douglas K. Rex, MD, MASGE
CITATION
Maida M, Ramai D, Mori Y, Dinis-Ribeiro M, Facciorusso A, Hassan C. The role of generative language systems in increasing patient awareness of colon cancer screening. Endoscopy 2024 Aug 14. (Epub ahead of print) (https://doi.org/10.1055/a-2388-6084)
COMMENT
This study affirms important principles for FIT screening. First, cancers detected by FIT remain in an early stage during subsequent screening rounds. Further, increasing the cutoff level of hemoglobin for positive test results reduces sensitivity but does not shift the fraction of detected cancers substantially toward later-stage CRC. Thus, programs can adjust the cutoff to levels appropriate for their colonoscopy resources, and the detected cancers will be in an early stage across a range of hemoglobin thresholds for positive tests.