Study finds nearly half of FDA-approved AI medical devices aren't trained on real patient data
Artificial intelligence (AI) technology is evolving rapidly and is already being applied to the medical field. However, some people are skeptical of AI-based medical devices and tools due to concerns about patient privacy and device accuracy. A team from multiple research institutions, including the University of North Carolina School of Medicine, Duke University, Oxford University, Columbia University, and Miami University, published a paper evaluating the extent to which AI and algorithmic technology is approved for use in the medical field.
Not all AI health tools with regulatory authorization are clinically validated | Nature Medicine
Almost half of FDA-approved AI medical devices are not trained on real patient data, research reveals
https://medicalxpress.com/news/2024-08-fda-ai-medical-devices-real.html
According to the research team, the number of AI medical device approvals by the U.S. Food and Drug Administration (FDA) is rapidly increasing, from an average of two approvals per year in 2016 to 69 approvals per year at the time of writing. This indicates that the commercialization of AI medical technology is progressing rapidly.
The research team analyzed all AI medical device applications listed in the FDA's official database, Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices .
As a result, it was found that of the 521 AI medical devices approved by the FDA, 144 had clinical validation data from retrospective validation , 148 from prospective validation , 22 from randomized controlled trials , and 226 had no clinical validation data.
Retrospective validation involves inputting historical image data, such as chest X-rays taken before the COVID-19 pandemic, into an AI model to validate it. Predictive validation, on the other hand, is based on real-time data from patients, which is scientifically stronger. Randomized controlled trials are a way to evaluate the performance of a device by randomly allocating, for example, CT scan data to a human radiologist (the control group) and an AI (the experimental group).
However, the research team noted that approximately 43% of FDA-approved AI medical devices do not publish clinical validation data.
In addition, some devices used computer-generated fictitious images instead of actual patient data, and in some cases did not meet clinical validation requirements. The research team also pointed out that the FDA's guidance published in September 2023 did not clearly distinguish between different types of clinical validation studies in its recommendations to manufacturers.
As a result of their findings, the research team strongly encourages the FDA and device manufacturers to clearly distinguish between different types of clinical validation studies and to make their results publicly available, and they also suggest that definitions of retrospective validation, prospective validation, and randomized controlled trials be standardized for use across the industry.
Related Posts:
