Spin-off use of adverse events data: why and how. The case of FDA’s MAUDE

Objectives: This paper attempts to measure the impact of the second stage exploitation of FDA’s MAUDE database on patient safety, technology assessment and other scientific fields. Methods: Five bibliographic databases have been queried with the terms “Manufacturer and User Facility Device Experience Database” and “FDA AND MAUDE”. A number of eligibility criteria where applied on the results, which led to a final group of 117 papers. An extensive study of these publications resulted to a number of interesting findings. Results: The results concern the evolution of the database exploitation over time, and are examined according to the device groups that the identified papers are referring to, the research goals of these papers, the reasons that led the authors of these papers to use MAUDE data and finally how these data were used within their research methodology. Conclusions: Patient safety and technology assessment are two of the scientific fields on which MAUDE database has the greatest impact. On average, more than 10 peer-reviewed papers each year involve MAUDE data as a mean to reach their research goals. This proves that MAUDE is an exploitable and valuable data source for research in these scientific fields.


INTRODUCTION
Patient safety, health technology assessment and medical device vigilance are fields that heavily rely on data availability. They need valid data from various sources in order to extract useful information. A significant source of data for medical devices (MDs) appears to come from the medical devices vigilance and post-market surveillance mechanisms that are imposed by the relevant regulatory systems, in most part of the world. The Food and Drug Administration (FDA) Manufacturer and User Facility Device Experience (MAUDE) database is such a source.
Millions of medical devices are used today in various places (hospitals, clinics, houses, etc.) and thousands of new models enter the market every year. Undoubtedly these MDs have a significant contribution to the improvement of the healthcare services provided. However, medical technology, like any other technology, is not risk free. There are numerous cases where devices have been recalled because of their involvement in adverse incidents compromising patients' health or cases where a "promising" innovative approach has to be withdrawn after a relatively short period of use, because it is not proven as safe as expected. 1,2,3,4 The largest MDs markets today (USA, EU, Japan, etc.) are ruled by regulatory frameworks (Regulations, laws, directives, guidelines) according to which a medical device has to comply with specific safety provisions in order to enter these markets 5,6 . One safety requirement, common to all these frameworks, is the adverse event reporting system or vigilance system that follows the medium and high risk devices, after they have entered the market, in parallel with the post-market surveillance. 5,6,7 A MDs vigilance reporting system aims to increase patient safety by preventing the recurrence of reported adverse events. This is achieved by mandating users and manufacturers of medical devices to report to the health authorities, incidents where a medical device contributes to an adverse event. According to this mechanism, whenever a medical device is potentially contibuted in a death or injury of a patient or user, the manufacturer has to report this event and assess if corrective actions should be taken. In parallel, a user reporting system encourages users to report to the manufacture and/or to authorities any such incident that comes to their attention. The principal purpose of the medical devices vigilance and user reporting systems is to improve the safety of patients, users and others, by reducing the likelihood of reoccurrence of a similar event elsewhere in the future. This is to be achieved by the evaluation of reported incidents and, where appropriate, dissemination of information, which could be used to prevent such repetitions, or to alleviate the consequences of such incidents.
USA is the biggest medical device market 8,9 and FDA, as the relevant organization for market surveillance, is also responsible for medical devices vigilance. FDA has implemented since the 1990s a database called Manufacturer and User Facility Device Experience Database 10 for reporting of medical devices related with adverse events. Nowadays, this database receives more than 800.000 reports annually. 11 Today, MAUDE contains more than 4 million medical device reports (MDRs) 11 of suspected device-associated deaths, serious injuries and malfunctions as well as other non-conformities such as packaging and labelling problems, unsterilized delivery etc. MAUDE contains MDRs filed by manufacturers and importers from August 1996 to present, all mandatory user facility reports from 1991 to present and voluntary reports filed after June 1993 12 . A portion of the database is open to the public, providing valuable information on MDs safety. It is accessible through the FDA's website (www.fda.gov) and can be queried through a search form. In addition, all main datasets are provided to the public as text files importable to common databases.
After an evaluation process, the high volume MDRS or the ones involved with a death are investigated by FDA and in many cases this investigation leads to corrective actions, with obvious benefits for the safety of both patients and users. At a second stage, this huge amount of data appears to be a valuable source for further research. Retrospective analysis studies, data extraction techniques and other scientific use of these data, offer spin-off benefits to patient safety, medical device technology assessment and other scientific fields 13, 14. This study attempts to measure quantitatively and qualitatively the second stage exploitation of MAUDE and the impact of this exploitation on scientific research.

MATERIALS AND METHODS
Five international bibliographic databases (Science-Direct, Journals@Ovid Full Text, Pubmed, Web of Science and Scopus) have been queried with the terms "Manufacturer and User Facility Device Experience Database" and "FDA AND MAUDE" in order to find all publications that contain these terms. The databases were queried in January 2015. The results of these queries were consolidated through the removal of duplicates, which led to an initial number of 1.016 unique publications. (The results from each database appear in Table 1).
This set of results was filtered according to the publication type, language and publication time so as to keep only peer-reviewed papers, written in English, from 2005 to 2014. Books, editorials, commentaries, letters, comments on a paper, publications in conference proceedings, etc., were excluded from the final selection. After filtration, 381 scientific papers remained.
The next step was to extract out of these 381 papers the ones that have used directly data from MAUDE. This selection led to a final group of 117 papers. Among these papers, there were 4 where the authors searched the MAUDE database but the results were found to be irrelevant to this work. However, these 4 papers were decided to be included in the final group because, although they finally did not use any data from MAUDE, they took into consideration the content of the database. The list of 117 papers appears in the Reference section (Ref: 14-25, 28, 31, 33-135).
It should be mentioned that among the 264 excluded papers, more than 50 referred to MAUDE data, but this reference was either limited to a single comment about one or two cases or indirect, using the results of other papers that had used the original data.
The last step was to study again in more detail the final group of these 117 papers, focusing on the device groups that these papers referred to, the evolution of the database exploitation with time, the research goals of these papers, the reasons that led the authors of these papers to use MAUDE data, how they finally used these data within their research methodology, etc. The flowchart for the query methodology is shown in Figure 1.
The bibliographic search results were processed initially with the Mendeley desktop references management software and later with Microsoft Excel.

RESULTS
The analysis of the final set of 117 papers revealed the following: A) Since MAUDE is a database containing MDRs, each record is related with a medical device. Hence retrieval of data for the second stage usage is also related with medical devices. The analysis carried out identified the device groups that were used as a reference in the papers. These device groups were grouped, where applicable, into more generic device categories. It should be mentioned that although 24 papers were focused exclusively on a device type rather than on a group as a whole, only the device group was considered for the purposes of this analysis.  According to the analysis performed, the general category of Cardiology Devices was the most frequently referred (29 papers), while Implantable Devices (22 papers) and Endoscopy Devices (14 papers) were the next ones. As regards the device groups, Stents (9 papers), Implantable Cardioverter-Defibrillators (7 papers), Meshes (6 papers), Vena Cava Filters (6 papers), Septal Occluders (4 papers), and Cochlear Implants (4 papers) were the leaders. Finally, there were 8 papers that have not been included in this part of the analysis, since they used a trend line shows that the number of papers that used the MAUDE data increases with time (slope =0.6). (Figure 2) C) Although it was difficult and maybe risky to summarize and classify the research objectives of papers covering various scientific areas and subjects, into a few generic objectives' categories, such an attempt was made in order to outline the research orientation of the papers that use data from the MAUDE database.
The most common objectives among these papers were "to review/identify the reported adverse events/complications related with a device group or type" (31 papers), "to evaluate adverse events" (22), "to evaluate design characteristics of a device group or type" (15 papers), "to explain why these events occur" (14 papers), and "to overview a medical technology and/or its performance" (10 papers). Table 3 presents the results of this analysis.  To evaluate adverse events/complications 22 To raise awareness of the potential hazard of auricular burns associated with operating microscope use during otologic surgery 16 To examine/evaluate design characteristics of a medical device 15 We sought to determine if perforation rates are related to cannula design 17 To explain why specific adverse events/ complications occur 14 This study was undertaken to analyze bleeding problems with tension-free vaginal tape (TVT) operations 18 To overview a medical technology and/or its performance 10 This study sought to determine whether infusion device event logs could support accident investigation 19 To assess the frequency and/or severity of adverse events/complications 8 The purpose of this study is to use large databases to assess the frequency and severity of such complications and compare them with those of surgical atrial septal defect closure 20 To evaluate/test a method or a hypothesis 7 The aim of this article is to evaluate a new system and procedure, dedicated to oxygenator change-out 21 To review a new technology/procedure 7 This document will review the biliary and pancreatic stone extraction devices that are currently commercially available in the United States 22 To discuss regulatory issues 2 The present analysis aimed to compare the 510(k) and PMA approvals and recalls on the basis of the number of devices approved in each group 23 To estimate cost 1 To estimate the rates and costs of intravenous patient-controlled analgesia (IV PCA) errors from the hospital or integrated health system perspective 24 D) Equally difficult was the attempt to examine and classify the purpose for which the MAUDE data were used within those papers. The findings of this analysis were similar with the findings of the analysis of the papers' objectives. In brief, the main reasons for the use of MAUDE data was "to summarize or review adverse events" (53 papers) as well as "to explain why these events occur" (42 papers). Additionally, it was found that 36 papers dealt with the evaluation of adverse events or complications, and 32 papers provided directly suggestions for patient safety measures. Finally, it should be noted that in each paper these data could have been used for more than one purpose. All the findings of this analysis are presented in Table 4.

DISCUSSION
The final number of 117 papers that were found to have used MAUDE data cannot be considered as covering the whole spectrum of the respective research activities. The actual range of MAUDE data usage must be considered even greater if it is taken into account that among the publications that were excluded by the present study, there were many papers that a) refer to a unique case from MAUDE, b) use partially or complementarily data from it or c) refer to other papers based on MAUDE data analysis. In addition, it was also found that there were many other kinds of publications, such as books, editorials, and publications in conference proceedings, which were used on MAUDE data.
The device groups that the papers focused on were mainly cardiology devices (Stents and Implantable Cardioverter-Defibrillator), implantable devices (Meshes and Cochlear implants), endoscopy and laparoscopy devices. It is surprising that high risk device groups that are used widely in hospitals, such as Respirators, Anesthesia Machines, ECG, etc., were not among the devices of this list. One possible reason for this fact is that the researchers have directed their attention to devices that had entered the market within or near the period under examination (drug eluted stents, robot assisted surgery, transcervical sterilization, etc.) or to device groups containing products which have been involved in serious recalls (Stents, Occluders, Cardioverter, Defibrillators etc.) 1 .
The number of papers that use data from MAUDE appears to increase with time, having a time trend with a rate of 0.6 ( Figure 1). It is expected that in the near future the second stage exploitation of MAUDE data will further increase given that FDA makes a constant effort to improve the quality of data and their accessibility (Unique Device Identifier, Total Product Life Cycle, Open FDA etc.) 2, 3, 11 25, 26 in combination with the fact that new or improved management and analysis techniques of big data emerge.

TABLE 4. Purposes of MAUDE data use
The MAUDE data have been used:

Number of Papers
To summarize the adverse events related with a device group or type 53 To explain why specific adverse events occur 42 To evaluate adverse events or complications 36 To suggest patient safety measures 32 To assess a device group or type 30 To estimate how frequent is the occurrence of an adverse event or to calculate trends 30 To assess the safety of a technology or of a medical procedure 24 To assess the severity of adverse events/complications 15 The examination of the papers' research objectives clearly shows that the majority of the papers under consideration contributes directly or indirectly to patient safety by reviewing or summarizing the adverse events/ complications related with a specific device group or type (31 papers), by evaluating adverse events/complications (22 papers), by explaining why these events occur (15 papers) or by assessing the frequency or severity of adverse events (8 papers). Additionally, the contribution to technology assessment is also significant through the evaluation of the devices' design characteristics (14 papers), the overview of a medical technology and its performance (10 papers) and the review of new technology and/or medical procedures (7 papers). Finally, the papers in question have a contribution in other fields too. For example, 7 papers used MAUDE data in order to test or evaluate a method and 2 papers discussed the regulatory issues for medical devices.
The fact that the MAUDE database is a useful source for patient safety purposes is further supported by the examination of the manner in which these data are used in the papers. It was found that MAUDE data have been used among others to summarize the adverse events related to a device or a medical procedure, to explain why adverse events occur and to suggest specific measures. The ultimate goal of the above-mentioned uses was to inform the medical community as well as MD designers and manufactures about the problems that could arise, the likelihood for them to occur, the underlying mechanisms that lead to these complications, the ways to avoid or to deal with these events and the measures to eliminate their consequences. Besides, MAUDE appears to be a useful tool as regards technology assessment too, since its data have been used in order to assess the use of medical technologies and medical devices, as well as to estimate the risk of the utilization of a device or procedure. It is also worth mentioning that from this analysis, it was found that 14 of the papers used the MAUDE database as a source in order to test or evaluate a method, a procedure or a hypothesis. For example, MAUDE data were used to evaluate the role of human factors in acute care equipment decisions 27 and to examine whether the log files could assist in an accident investigation 19 .
During the papers' analysis, other useful information was also gathered, pertaining to research limitations inserted by the use of MAUDE data as well as to the quality and integrity of these data. In many papers it is mentioned that the MAUDE data and the use of adverse event reports data in general, inserted certain limitations dealing with the reporting rate and the denominator issue 28 . As regards the reporting rate, there is a general belief that not only adverse events are under-reported but there is also a lack of information about the ratio representing the number of adverse events reported versus the number of real events that have occurred. Similarly, there is a lack of baseline numbers (e.g. total number of surgical procedures relevant to a product, total number of specific devices used, etc.) that could be used as denominators. Both these limitations make the data unsuitable for determining rates 29,30 .
Moreover, there was criticism as regards the consistence and quality of the MAUDE data. Some researchers have doubts about their quality, stating that the data provided by FDA are not structured in a common way, are not complete and their accuracy is debatable, thus obstructing the analysis procedures. Others commented that the information and degree of detail contained within these reports are highly variable, making interpretation of the reports difficult and causality often uncertain 29,31 .
During the period 2005-2014, MAUDE data could be searched either by an online search form provided by the FDAs' web site or by downloading them in txt formatted files. The majority of the studies have used the online search form. There were only a few that have used the MAUDE data provided in txt format. This is probably because the insertion of these txt files into a relational database is not an easy task given the amount of data (some tables have more than 3 million rows) and because the txt files need some technical preparatory actions in order to be ready for insertion. It is expected that the openFDA web site (https://open.fda.gov/) which provides capabilities for easier and more comprehensive access to the data in addition with the further use of the database with contemporary big data analysis tools or data mining techniques will lead to a more intense exploitation of MAUDE database.
Finally, it is worth mentioning the positive impact of the transparency of MAUDE database comparing it with the European Databank on Medical Devices (EUDAMED). In the EU, legislative changes imposed stricter and more detailed monitoring and enforcement requirements for both notified bodies and national competent authorities, in response to increasing safety concerns. Recently, the enforcement of a more rigorous new legislation in the form of two Regulations 136,137 has been voted by the European Parliament. The use of the European Databank on Medical Devices (EUDAMED), containing regulatory information on MDs available on the EU market, including recalls, is also reinforced. However, regarding the EU user reporting system for medical devices adverse events, there is not an overall collection of the reports submitted to the national competent authorities. This is due to the decentralised structure of the EU regulatory system, in combination with the fact that there is no provision for a centralised collection into the EUDAMED. Additionally, the EU policy that does not allow the public access to all these data, including the recalls, prohibits their analysis by independent researchers.
A research comparing the impact of the transparency of EU vigilance system with the one of FDA for the period 2004-2015, found that there are no papers or reports, even from a central EU body, based on the EUDAMED data 138 . However it is a fact that EUDAMED can provide similar information. As an example, Bliznakov et.al 139 performed a survey on medical device recalls, concerning only devices using software, based on FDA data for the period 1995-2002. It was found that about 25% of the recalls studied, were caused by software failures. As might be expected, the proportion of these recalls due to software problems increased, from 17% in 1995 to 34% in 2002. Follow up studies 140,141 revealed that this proportion went up to 40% in year 2012. These authors, performed in parallel a survey on recalls caused by software failures using EUDAMED data, and found very similar results. Unfortunately, those results could not be published due to the restrictions on the use of EUDAMED data.

CONCLUSIONS
FDA provides public access to a portion of its postmarket surveillance database, thus allowing researchers outside FDA to carry out analyses and studies based on the raw data, with a consequent spin-off benefit for public health. The fact that, in spite of the limitations, more than 10 peer-reviewed papers each year use MAUDE data shows that MAUDE is an exploitable and valuable data source.
According to the analysis of the papers, MAUDE database is used mainly for research works related to patient safety and technology assessment compared to other scientific areas. It is also observed that the MAUDE data are mainly used to evaluate devices that are relatively new to the market, or to investigate issues related with these devices. Additionally, it was found that MAUDE is a useful data source when it is required to summarize adverse events related with a device as well as when the reasons that could lead to an adverse event have to be examined. Finally, MAUDE data exploitation increases with time and is expected to be even more intensive in the future.
Undoubtedly, there are improvements that could increase the exploitation of MAUDE database. However, despite limitations, restrictions and criticism, it is a common conclusion among the majority of the papers studied, that the MAUDE database is a useful and valuable tool for patient safety and technology assessment. The benefits resulting from the MAUDE use should be taken into consideration by the EU, so as to move in the direction of enhancing and improving the data collection procedures from the vigilance system as well as to increase the transparency of EUDAMED as explicitly stated in the regulations 136, 137 : "..vigilance and market surveillance should be significantly reinforced, whilst provisions ensuring transparency and traceability regarding medical devices should be introduced, to improve health and safety … The objectives of the database are to enhance overall transparency, including through better access to information for the public and healthcare professionals …". Additionally individual researchers should be allowed to have access to relevant data, in order to be able to perform similar studies that significantly contribute to equipment improvement and patient safety.