Journal of Postgraduate Medicine
 Open access journal indexed with Index Medicus & ISI's SCI  
Users online: 4270  
Home | Subscribe | Feedback | Login 
About Latest Articles Back-Issues Articlesmenu-bullet Search Instructions Online Submission Subscribe Etcetera Contact
 
  NAVIGATE Here 
  Search
 
  
 RESOURCE Links
 ::  Similar in PUBMED
 ::  Search Pubmed for
 ::  Search in Google Scholar for
 ::Related articles
 ::  Article in PDF (601 KB)
 ::  Citation Manager
 ::  Access Statistics
 ::  Reader Comments
 ::  Email Alert *
 ::  Add to My List *
* Registration required (free) 

  IN THIS Article
 ::  Abstract
  ::  Background
Materials and Me...
  ::  Results
  ::  Discussion
  ::  Conclusion
 ::  References
 ::  Article Figures
 ::  Article Tables

 Article Access Statistics
    Viewed1333    
    Printed63    
    Emailed0    
    PDF Downloaded21    
    Comments [Add]    

Recommend this journal


 


 
  Table of Contents     
ORIGINAL ARTICLE
Year : 2022  |  Volume : 68  |  Issue : 3  |  Page : 133-137

An audit of safety reporting in randomized controlled trials over a five-year period in a high impact factor journal


1 Department of Clinical Pharmacology, Seth GS Medical College and KEM Hospital, Mumbai, Maharashtra, India
2 Department of Pharmacology and Therapeutics, Seth GS Medical College and KEM Hospital, Mumbai, Maharashtra, India

Date of Submission25-Jan-2021
Date of Decision10-Apr-2021
Date of Acceptance07-May-2021
Date of Web Publication03-Sep-2021

Correspondence Address:
N J Gogtay
Department of Clinical Pharmacology, Seth GS Medical College and KEM Hospital, Mumbai, Maharashtra
India
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/jpgm.JPGM_78_21

Rights and Permissions


 :: Abstract 


Background: Randomized controlled trials [RCTs] form the corner-stone of evidence-based medicine. RCTs published in high impact factor journals such as the New England Journal of Medicine [NEJM] are a key driver of clinical practice and policy decisions. RCTs are expected to report both efficacy and safety, however, safety reporting in many studies tends to be poor. The present audit was undertaken with the primary objective of evaluating safety reporting during a five-year period in all RCTs published in the NEJM.
Methods: PubMed alone was searched for RCTs published in NEJM from 2013-17. Each RCT was searched for the following outcome measures –whether the trial was sponsored by pharmaceutical industry or investigator initiated, phase of trial, nature of intervention and therapeutic area in terms of reporting of safety outcomes [with 'P values' or '95% confidence interval'].
Results: A total of n=623 articles reported safety outcomes of which 275/623 (44.1%) articles reported statistics for safety outcome. There was significant difference in reporting of safety statistics between investigator initiated studies and pharmaceutical industry sponsored studies, [cOR=4.0, 95% CI 2.8- 5.5 P < 0.001]; phase 3 and phase 4 trials, [cOR 0.67, 95% CI 0.5 - 0.9, P = 0.02]; trials involving drugs and surgery, [ cOR 2.07, 95% CI 1.2-3.5, P = 0.01] and in therapeutic areas, cardiovascular and oncology [cOR 0.26, 95% CI 0.1-0.4, P < 0.0001].
Conclusions: Safety reporting in RCTs continues to take a back seat relative to efficacy reporting and is worse for pharmaceutical industry funded studies. Safety reporting should be emphasized in the CONSORT guidelines.


Keywords: Investigator-initiated trial, P-value, randomized controlled trial, safety outcome


How to cite this article:
Konwar M, Mamde A, Patankar P, Thatte U M, Gogtay N J. An audit of safety reporting in randomized controlled trials over a five-year period in a high impact factor journal. J Postgrad Med 2022;68:133-7

How to cite this URL:
Konwar M, Mamde A, Patankar P, Thatte U M, Gogtay N J. An audit of safety reporting in randomized controlled trials over a five-year period in a high impact factor journal. J Postgrad Med [serial online] 2022 [cited 2022 Sep 30];68:133-7. Available from: https://www.jpgmonline.com/text.asp?2022/68/3/133/325749





 :: Background Top


Well-designed randomized controlled trials (RCTs) are the gold standard for evaluating the efficacy and safety of a new treatment and form the corner-stone of evidence-based medicine.[1] RCTs published in high impact factor journals such as the New England Journal of Medicine (NEJM) are a key driver of clinical practice and policy decisions. Reporting of both efficacy and safety in these RCTs is equally important as benefit-risk assessment can be done only when both are present in the manuscript.[2]

Evidence shows that safety reporting often takes a backseat relative to the reporting of efficacy. Most studies in literature have only analyzed the safety reporting in terms of adequcacy of reporting of specific adverse effects and their severeity.[3],[4],[5] None of the studies have looked at specifics such as therapeutic area, funding support and type of intervention and phases of trials with regards to safety statistics.

The extension of the consolidated standards of reporting trials (CONSORT) statement for harm (2004) gives specific guidelines for the reporting of the harm related results of the clinical trials to address the shortcomings in analysis and reporting of harms. The idea is that the readers can assess benefit-harm ratio in the application of medical interventions.[6] to make informed decisions while treating individual patients.

Against this backdrop, the present study was undertaken with the primary objective of evaluating safety reporting (relative to efficacy reporting) of all RCTs published in the NEJM over a five-year period. We chose the NEJM for analysis as it multidisciplinary, weekly and with a large worldwide readership and an impact factor of 74.69 in 2019.[7]


 :: Materials and Methods Top


Ethics: The study protocol EC/OA 140/2018 was granted exemption from review on 5th Jan 2018 by the Institutional Ethics Committee (IEC).

Search strategy and eligibility criteria: All articles published in the NEJM between 2013 and 2017 were electronically searched and extracted from the PubMed database using the keywords 'N Engl J Med' and Clinical trial. Inclusions were RCTs whereas observational studies, follow-up RCTs, review articles, case series and case reports were excluded.

Two authors MK and AM assessed the articles independently to confirm that they were truly RCTs. In addition, a hand search was carried out for each article by the same authors independently. Any discrepancy was resolved by consensus or by referral to senior authors (NG and UT).

Data extraction: A standardized data collection form was used to extract the following information from the studies- reporting of safety outcomes (with corresponding P values and 95% confidence intervals) both in the abstract and full-text paper, the type of sponsor (whether investigator-initiated or funded by the pharmaceutical industry), the phase of the RCT, nature of the intervention and therapeutic area. Additionally, data was also collected from regulatory sites of the FDA and EMA and clinical trial registries (clinicaltrials.gov and iscrtn.com) if not found in the published paper. A formal email was sent to the corresponding author where more information was needed or if it was missing.

Outcome measures

These were a) Proportion of RCTs reporting statistics for safety outcomes in the full-text article and/or in their abstract b) Nature/source of funding {Investigator-initiated or Pharmaceutical industry sponsored studies} c) The phase of the study (whether I, II, III, IV, I/II or II/III) d) Nature of the intervention e) Its therapeutic area f) Presence [or lack there of] of statistical parameters for safety and g) The association of safety information with i) Nature of funding ii) Phase of the study iii) Nature of Intervention and iv) Therapeutic area of the intervention.

Statistical analysis

Both descriptive and inferential statistics were applied to the data. Categorical data (for example therapeutic area) were expressed as proportions and quantitative data (total RCTs) as median (range). Normality of quantitative data was assessed using the Kolmogorov Smirnov test. Association of safety with respect to nature of funding, therapeutic area, with safety statistics (present or absent] was done at a significance level of P < 0.05 using the Chi-square test and a crude odds ratio (with 95% CI) was calculated. All statistical tests were done using PRISM version 7.


 :: Results Top


Search findings between 1st January 2013 to 31st December 2017, a total of n = 995 articles were identified in the PubMed search. Of these, n = 239 abstracts were excluded as they did not meet the eligibility criteria allowing for 756 full-text articles for review. Of these 110/756 met the exclusion criteria and thus n = 646/756 (85.4%) RCTs were included for the analysis. [Figure 1]. Of these, in 23/646 RCTs, safety outcomes were not applicable because of the nature of the RCTs, for example-simulation-based trial[8] and thus n = 623/756 (82.4%) RCTs formed the final sample.
Figure 1: Flow chart showing selection process of the studies

Click here to view


Reporting of safety outcomes

A total of 275/623 (44.1%) articles reported statistics for safety outcome in term of 'P value' (269/623, 43.2%) and/or 95% CI (6/623, 01%). Safety statistics were reported in 117/275 (42.5%) articles both in the abstract and the full text, while 126/275 (45.8%) articles reported only in the full text, and 32/275 (11.6%) articles reported only in the supplement.

Type of RCTs

A total of n = 299/623 (48%) RCTs were investigator initiated studies (IIS) and n = 324/623, (52%) were pharmaceutical industry sponsored studies (PISS). Most of the RCTs were in phase 3 (368/623, 57.8%), followed by phase 2 (60/623, 9.6%) and phase 4 (48/623, 7.7%). However, in 121/623 (19.4%) RCTs, the phase of the trial was not mentioned in the article or in the clinical trial registries. RCTs associated with drugs as a mode of intervention (n = 403/623, 64.7%) was highest followed by surgery (n = 63/623, 10.1%), medical devices (n = 23/623,3.7%) and vaccines (n = 22/623,3.5%). The largest numbers of the RCTs was in the therapeutic area of cardiovascular system (n = 172/623, 27.6%) which was followed by oncology (n = 101/623, 16.2%) and infectious disease (n = 94/623, 15.1%) [Table 1].
Table 1: Demographics of randomized controlled trials

Click here to view


Subgroup analysis of different phase of trials, interventions and therapeutic areas in respect to the RCTs being an IIS or PISS is shown in [Table 2].
Table 2: Subgroup analysis of randomized controlled trials

Click here to view


Association of presence or absence of statistics for safety with respect to:

  1. Investigator-initiated [IIS] or pharmaceutical industry sponsored studies [PISS]: There was higher reporting of safety statistics in IIS relative to PISS, (n = 183/299,61.2) versus (n = 92/324, 28.4%) [cOR = 4.0, 95% CI 2.8-5.5], P < 0.001.
  2. Phase of trials: There was lower reporting of safety statistics in phase 3 trials (n = 148/368, 40.2%) [cOR 0.67, 95% CI 0.5-0.9], P = 0.02; whereas, more safety statistics were reported in phase 4 trials (n = 33/48, 68.7%) [cOR 3.03 95% CI 1.6-5.9], P < 0.01, as compared to other phases of the clinical trials.
  3. Mode of intervention: RCTs involving drugs had poorer reporting of safety statistics (n = 150/403,37%) [cOR 0.45, 95% CI 0.3-0.6], P < 0.01 whereas, surgery as a mode of intervention had more reporting of safety statistics (n = 38/63, 60%) [cOR 2.07, 95% CI 1.2-3.5], P = 0.01, relative to other nature of interventions.
  4. Therapeutic areas: Cardiovascular studies, n = 120/172 (70%) had better reporting of safety statistics (n = 120/172,70%) [cOR 4.4, 95% CI 3.0-6.4], <0.0001 whereas oncology trials had poorer reporting of safety statistics (n = 20/101,19.8%) [cOR 0.26, 95% CI 0.1- 0.4], P < 0.0001. relative to other therapeutic areas.


The associations of reporting of safety statistics is shown in [Table 3].
Table 3: Association of reporting of safety statistics

Click here to view



 :: Discussion Top


In the audit conducted by us on reporting of safety statistics in 623 RCTs published over a five year period, we found that only 43% of RCTs have reported safety statistics in the form of 'P value'or 95% confidence intervals. Moreover, 11% RCTs reported safety statistics only in the supplement indicating that safety reporting is not a priority. We also found higher reporting of safety statistics in Investigator initiated studies [IIS] relative to studies funded by the pharmaceutical industry.

About two-third IIS had better reporting of safety outcomes whereas less than one third PISS reported safety statistics. This indicates the efficacy centric approach of the latter whereas both safety and efficacy is paramount for investigators as they treat patients on day to day basis and have to explain both benefit and risk to the patients to enable them to make an informed choice. Additionally differences in such reporting highlights the lack of a consensus on the use of a statistical tool in reporting of adverse events. Although, the CONSORT statement extension for harm-related data published in 2004 provided guidelines in such reporting, the use of statistics in safety data is not clearly defined. The guideline highlighted common poor reporting practices for harm related data of which one of them is; 'Reporting only the adverse events that reach a P value threshold in the comparison of the randomized arms (for example, P < 0.05)'.[7]

The answer to better reporting of safety statistics in phase 4 trials over phase 3 trials may lie in the objective of these trials. Phase 4 trials are more concerned about monitoring drug effectiveness in the general population (widespread use) while collecting information about adverse effects which may result in more reporting of safety 'P value or 95% CI'. Phase 4 studies also have higher sample size compared to the phase 3 studies and hence led to greater reporting of safety signals.

We found that cardiovascular trials were more likely to report safety statistics. This may be likely because these trials largely used composite outcome (which included safety) as a primary endpoint. Only one-fifth of the oncology trials reported safety statistics. This is surprising, considering the high prevalence of adverse events associated with oncology drugs. One reason may be inability of the investigators to distinguish AEs caused by drugs and disease. In addition the patients may not report adverse events by themselves unless they are specifically requested to report them because they think it is a necessary part of the therapy.

Often, safety takes a backseat to efficacy and P value of safety are criticized and tend to be overinterrupted for secondary outcomes. Safety endpoint as a secondary outcome also faced criticism provided multiplicity issues are adjusted, else there could be over interpretation of the P value or the confidence interval adding potential concern for too many outcomes.

There is a responsibility of the authors regardless of whether they are from academia or pharmaceutical industry in being transparent in reporting of safety statistics (transparency in reporting). Responsibility of the journal editor or the peer-reviewers to see efficacy and safety are equally well reported in the manuscript. So, the CONSORT should consider revising their own guidelines and to give specific recommendation for reporting of safety statistics. Our study is limited by being a retrospective audit only from a single journal.


 :: Conclusion Top


Safety reporting in RCTs continues to take a back seat relative to efficacy reporting and is worse for pharmaceutical industry funded studies. Uniformity in reporting statistics for both efficacy and safety outcomes across all studies will help physicians and patients to make an informed choice about treatments.

Financial support and sponsorship

A grant in aid was provided by Diamond Jubilee Society Trust, Seth GS Medical College and KEM Hospital.

Conflicts of interest

There are no conflicts of interest.



 
 :: References Top

1.
Sim I, Olasov B, Carini S. An ontology of randomized controlled trials for evidence-based practice: Content specification and evaluation using the competency decomposition method. J Biomed Inform 2004;37:108-19.  Back to cited text no. 1
    
2.
Pitrou I, Boutron I, Ahmad N, Ravaud P. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med 2009;169:1756-61.  Back to cited text no. 2
    
3.
Garg SK, Henry RR, Banks P, Buse JB, Davies MJ, Fulcher G, et al. Effects of sotagliflozin added to insulin in patients with type 1 diabetes. N Engl J Med 2017;377:2337-48.  Back to cited text no. 3
    
4.
Hussain M, Fizazi K, Saad F, Rathenborg P, Shore N, Ferreira U, et al. Enzalutamide in men with nonmetastatic, castration-resistant prostate cancer. N Engl J Med 2018;378:2465-74.  Back to cited text no. 4
    
5.
Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: An evaluation of 7 medical areas. JAMA 2001;285:437-43.  Back to cited text no. 5
    
6.
Ioannidis JP, Evans SJ, Gøtzsche PC, O'Neill RT, Altman DG, Schulz K, et al. Better reporting of harms in randomized trials: An extension of the CONSORT statement. Ann Intern Med 2004;141:781-8.  Back to cited text no. 6
    
7.
nejm.org [homepage on the Internet]. United States: New England Journal of Medicine. 2021. Available from: https://www.nejm.org/about nejm /about nejm. [Last accessed on 2021 Apr 09].  Back to cited text no. 7
    
8.
Arriaga AF, Bader AM, Wong JM, Lipsitz SR, Berry WR, Ziewacz JE, et al. A simulation-based trial of surgical-crisis checklists. N Engl J Med 2013;368:246-53.  Back to cited text no. 8
    


    Figures

  [Figure 1]
 
 
    Tables

  [Table 1], [Table 2], [Table 3]



 

Top
Print this article  Email this article
 
Online since 12th February '04
2004 - Journal of Postgraduate Medicine
Official Publication of the Staff Society of the Seth GS Medical College and KEM Hospital, Mumbai, India
Published by Wolters Kluwer - Medknow