 |
|
||||
|
|||||
| About | Latest Articles | Back-Issues | Articles
|
Search | Instructions | Online Submission | Subscribe | Etcetera | Contact |
| |||||||||||||||||||||
|
Impact factor of medical education journals and recently developed indices: Can any of them support academic promotion criteria? SA Azer1, A Holen2, I Wilson3, N Skokauskas41 Department of Medical Education, Curriculum and Research Unit, College of Medicine, King Saud University, Riyadh, Saudi Arabia 2 Faculty of Medicine, Norwegian University of Science and Technology (NTNU), St. Olav University Hospital, Trondheim, Norway 3 Graduate School of Medicine, University of Wollongong, New South Wales, Australia 4 Centre for Child and Adolescent Mental Health and Child Protection, Department of Neuroscience, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
Correspondence Address: Source of Support: None, Conflict of Interest: None DOI: 10.4103/0022-3859.173202
Journal Impact Factor (JIF) has been used in assessing scientific journals. Other indices, h- and g-indices and Article Influence Score (AIS), have been developed to overcome some limitations of JIF. The aims of this study were, first, to critically assess the use of JIF and other parameters related to medical education research, and second, to discuss the capacity of these indices in assessing research productivity as well as their utility in academic promotion. The JIF of 16 medical education journals from 2000 to 2011 was examined together with the research evidence about JIF in assessing research outcomes of medical educators. The findings were discussed in light of the nonnumerical criteria often used in academic promotion. In conclusion, JIF was not designed for assessing individual or group research performance, and it seems unsuitable for such purposes. Although the g- and h-indices have demonstrated promising outcomes, further developments are needed for their use as academic promotion criteria. For top academic positions, additional criteria could include leadership, evidence of international impact, and contributions to the advancement of knowledge with regard to medical education. Keywords: Academia, Article Influence Score (AIS), h- and g-indices, Journal Impact Factor (JIF), Medical education, Academic promotion, Research publications
Research is a fundamental aspect of academic life. It also represents an aspect of scholarship in medical education. Each month, approximately 60,000-65,000 new health-related research articles are published and indexed in the PubMed portal. [1] In most journals, however, the quality of the publications varies. Some papers are not clearly written, have poorly described methods, or use tools of low validity and reliability in spite of the Journal Impact Factor (JIF). [2] In academia, there is a need for introducing new indices to define the quality of research publications. Academic departments, research centers, and funding bodies are increasingly interested in ways to assess academics' research production and the quality of individuals' research outcomes. In most universities, promotion and tenure systems reward individual achievements using general citation-based journal rankings. Although JIF is meant for journal rankings, several institutes let the ranking of journals where researchers published their work influence the academic career progression and the funding of grants. [3],[4],[5],[6] Medical educators, like other academics, are under pressure to publish their work in top-ranking journals listed in the Science Citation Index (SCI), Social Sciences Citation Index (SSCI), and Journal Citation Reports (JCR). For the preceding year, the JIFs are published in the JCR each June. The JIF of a scientific journal is the ratio of the number of citations found for the two preceding years of articles published and divided by the number of citable items published in the same two years. [7],[8] In a competitive research environment, alternative citation tracking allows researchers and universities to:
This would enable researchers to work on the quality of their research to match the standards required by top journals in their field. [9],[10],[11] Several studies have examined journal rankings in journals of different disciplines including nursing, [12],[13] nutrition, [14] public health, [15] neurosurgery, [16] dermatology, [17] forensic science and toxicology, [3] psychology, [18] orthopedics, [19] radiation oncology, [20] and medical informatics. [21] For medical education, however, no studies have assessed the impact factor or discussed possible new tools for citation analysis. In the same vein, the h- and g-indices and the Article Influence Score (AIS) have not been studied in relation to medical education. [22],[23],[24] The first part of this paper aims to review data sources and approaches for citation analysis. This knowledge is then applied to the assessment of 15 medical education journals to define highly regarded medical education titles by gathering data for each tool for these journals from Web of Science. We also aim to examine the strengths and limitations of using the JIF and other indices: h- and g-indices and the AIS. The second part aims to assess whether any of these indices would add more evidence to support the policies and criteria of academic promotion and grant assessments and their current use in medical education.
Journal Impact Factor (JIF): A critical review The JIF has emerged as a tool for ranking, evaluating, categorizing, and comparing scientific journals. [3],[8],[25] The Institute for Scientific Information (ISI), a component of Thomson Scientific, was behind this development. A listing of journals' citations and their JIFs is made available by the ISI (Philadelphia, PA, USA), and it is also included in the JCR. It is important to note that the citation data of a single year and the citation data from only the two previous years' articles constitute a significant limitation of the JIFs. [26] Considering the fact that the average paper is not cited in the first year after publication, data gathered for 1-2 years post publication is likely to provide an unrepresentative low snapshot of the Impact Factor. However, other researchers have shown that the relative short-term citation impact measured in the window underlying the JIF is a good predictor of the citation impact of the journals in the years to come. [6] Another criticism of JIF is related to its calculation. JIF depends on which article types Thomson Scientific deems "citable". Another limitation of the JIF is that the quality of the articles varies within a journal; the distribution of citations is skewed by only a few articles close to the population mean. [27],[28],[29],[30] Therefore, the publication of review articles (which usually acquire far more citations than research articles) or the publication of just a few very highly cited research papers can improve a journal's JIF. It has been shown that less than 20% of the articles published in a journal account for more than 50% of the total number of citations. Many articles are not cited at all, or they are cited because some readers disagree with the authors. [24],[31],[32] Accordingly, a single publication cannot be judged by the JIF. Added to this is the bias that may occur due to self-citations. [33] However, the JIF may be misused or abused by journals with the aim to improve their impact factor. For example:
More on recently developed indices To resolve the problems related to self-citations, Eigenfactor TM Metrics (http://www.eigenfactor.org/) was created by Carl Bergstrom, Jevin West, and Marc Wiseman at the Information School, University of Washington, Seattle, Washington, United States. [32],[33],[34],[35] The Eigenfactor Score is somewhat similar to a JIF but is corrected for the journal's self-citations. Therefore, references from one article in a journal to another published in the same journal are removed during the calculation of the Eigenfactor. Google Scholar and Scopus Google Scholar was launched in 2004 as a gateway to scholarly literature. [36] The database is readily available free of charge and shows the number of citations of and details about the journals citing each paper. However, the contents are not organized under subject headings. This makes it difficult to assess a researcher's publication outcomes. In addition, it shows a broader range of sources than JCR or Scopus, resulting in the inclusion of nonjournal sources. Scopus is an indexing database built by Elsevier Co. and launched in 2004. The database claims 4600 health sciences titles and shows 100% coverage of the databases MEDLINE/PubMed, Embase, and Compendex. More details about Scopus have been highlighted elsewhere. [12],[36],[37] However, neither Google Scholar nor Scopus have addressed the limitations of JIF. The h-index In 2005, JE Hirsch proposed the h-index to assess the impact of an individual author. [22],[23],[36] The h-index has been shown to be of no value in journal ranking. To determine the h-index of an author, papers are ranked in a decreasing order of their received citations; the h-index is the (unique) highest number of papers that received h or more citations. [22],[23] The h-index may have several advantages, as outlined in [Table 1]. However, the h-index is not sensitive enough to indicate changes even if the paper receives 5, 50, or 500 more citations: The index does not capture such changes in citations over time. [23],[38]
The g-index Because of the limitations of the h-index and its insensitivity to highly cited articles, Egghe proposed the g-index. [23] The g-index is sensitive to the most cited articles. The g-index is defined as the highest number of papers that together received g2 or more citations. In other words, the higher the number of citations received for an article, the higher the g-index. [23] To explain the differences between the h- and g-indices and the sensitivity of the latter to highly cited articles, let us look at two examples. Researcher A has published five articles with 5 citations. This researcher has an h-index of 5. Researcher B has published 5 papers; four of them attracted 5 citations each, and the remaining one attracted 15 citations. The h-index for researcher B is also 5, while the g-index will vary depending on the number of citations attracted by the best article he/she has published. If the citations attracted by the best article were 15, 25, or 50, the g-index would be 6, 7, and 9, respectively. Therefore, the g-index is more sensitive in assessing a researcher's productivity than the h-index and far more accurate than the JIF in assessing individual researchers. The Article Influence Score (AIS) This index calculates the relative importance of the journal on a per-article basis. The AIS is obtained by dividing the Eigenfactor Score by the number of articles published in the journal and normalized to make the overall AIS of all journals 1.0. It is roughly analogous to the 5-year JIF; it is the ratio of the journal's citation influence to the size of the journal's article contribution over a period of 5 years. [39] [Table 1] summarizes key information, strengths, and weaknesses of different metrics.
Academic promotion For staff promotion, the universities often count such parameters as: 1. Number of papers published in peer review journals; 2. Number of papers published in top-ranking journals [7] ; 3. Number of citations and cites per paper; 4. Other scholarly work such as the number of patents, the number of graduate students supervised, conference papers at national and international levels, research books, chapters of books, and monographs; and 5. The number of grants and research projects with the applicant as the principal researcher or associate investigator. [40] Interestingly, there has been limited discussion in the literature about academic promotion, but extensive documentation on university webpages. The existing literature criticizes such bibliometrics in decision-making. Notably, this has resulted in a discussion concerning the academic nursing profession, [41] similar to that seen in medical education: The amount of research is limited, but there is also considerable diversity in the research methodology. The wide use of JIF in academic appointments and promotions takes two forms: The "quality" of the journals in which the applicant is publishing and the "quality" of the papers as measured by the number of citations. Citation indices and staff promotion [Table 2] shows 16 highly regarded medical and allied health education journals with the JIF scores from 2000 to 2011. The total cites in 2011 under the category "Education, Scientific Discipline" were 42,997, and the Median Impact Factor was 0.902 for a total of 33 journals indexed under this category. Only 16 journals were selected for this study as the other journals covered other disciplines.
Interestingly, Advances in Health Sciences Education, which was indexed for the first time in 2003, has demonstrated progressive increases in its JIFs over the following years. Other journals, such as Teaching and Learning in Medicine, which was indexed in 2000, have failed to demonstrate significant improvement in its JIFs over these past years. The recently published journal Anatomical Sciences Education, however, was indexed for the first time in 2010, with a JIF of 2.976. The largest increase was found for Academic Medicine and Medical Education, whose JIF scores increased from 1.554 and 1.078 in 2000 to 3.524 and 3.176 in 2011, respectively. Two other journals with noteworthy performance were Advances in Health Sciences Education and Advances in Physiology Education. Although Medical Teacher has shown progressive increases in its JIF scores over the years, the improvement in the JIF values has been small. [Table 3] shows that 10 journals indexed in 2011 had 5-year JIF scores ranging from 3.189 (Medical Education) to 0.600 (Journal of Biological Education). The correlation between the 2-year JIF and 5-year JIF for these journals was high (r = 0.89, P < 0.001), which is consistent with other studies. [25]
[Table 4] summarizes additional information about medical and allied health journals indexed in the JCR. For each journal, the table shows the number of citable articles and citable reviews in 2011 for 15 journals (no information available on Anatomical Sciences Education) as well as the number of references and the ratio of references to total citable items (articles and reviews). The number of citable reviews varied widely.
From [Table 4] it appears that the mean number of references in the citable articles varied widely. It ranged from a low of 16.6 (Biochemistry and Molecular Biology Education) to a high of 35.6 (Medical Education). [Table 5] shows the ranking of medical and allied health education journals and the AIS of each journal. As is the case with JIF, only a few manuscripts enhance this score, while most manuscripts have not acquired a sufficient number of citations.
In this paper, JIF has been analyzed and compared with later developments in the use of citations for the evaluation of research quality in general, and the journals addressing medical education have been explored in some depth. The introduction of JIF in 1997 was a major milestone. Today, however, the limitations of JIF are clearly felt by many, [24],[31],[32],[34],[42] and there is a growing need for additional, more sophisticated tools in all stages of scientific endeavor to optimize future success in research funding and academic recruiting. The development of medical education is today ever more guided by research, [43],[44] but so far, no citation analysis of the JIF in comparison to the AIS, h-indices, and g-indices has been made. The ranking of medical education journals will probably fill an information gap within the health sciences. In this analysis, a number of well-regarded medical and allied health journals listed in JCR have been selected, analyzed, and compared. From the analyses of the citation indices, the realization emerges with some strength that the current use of JIF does not serve the best of academic interests; an unjustifiable discrepancy between the journal ranking and the author ranking can be considerable. Moreover, there is a JIF bias in favor of publications within fields having a rapid turnover. JIF does not have the sensitivity and specificity to adequately meet the current needs and expectations for advances in the academic community across research fields. Accordingly, when the funding of individual researchers or groups is to be decided or when making decisions about academic promotions, the use of the h- and g-indices together with the AIS is more likely to result in better assessments. The San Francisco Declaration on Research Association (DORA) recommends that JIF should not be used as a surrogate measure of the quality of an individual research article. [45] Another important issue is the growing realization that JIFs are biased toward certain fields of research. For example, JIF is strongly in favor of high-profile disciplines with a rapidly cycled field of discoveries and turnover, such as molecular biology and biochemistry. This does injustice to low-profile disciplines such as health education, nursing, and midwifery. [46] The speed of turnover makes it difficult for medical educators to compete with colleagues from some other disciplines. It is also important to realize that the highest impact factors for journals covering medicine, biochemistry and molecular biology, biochemical research methods, and biology are 53.298, 34.317, 19.276, and 11.452, respectively, while the highest impact factor for medical education journals is only 3.524 (for Academic Medicine). Furthermore, the numbers of journals in the area of medicine (general and internal), biochemistry and molecular biology, biochemical research methods, and biology indexed in the JCR are 155, 200, 72, and 85, while only 14 journals are indexed under medical education, and one for dentistry education, and another one for pharmacy and pharmaceutical education. This situation leaves limited opportunities for medical and allied health educators to publish their work in high-impact journals. As another example, consider that a medical educator publishes an article in Academic Medicine, a journal with a JIF of 3.524, and another colleague from the Department of Medicine at the same institute publishes in Annals of Medicine, a journal with a JIF of 3.516. Both journals have nearly the same JIF, but Academic Medicine is the top journal in medical education, while Annals of Medicine is ranked #19 in its own field. This major difference is totally ignored if only the JIF is considered in the academic assessment of research outcomes. Nevertheless, better indices provide vital support in decision-making for research for funding, recruitment, and improved teaching in the competitive environment of academia. In certain ways, a change in the current use of citation indices will sharpen the competition in wholesome ways. More importantly, it is likely to enable better decisions and more fairness with regard to assessments of the publication output of individuals and research groups across disciplines and methodologies. In addition to these metrics, a battery of other indices should form the basis for academic promotion, particularly for top positions, including the following: 1. Invitations to speak internationally about research, 2. A sustained record of being the principal investigator in funded research, 3. Services as an editor and/or editorial board member of medical education journals and scientific journals, and years as peer reviewer to top international journals in the field, 4. Leadership roles on national and international committees of major medical education societies, and major conferences on medical education, 5. Prestigious national and international awards for research and innovations in medical education, 6. Leadership in international collaboration in research and publication as principal investigator, and 7. Leadership and accumulated achievements in specific areas in medical education. Each of these indices could be standardized by a numerical system. For example, invitations as a keynote speaker may be evaluated by using the following scoring system: 0 = not invited, 1 = invited to speak in a meeting held within their own university, 2 = invited to speak at a national conference, 3 = invited to speak at an international university ranked lower than their own, 4 = invited to speak at an international university ranked higher than their own, 5 = invited to speak at a major international conference. Indices such as these could enhance assessment for academic promotion.
Given the need for tighter links between research quality and funding as well as recruitment practices, it is time to revise the scientific evaluations also within medical teaching; institutional decisions should preferably be evidence-based and favor individuals with solid scientific merit rather than be driven by coincidental or ideological motives. In the absence of better tools, rough approximations of scientific quality were derived from the JIF in the past. Although AIS and the g- and h-indices have shown promising outcomes, further developments are needed. Other key indices, particularly for top academic positions, should also be considered. Financial support and sponsorship This work was funded by the College of Medicine Research Center, Deanship of Scientific Research, King Saud University, Riyadh, Saudi Arabia. Conflicts of interest The authors declare that they have no conflict of interest and that the whole manuscript has been created by the authors.
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5]
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||
