Journal of Postgraduate Medicine
 Open access journal indexed with Index Medicus & ISI's SCI  
Users online: 126  
Home | Subscribe | Feedback | Login 
About Latest Articles Back-Issues Article Submission Resources Sections Etcetera Contact
 
  NAVIGATE Here 
  Search
 
 :: Next article
 :: Previous article 
 :: Table of Contents
  
 RESOURCE Links
 ::  Similar in PUBMED
 ::  Search Pubmed for
 ::  Search in Google Scholar for
 ::Related articles
 ::  Article in PDF (52 KB)
 ::  Citation Manager
 ::  Access Statistics
 ::  Reader Comments
 ::  Email Alert *
 ::  Add to My List *
* Registration required (free) 

  IN THIS Article
 ::  Abstract
 ::  Materials and Me...
 ::  Results
 ::  Discussion
 ::  References
 ::  Article Tables

 Article Access Statistics
    Viewed5827    
    Printed203    
    Emailed1    
    PDF Downloaded332    
    Comments [Add]    
    Cited by others 46    

Recommend this journal


 


 
CME
Year : 2008  |  Volume : 54  |  Issue : 3  |  Page : 214-216

A trial design that generates only ''positive'' results


Complementary Medicine, Peninsula Medical School, Universities of Exeter and Plymouth, 25 Victoria Park Road, Exeter EX2 4NT, United Kingdom

Correspondence Address:
E Ernst
Complementary Medicine, Peninsula Medical School, Universities of Exeter and Plymouth, 25 Victoria Park Road, Exeter EX2 4NT
United Kingdom
Login to access the Email id


DOI: 10.4103/0022-3859.41806

PMID: 18626172

Get Permissions


 :: Abstract 

In this article, we test the hypothesis that randomized clinical trials of acupuncture for pain with certain design features (A + B versus B) are likely to generate false positive results. Based on electronic searches in six databases, 13 studies were found that met our inclusion criteria. They all suggested that acupuncture is effected (one only showing a positive trend, all others had significant results). We conclude that the 'A + B versus B' design is prone to false positive results and discuss the design features that might prevent or exacerbate this problem.


Keywords: Acupuncture, bias, clinical trial, false positive, trial design


How to cite this article:
Ernst E, Lee M S. A trial design that generates only ''positive'' results. J Postgrad Med 2008;54:214-6

How to cite this URL:
Ernst E, Lee M S. A trial design that generates only ''positive'' results. J Postgrad Med [serial online] 2008 [cited 2014 Apr 23];54:214-6. Available from: http://www.jpgmonline.com/text.asp?2008/54/3/214/41806


Randomized clinical trials (RCTs) are designed to minimize bias in comparative tests of therapeutic effectiveness. However, they are not always free of bias. In recent years, we have seen a plethora of RCTs adopting a design where patients are randomized to receive either usual care (the control group) or usual care plus the experimental treatment. Schematically this design could be depicted as 'A + B versus B'. At first glance, such comparisons may seem reasonable. However, on closer inspection, doubts emerge regarding whether such RCTs are fair scientific tests of the experimental intervention. These doubts originate from the theoretical view that 'A plus B' will invariably amount to more than 'B' alone. Even in cases where treatment A is a pure placebo, its placebo and other nonspecific effects could lead to a better outcome in the experimental group than in the control group. This would be particularly likely if i) the experimental treatment is associated with sizable nonspecific effects, ii) a subjective outcome measure is used, and iii) the experimental intervention 'A' causes a deterioration of the condition being treated.

Here we evaluate RCTs with the 'A + B versus B' design to determine whether any of these trials produced 'negative' findings, i.e. a result where the experimental group experiences outcomes which are either the same or worse than those of the control group. As hundreds of such studies exist, we focus on one particular situation which fulfils the above listed three criteria: acupuncture as a treatment of pain.


 :: Materials and Methods Top


Electronic databases were searched from their respective inception up to 10 October 2007: Medline, AMED, British Nursing Index, CINAHL, EMBASE, and PsycInfo. The search terms used were: [acupuncture AND pain AND (usual care OR routine care OR standard care OR wait-list)] OR [acupuncture AND pain AND random]. In addition, screening of reference lists of all located articles and reviews was performed and our departmental files were hand searched.

Articles were considered if they reported an RCT in which human patients with any type of pain were treated with any type of acupuncture. Specifically, we included studies that fulfilled the following criteria: randomized patient allocation; pain as one outcome measure; control group receiving usual care; experimental group receiving usual care plus acupuncture. No language restrictions were imposed. We excluded feasibility studies, trials testing the effectiveness of TENS or laser treatment and studies that failed to include sufficient detail for assessment.

Hard copies of all articles were obtained and read in full by both authors. When an RCT had more than two treatment groups, we evaluated only the two groups ('A + B' and 'B') mentioned above. Both authors independently extracted key data [Table 1]. The methodology quality of all RCTs was assessed using the modified Jadad score. [1]


 :: Results Top


We found 200 RCTs, 13 of which met our inclusion/exclusion criteria [Table 1]. [2],[3],[4],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14] Eleven RCTs had a parallel group design, [2],[5],[6],[7],[8],[9],[10],[11],[12],[13],[14] and two followed a partial crossover design. [3],[4] Nine trials included two groups, [3],[4],[6],[8],[10],[11],[12],[13],[14] three had three groups [5],[7],[9] and one had four groups of patients [Table 1]. [2]

The RCTs differed in many aspects: they recruited patients with different types of pain, employed a range of methods for quantifying pain, used different types of acupuncture administered according to different treatment schedules, employed different types of usual care, and varied in terms of Jadad score and sample size. Despite these differences, all studies generated results that suggest significantly better outcomes in the experimental ('A + B') compared to the control group ('B'). The only exception is the RCT by Helms et al. , [2] which only showed a non-significant trend in that direction, most likely due to a Type II error caused by its very small sample size (n = 12).


 :: Discussion Top


Randomized clinical trials with the 'A + B versus B' design are currently popular, particularly for pragmatic studies. Our findings suggest that, in the realm of acupuncture for pain, this design is likely to generate false positive results. Alternatively acupuncture could, of course, be a highly effective intervention for reducing pain. However, recent RCTs which rigorously control for placebo effects by employing non-penetrating sham-devices as control interventions suggest that acupuncture is not superior to sham-acupuncture for pain. [15] Moreover, several RCTs with patients suffering from conditions other than pain (for which acupuncture has not been proven to be effective) indicate that this phenomenon may not be confined to studies of pain.

The reason for the inability of the 'A + B versus B' design to generate negative results (under the above-stated three conditions) seems obvious: even in the absence of any specific therapeutic effect, the results of such studies would be positive due to nonspecific effects such as a placebo-effect, the additional care given to patients, the therapist-patient relationship or social desirability. A further contributor could be the disappointment experienced by patients of the control groups when not receiving the experimental treatments they may have hoped for.

The design issues highlighted here are well known to research methodologists. They are, however, not always appreciated by clinicians who read published articles and may tend to equate RCTs with the highest level of reliability. Often the authors of these articles fail to discuss the drawbacks of the 'A + B versus B' design in sufficient detail. Certainly they were not emphasized by the authors of the studies included in this review.

Studies with three of more groups are usually designed to answer more than one research question. Therefore such RCTs might be valuable even if they include a comparison of 'A + B versus B'. Conclusions based specifically on any 'A + B versus B' comparison within such trials can nevertheless be criticized on the basis of the above arguments. Similarly, "pragmatic" trials that adopt a different design (for instance 'A versus B') are not affected by our criticism.

It would, of course, be an over-interpretation of our results to state categorically that RCTs with the 'A + B versus B' designs can, in principle, only generate positive results. We have shown this to be likely only for acupuncture as a treatment for pain. In other therapeutic areas, negative trials with that design may exist. Crucially, however, these RCTs do not fulfill the three additional criteria outlined in the introduction of this article. Interventions that are less prone to generating false positive findings when tested in a 'A + B versus B' study include those that not only do not improve but worsen the condition in question and treatments which do not generate sizable placebo-effects. Also, studies that employ objectively measurable endpoints might, in some cases, offer a protection against such false positive findings.

In conclusion, our systematic review of RCTs of acupuncture for pain control with the 'A + B versus B' suggests that this trial methodology is likely to produce false positive results. Such studies may therefore not be adequate scientific tests of the effectiveness of therapeutic interventions.

 
 :: References Top

1.White AR, Ernst E. A systematic review of randomized controlled trials of acupuncture for neck pain. Rheumatol 1999;38:143-7.  Back to cited text no. 1    
2.Helms JM. Acupuncture for the management of primary dysmenorrhea. Obstet Gynecol 1987;69:51-6.  Back to cited text no. 2  [PUBMED]  
3.Berman BM, Singh BB, Lao L, Langenberg P, Li H, Hadhazy V, et al . A randomized trial of acupuncture as an adjunctive therapy in osteoarthritis of the knee. Rheumatology 1999;38:346-54.  Back to cited text no. 3  [PUBMED]  [FULLTEXT]
4.Meng CF, Wang D, Ngeow J, Lao L, Peterson M, Paget S. Acupuncture for chronic low back pain in older patients: A randomized, controlled trial. Rheumatology 2008;42:1508-17.  Back to cited text no. 4    
5.Tukmachi E, Jubb R, Dempsey E, Jones P. The effect of acupuncture on the symptoms of knee osteoarthritis: An open randomised controlled study. Acupunct Med 2004;22:14-22.  Back to cited text no. 5  [PUBMED]  [FULLTEXT]
6.Vickers AJ, Rees RW, Zollman CE, McCarney R, Smith C, Ellis N et al . Acupuncture for chronic headache in primary care: Large, pragmatic, randomised trial. BMJ 2004;328:744.  Back to cited text no. 6    
7.Witt C, Brinkhaus B, Jena S, Linde K, Streng A, Wagenpfeil S et al . Acupuncture in patients with osteoarthritis of the knee: A randomised trial. Lancet 2005;366:136-43.  Back to cited text no. 7    
8.Witt CM, Jena S, Brinkhaus B, Liecker B, Wegscheider K, Willich SN. Acupuncture in patients with osteoarthritis of the knee or hip: A randomized, controlled trial with an additional nonrandomized arm. Arthritis Rheumat 2006;54:3485-93.  Back to cited text no. 8  [PUBMED]  [FULLTEXT]
9.Brinkhaus B, Witt CM, Jena S, Linde K, Streng A, Wagenpfeil S, et al . Acupuncture in patients with chronic low back pain: A randomized controlled trial. Arch Intern Med 2006;166:450-7.  Back to cited text no. 9    
10.Thomas KJ, MacPherson H, Thorpe L, Brazier J, Fitter M, Campbell MJ, et al . Randomised controlled trial of a short course of traditional acupuncture compared with usual care for persistent non-specific low back pain. BMJ 2006;333:623-6.  Back to cited text no. 10    
11.Witt CM, Jena S, Selim D, Brinkhaus B, Reinhold T, Wruck K, et al . Pragmatic randomized trial evaluating the clinical and economic effectiveness of acupuncture for chronic low back pain. Am J Epidemiol 2006;164:487-96.  Back to cited text no. 11  [PUBMED]  [FULLTEXT]
12.Witt CM, Jena S, Brinkhaus B, Liecker B, Wegscheider K, Willich SN. Acupuncture for patients with chronic neck pain. Pain 2006;125:98-106.  Back to cited text no. 12  [PUBMED]  [FULLTEXT]
13.Goertz CM, Niemtzow R, Burns SM, Fritts MJ, Crawford CC, Jonas WB. Auricular acupuncture in the treatment of acute pain syndromes: A pilot study. Mil Med 2006;171:1010-4.  Back to cited text no. 13  [PUBMED]  
14.Reinhold T, Witt CM, Jena S, Brinkhaus B, Willich SN. Quality of life and cost-effectiveness of acupuncture treatment in patients with osteoarthritis pain. Eur J Health Economics 2007;(epub: 19 7 2007).  Back to cited text no. 14    
15.Ernst E. Acupuncture: A critical analysis. J Intern Med 2006;259:125-37.  Back to cited text no. 15  [PUBMED]  [FULLTEXT]



 
 
    Tables

  [Table 1]

This article has been cited by
1 A systematic review and meta-analysis on the use of traditional Chinese medicine compound kushen injection for bone cancer pain
Bao Yanju,Liping Yang,Baojin Hua,Wei Hou,Zhan Shi,Weidong Li,Conghuang Li,Cihui Chen,Rui Liu,Yinggang Qin,Wenliang Lv
Supportive Care in Cancer. 2013;
[Pubmed]
2 Chinese Herbal Medicine in Treatment of Polyhydramnios: a Meta-analysis and Systematic Review
Fen Zhou,Yu-fang Hao,Yan Chen,Tong Wang
Chinese Medical Sciences Journal. 2013; 28(2): 72
[Pubmed]
3 Xiaoxuming decoction for acute ischemic stroke: A systematic review and meta-analysis
Deng-lei Fu,Lin Lu,Wenzong Zhu,Ji-huang Li,Hui-qin Li,Ai-ju Liu,Chenglong Xie,Guo-qing Zheng
Journal of Ethnopharmacology. 2013; 148(1): 1
[Pubmed]
4 Chuna therapy for musculoskeletal pain: A systematic review of randomized clinical trials in Korean literature
Tae-Woong Moon,Tae-Young Choi,Tae-Yong Park,Myeong Soo Lee
Chinese Journal of Integrative Medicine. 2013; 19(3): 228
[Pubmed]
5 Scalp acupuncture for stroke recovery: A systematic review and meta-analysis of randomized controlled trials
Seung-Jin Lee,Byung-Cheul Shin,Myeong Soo Lee,Chang-Ho Han,Jong-In Kim
European Journal of Integrative Medicine. 2013; 5(2): 87
[Pubmed]
6 Shengmai Injection, a Traditional Chinese Patent Medicine, for Intradialytic Hypotension: A Systematic Review and Meta-Analysis
Chao-yang Chen,Ling-yan Lu,Peng Chen,Kang-ting Ji,Jia-feng Lin,Peng-lin Yang,Ji-fei Tang,Yan Wang
Evidence-Based Complementary and Alternative Medicine. 2013; 2013: 1
[Pubmed]
7 Efficacy and safety of Suanzaoren decoction for primary insomnia: a systematic review of randomized controlled trials
Cheng-long Xie,Yong Gu,Wen-Wen Wang,Lin Lu,Deng-lei Fu,Ai-ju Liu,Hui-qin Li,Ji-huang Li,Yan Lin,Wen-jie Tang,Guo-qing Zheng
BMC Complementary and Alternative Medicine. 2013; 13(1): 18
[Pubmed]
8 Does the Couse of Astragalus-Containing Chinese Herbal Prescriptions and Radiotherapy Benefit to Non-Small-Cell Lung Cancer Treatment: A Meta-Analysis of Randomized Trials
Hailang He,Xianmei Zhou,Qian Wang,Yang Zhao
Evidence-Based Complementary and Alternative Medicine. 2013; 2013: 1
[Pubmed]
9 Acupressure for the Treatment of Musculoskeletal Pain Conditions: A Systematic Review
Young Chan Kim, Myeong Soo Lee, Eun-Seo Park, Jae-Hwan Lew, Beom-Joon Lee
Journal of Musculoskeletal Pain. 2012; : 1
[VIEW]
10 Editorial: Consensus on CAM methods for nursing research? : Editorial
Graeme D Smith
Journal of Clinical Nursing. 2012; 21(5-6): 599
[VIEW]
11 Moxibustion for the treatment of osteoarthritis: a systematic review and meta-analysis
Tae-Young Choi, Jiae Choi, Kun Hyung Kim, Myeong Soo Lee
Rheumatology International. 2012;
[VIEW]
12 Clinical efficacy and safety of Chinese herbal medicine for Wilsonæs disease: A systematic review of 9 randomized controlled trials
Yan Wang,Cheng-long Xie,Deng-lei Fu,Lin Lu,Yan Lin,Qi-qian Dong,Xiao-tong Wang,Guo-qing Zheng
Complementary Therapies in Medicine. 2012; 20(3): 143
[Pubmed]
13 Chinese Herbal Medicine Paratherapy for Parkinsonæs Disease: A Meta-Analysis of 19 Randomized Controlled Trials
Yan Wang,Cheng-Long Xie,Lin Lu,Deng-Lei Fu,Guo-Qing Zheng
Evidence-Based Complementary and Alternative Medicine. 2012; 2012: 1
[Pubmed]
14 A study that can only generate positive results
E Ernst
Focus on Alternative and Complementary Therapies. 2011; 16(3): 237
[Pubmed]
15 Acupressure for Treating Neurological Disorders: A Systematic Review
Jeong-Sook Lee, Myeong Soo Lee, Kyungyoon Min, Jae-Hwan Lew, Beom-Joon Lee
International Journal of Neuroscience. 2011; 121(8): 409
[VIEW]
16 The feasibility of a pragmatic randomised controlled trial to compare usual care with usual care plus individualised homeopathy, in children requiring secondary care for asthma
E.A. Thompson, A. Shaw, J. Nichol, S. Hollinghurst, A.J. Henderson, T. Thompson, D. Sharp
Homeopathy. 2011; 100(3): 122
[VIEW]
17 Cupping for Treating Pain: A Systematic Review
Jong-In Kim,Myeong Soo Lee,Dong-Hyo Lee,Kate Boddy,Edzard Ernst
Evidence-Based Complementary and Alternative Medicine. 2011; 2011: 1
[Pubmed]
18 Massage therapy for children with autism spectrum disorders: A systematic review
Lee, M.S., Kim, J.-I., Ernst, E.
Journal of Clinical Psychiatry. 2011; 72(3): 406-411
[Pubmed]
19 Effectiveness of Pharmacopuncture for Asthma: A Systematic Review and Meta-Analysis
Feng-Yan Shen, Myeong Soo Lee, Sung-Ki Jung
Evidence-based Complementary and Alternative Medicine. 2011; 2011: 1
[VIEW]
20 Acupuncture for Children with Autism Spectrum Disorders: A Systematic Review of Randomized Clinical Trials
Myeong Soo Lee, Tae-Young Choi, Byung-Cheul Shin, Edzard Ernst
Journal of Autism and Developmental Disorders. 2011;
[VIEW]
21 Yoga may help reduce blood pressure in HIV-infected adults with cardiovascular disease risk factors : Summaries/Commentaries
MS Lee, K Yarasheski
Focus on Alternative and Complementary Therapies. 2011; 16(2): 157
[VIEW]
22 Errors of alternative medicine: lessons for general practice
Edzard Ernst
European Journal of General Practice. 2011; : 1
[VIEW]
23 Moxibustion for rheumatic conditions: a systematic review and meta-analysis
Tae-Young Choi, Tae-Hun Kim, Jung Won Kang, Myeong Soo Lee, Edzard Ernst
Clinical Rheumatology. 2011;
[VIEW]
24 Acupuncture for Bell’s palsy: A systematic review and meta-analysis
Jong-In Kim, Myeong Soo Lee, Tae-Young Choi, Hyangsook Lee, Hyo-Jung Kwon
Chinese Journal of Integrative Medicine. 2011;
[VIEW]
25 Are we missing the æevidence-based boatæ?
Ernst, E.
Focus on Alternative and Complementary Therapies. 2010; 15(1): 1-2
[Pubmed]
26 A systematic review of homoeopathy for the treatment of fibromyalgia
Rachel Perry, Rohini Terry, Edzard Ernst
Clinical Rheumatology. 2010; 29(5): 457-464
[Pubmed]
27 Classic flaws in clinical CAM research : Comments
Edzard Ernst
Focus on Alternative and Complementary Therapies. 2010; 15(3): 207-209
[Pubmed]
28 Moxibustion for stroke rehabilitation: Systematic review
Lee, M.S., Shin, B.-C., Kim, J.-I., Han, C.-H., Ernst, E.
Stroke. 2010; 41(4): 817-820
[Pubmed]
29 The anarchy of evidence
Ernst, E.
British Journal of General Practice. 2010; 60(571): 135
[Pubmed]
30 From alternative medicine to æalternative scienceæ?
E. Ernst
Clinical Medicine. 2010; 10(5): 428
[Pubmed]
31 Complementary therapies for supportive cancer care
Edzard Ernst
Supportive Care in Cancer. 2010; 18(11): 1365
[VIEW]
32 The anarchy of evidence
Ernst, E.
British Journal of General Practice. 2010; 60(571): 135
[Pubmed]
33 Moxibustion for hypertension: A systematic review
Kim, J.-I., Choi, J.-Y., Lee, H., Lee, M.S., Ernst, E.
BMC Cardiovascular Disorders. 2010; 10: art 33
[Pubmed]
34 Is acupressure effective for stroke rehabilitation?
Lee, M.S.
Focus on Alternative and Complementary Therapies. 2009; 14(4): 312-313
[Pubmed]
35 Ginseng for cognitive function in alzheimeræs disease: A systematic review
Lee, M.S., Yang, E.J., Kim, J.-I., Ernst, E.
Journal of Alzheimeræs Disease. 2009; 18(2): 339-344
[Pubmed]
36 Therapeutic needling in osteopathic practice: An evidence-informed perspective
Rickards, L.D.
International Journal of Osteopathic Medicine. 2009; 12(1): 2-13
[Pubmed]
37 The application of transcutaneous electrical nerve stimulation to acupuncture points (Acu-TENS) for pain relief: a discussion of efficacy and potential mechanisms
Lesley Brown,Mike Holmes,Alice Jones
Physical Therapy Reviews. 2009; 14(2): 93
[Pubmed]
38 Qigong for type 2 diabetes care: A systematic review
Myeong Soo Lee,Kevin W. Chen,Tae-Young Choi,Edzard Ernst
Complementary Therapies in Medicine. 2009; 17(4): 236
[Pubmed]
39 Therapeutic needling in osteopathic practice: An evidence-informed perspective
Luke D. Rickards
International Journal of Osteopathic Medicine. 2009; 12(1): 2
[Pubmed]
40 Project MUSE Journals Perspectives in Biology and Medicine Volume 52, Number 2, Spring 2009 Complementary and Alternative Medicine: Between Evidence and Absurdity
Ernst, E.E. and Ernst, E.
Perspectives in Biology and Medicine. 2009; 52(2)
[Pubmed]
41 Cupping for Treating Pain: A Systematic Review
Kim, J.I. and Lee, M.S. and Lee, D.H. and Boddy, K. and Ernst, E.
Evidence-based Complementary and Alternative Medicine. 2009;
[Pubmed]
42 Complementary and Alternative Medicine
Ernst, E.
Perspectives in Biology and Medicine. 2009; 52(2): 289-303
[Pubmed]
43 Complex traditional chinese medicine for poststroke motor dysfunction: A systematic review
Junhua, Z., Menniti-Ippolito, F., Xiumei, G., Firenzuoli, F., Boli, Z., Massari, M., Hongcai, S., Raschetti, R.
Stroke. 2009; 40(8): 2797-2804
[Pubmed]
44 Qigong for type 2 diabetes care: A systematic review
Lee, M.S., Chen, K.W., Choi, T.-Y., Ernst, E.
Complementary Therapies in Medicine. 2009; 17(4): 236-242
[Pubmed]
45 How to mislead people with clinical trials
Ernst, E.
Biologist. 2008; 55(4): 215-217
[Pubmed]
46
. ;
[Pubmed]



 

Top
Print this article  Email this article
Previous article Next article
Online since 12th February '04
© 2004 - Journal of Postgraduate Medicine
Official Publication of the Staff Society of the Seth GS Medical College and KEM Hospital, Mumbai, India
Published by Medknow