Researching complementary and alternative therapies: frameworks for CAM evaluation



Survivorship Research Group, Centre for Medical Psychology and Evidence-based Decision-making, Central Clinical School, Sydney Medical School, New South Wales and Psycho-Oncology Cooperative Research Group, School of Psychology, Faculty of Science, University of Sydney, New South Wales


Complementary and alternative medicine encompasses a vast array of interventions aimed at improving the health of individuals. A large proportion of people use complementary and alternative medicine after a diagnosis of cancer and there is a need to understand these interventions, their efficacy and interaction with conventional medical treatments. The quality and rigour of complementary and alternative medicine research has been frequently criticised. Some deficiencies in reporting of complementary and alternative medicine research can be addressed by improved research design. Further improvements are possible through the use of frameworks for evaluation of complex or whole systems that clearly document the complementary and alternative medicine intervention, placing it in the context of treatment delivery and the philosophical assumptions underpinning the intervention. These frameworks provide guidance as to the staged and systematic development of complementary and alternative medicine interventions. Using these frameworks to document the complementary and alternative medicine intervention development supports the inclusion of the philosophical concepts at the core of the intervention. Doing so is likely to assist in the development of a shared language between complementary and alternative medicine researchers and evidence-based practitioners.

The term complementary and alternative medicine (CAM) has come to encompass a wide variety of treatments ranging from biological agents such as herbs, to the use of meditation, acupuncture, aromatherapy and hypnotherapy. CAM is used by many people after a diagnosis of cancer – around 80% of adults with cancer in the US and 65% in Australia.1-4 These interventions are increasingly incorporated into routine cancer care in Australia and other western countries.5,6

While less invasive forms of CAM, such as meditation, are thought not to interfere with conventional cancer treatments, there is evidence of potential interaction between some herbal medications and some cytotoxic drugs,7 via biochemical pathways.8 Other studies have reported that almost half of all people with cancer (47%) use nutritional supplements including antioxidants,9 yet there are data to suggest that taking antioxidants (including high dose vitamin C) concurrently with radiotherapy or chemotherapy may be harmful.10 Coupled with apparent gaps in knowledge of CAM among Australian oncologists,11 there is a real concern that CAM may reduce the effectiveness of conventional anti-cancer treatments and/or increase their side-effects.

Despite their high usage, few CAM have been evaluated in high quality clinical trials and the optimal approach to CAM evaluation continues to be debated in the literature. There is a clear need to evaluate and encourage the development of an evidence-base for CAM, supported by policy and funding changes in the US and Australia.12 The question now is how should CAM be evaluated?

Common criticisms of CAM research

While randomised control trials are recognised as the most rigorous approach to providing evidence of intervention efficacy, trials lacking methodological rigour may introduce bias or other confounders, consequently resulting in either under or overestimation of treatment effects.13 The Consolidated Standards of Reporting Trials (CONSORT) statement was developed to encourage clear and full reporting of randomised control trials that would enable readers to assess the methodological quality of a trial.14

Reviews of reporting of CAM trials suggest the reporting quality is poor, consequently making it difficult to interpret results and incorporate them into an evidence-based clinical practice.13,15 One review of 207 randomised control trials on homeopathy, herbal medicine and acupuncture found their methodological quality to be variable, with the majority having shortcomings in reporting, methodology or both.16 Most trials of CAM failed to adequately describe the random sequence generation, method of allocation concealment, number of participants dropping out from treatment and the reasons for drop out.16 

Inadequacies in the reporting of CAM trials may reflect inadequacies in the design of studies. Linde et al reported that blinding in herbal and acupuncture trials was less clearly successful than in homeopathy trials, while random allocation of treatments was less clearly performed in homeopathy trials. Additionally across all three areas, intention-to-treat analysis was rare.16

Inadequate design and reporting of CAM research needs to be considered in context with improvements in design and reporting of conventional medicines. Moher and colleagues reviewed the quality of reports of systematic reviews in paediatric CAM, finding that overall the reporting quality was similar between CAM and conventional therapy reviews.17 This finding, coupled with Linde et al’s report of higher quality reporting of CAM research in more recent publications of larger trials in Medline listed journals,16 suggests that as research design and reporting of conventional interventions improves, it is likely it improves in CAM research too.

Heart of the problem

Important differences in the philosophical approaches of CAM and western health practitioners exist; these differences, and the lack of a shared language, lie at the heart of disputes about CAM evaluation. 

The paradigm CAM practitioners work from differs to that of the western biomedical model, in which mind and body are identified as distinct entities and health systems are viewed mechanistically as cause and effect. CAM retains an integrated approach to mind and body. Aiming to deliver holistic care, CAM practitioners use concepts of disharmony or imbalance to diagnose problems and prescribe treatments, rather than symptoms of organ dysfunction. For some concepts fundamental to CAM practice, there are no equivalents within the western medical practice. Developing a shared language between the two approaches is key to conducting CAM research successfully.

Difficulty also arises in the translation of CAM terminology into scientific English. As CAM practice is based on concepts and terms that lack an equivalent translation or conceptualisation in western scientific thinking and language communication can be difficult. For example, in Chinese medicine the term ‘Qi’ is used. Translated as ‘life force energy’; it is a concept that has not been fully incorporated into western medical models. Qi is not measurable or quantifiable with current diagnostic tools and tests. In terms of treatment strategy, generally, conventional medicine focuses on treating individual organs, body parts, or body systems and predicting specific responses to treatments. CAM treatment emphasises emotion and balance in body function as a whole system, with the expectation that treatment is slow, and occurrs over extended durations without undesirable side-effects.

Understanding the philosophical differences between CAM and conventional western medicine, it is important to understand that the paradigm of illness and treatment used by CAM practitioners is a cornerstone in the development of high quality CAM research. The question is then which methodological approaches will enable the philosophical and language of CAM to be considered within the research design.

Complex systems approach

As randomised control trials have become established as the gold standard for evaluation of a single intervention, such as a drug, the methodology has been applied to other interventions with varying degrees of success. In 2000, the Medical Research Council (MRC) UK proposed a framework for the evaluation of complex interventions.18

Complex interventions  involve several components, or interconnecting parts, required for the intervention to function effectively.18 In a complex intervention, the individual components may act independently as well as inter-dependently in a way that make it difficult identify the ‘active ingredient’.   The evaluation of complex interventions requires researchers to define and develop interventions fully.19 Failing to do this commonly leads to difficulties in interpretation and implementation of research results. 

The framework proposed by the MRC equated the development and evaluation of complex interventions with the drug development process in that both have multiple and distinct phases.19 The phases proposed were:

  • Theoretical: identifies evidence to support hypotheses regarding a specific intervention
  • Modelling: aims to improve the understanding of intervention components and their relationships. This stage may involve qualitative evaluation, as well as surveys or case studies
  • Exploratory Trial: develops the optimum intervention and study design, including feasibility and acceptability of the intervention
  • Definitive randomised control trial: the design phase should include size, unit of randomisation, population and whether concealment is feasible
  • Long-term implementation – examines the intervention as it is implemented in practice.
Figure 1: Depicts the sequential phases of developing evidence for complex interventions Figure 2: Depicts an iterative approach to developing and evaluating complex interventions

Many CAMs are multi-faceted interventions comprising botanical ingredients, practitioners and their attributes, a personalised schedule of visits and specific belief systems about health and wellbeing.  Identifying the active component is difficult and effects of the intervention may be diminished if the intervention  is not delivered in its entirety. CAMs and their modes of delivery commonly meet the definition of complex interventions, however their focus is often healing rather than on the disease process.

In order to fully document and evaluate CAM interventions, it is important to be explicit about the fundamental philosophical assumptions underpinning the intervention. One approach to doing this systematically is a Whole Systems Research (WSR) approach.

Whole systems approach

A stepped approach to the development of CAM research, as suggested by Verhoef et al,20 is built on the idea of WSR, offering high likelihood of identifying and systematically evaluating potentially useful CAM. The concept of WSR incorporates both qualitative and quantitative research methods to study the effectiveness of an intervention, along with the process, context, outcomes and philosophy.20 Within this approach, acknowledgement of the philosophical foundations of a specific CAM and an emphasis on the healing process will support better theoretical models of how a specific CAM works and may lead to improved integration of CAM theories and conventional mechanistic approaches. It will certainly contribute to the development of better approaches to assessing CAM.20 

Verhoef and her CAM research team developed a guideline for CAM WSR research.20 The WSR CAM guideline recommends the integration of multiple designs and methods, including quantitative methods, qualitative research and case studies to develop innovative CAM designs, suitable to each CAM intervention. 

In studying a CAM not previously researched, it is suggested that small qualitative studies are the first step; these studies should be performed in patients with clearly documented medical and psycho-social histories and belief systems. The aim of initial studies is to develop an understanding of the possible effects of CAM (similar to case studies or series). Using the findings from qualitative studies guides the delivery and evaluation of an intervention and the appropriate populations. Determining an  appropriate target group for treatment is similar to approaches emerging for optimal use of targeted, biological agents in people with specific genetic mutations.

A three arm design for CAM studies (intervention, placebo control and usual care control), rather than the usual two arm design used in conventional medicine (intervention versus placebo control), has been recommended. Use of a three arm design will improve CAM evaluation by assessment of the CAM placebo effect. However, it will add significantly to the financial costs of the research project. Where blinding of treatments is not possible, this must be acknowledged and the inclusion of an attention-control group (in addition to standard care alone) needs to be considered. Improving the rationale for a CAM intervention with rigorous qualitative data and incorporating relevant control groups will result in a vastly improved evidence base for CAM and its interaction with conventional therapies.

CONSORT statement and CAM

The CONSORT statement, first developed and published in 1996,14 was revised 2001 and 2010.21,22 The statement aims to improve the clarity of reports of randomised control trial results, thereby reducing bias associated with poorly reported trials. While it is concerned with reporting what was done and found in research, it indirectly affects research design and conduct by encouraging investigators to consider what must be included to ensure transparent reporting of trial results and thereby minimising deficiencies in the research design.

Several extensions of the CONSORT statement have been developed to provide guidance on reporting of harms in randomised trials,23 herbal interventions,13,24 non-pharmacologic interventions,25 pragmatic trials,26 and trials of acupuncture. 27 During the development of studies evaluating CAM interventions, reviewing the CONSORT statement and relevant extensions is likely to assist investigators in clearly and comprehensively documenting the research and interventions they are seeking to address. Such transparent reporting will reduce the problem of bias resulting from poor reporting and will increase the reproducibility of the intervention.

Research teams

As discussed earlier, CAM research is frequently criticised for poor research design and limited reproducibility. To address these criticisms, the design of CAM research needs to be improved as discussed above. It is also important that the CAM research team be multidisciplinary, including CAM practitioners, conventional health professionals and academic researchers. The breadth of experience and skills of such multidisciplinary teams will help establish clear clinical questions, optimal research design, conduct and reporting. 

The logistics of delivering CAM therapy in the conventional hospital setting can be challenging and may limit the implementation of CAM supported by evidence. Training and motivating CAM research team members is essential in CAM research, as it is in trials of non-CAM therapies. Motivating research staff with CAM education may improve recruitment of participants for the CAM clinical trials when big sample sizes are required.


In order to support the integration of CAM interventions with conventional western medicine, it is essential to develop an evidence base for the use of CAM.  Frequently, the quality and rigour of CAM research is criticised, however, there is evidence of increasing quality of CAM research. Further improvements will be achieved through incorporation of the complex intervention framework or a WSR approach during the study design. Ensuring that CAM protocols comprehensively document the intervention, its context and philosophical assumptions, along with all aspects of the study design and the planned statistical analysis, will support clear and accurate reporting of the CAM study results. Clear reports of study results can be better appraised and integrated into routine clinical practice by clinicians.


1. Cassileth B, Deng G. Complementary and alternative therapies for cancer. The Oncologist. 2004;9:80-9.
2. Richardson MA, Sanders T, Palmer JL, Greisinger A, Singletary SE. Complementary/Alternative Medicine Use in a Comprehensive Cancer Center and the Implications for Oncology. J Clin Oncol. 2000:July 1;18(13):2505-14.
3. Oh B, Butow P, Mullan B, Clarke S, Tattersall M, Boyer M, et al. Patient-doctor communication: The use of complementary and alternative medicine by adult patients with cancer. J Soc Integr Oncol.. 2010;8(2):56-64.
4. Miller M, Boyer MJ, Butow PN, Gattellari M, Dunn SM, Childs A. The use of unproven methods of treatment by cancer patients. Frequency, expectations and cost. Support Care Cancer. 1998:Jul;6(4):337-47.
5. Oh B, Butow P, Mullan B, Clarke S, Beale P, Pavlakis N, et al. Impact of Medical Qigong on quality of life, fatigue, mood and inflammation in cancer patients: a randomized controlled trial. Ann Oncol. 2009:October 30;mdp479.
6. American Health Association. More hospitals offering CAM services, Health Forum reports; 2008 September 29, 2008.
7. Werneke U, Earl J, Seydel C, Horn O, Crichton P, Fannon D. Potential helath risks of complimentary alterative medicines in cancer patients. Br J Cancer.2004;90:408-13.
8. Etheridge A, Black S, Patel P, So J, Matthews J. An in vitro evaluation of cytochrome P450 inhibition and P-glycoprotien interaction with goldenseal, Gingko biloba, grape seed, milk thistle, and ginseng extracts and their constituents. Planta Medica. 2007;73(8):731-41.
9. Oh B. Exploration of Complimentary and Alternative Medicine by Patients with Cancer and Evaluation of Medical Qigong. In press 2009.
10. D’Andrea G. Use of Antioxidants during Chemotherapy and Radiotherapy should be avoided. CA: A Cancer Journal for Clinicians. 2005;55:319-21.
11. Newell S, Sanson-Fisher R. Australian oncologists’ self-reported knowledge and attitudes about non-traditional therapies used by cancer patients. Med J Aust. 2000;172:110-3.
12. National Health and Medical Research Council. Complementary and Alternative Medicine Special Call for Research Applications. Canberra; 2007 [Internet]. Available from:
13. Gagnier JJ, Boon H, Rochon P, Moherd D, Barnesg J, Bombardiera C, et al. Recommendations for reporting randomized controlled trials of herbal interventions: explanation and elaboration. J Clin Epidemiol. 2006;59:1134-49.
14. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996;276(8):637-9.
15. Linde K, ter Riet G, Hondras M, Vickers A, Saller R, Melchart D. Systematic reviews of complementary therapies – an annotated bibliography. Part 2: Herbal medicine. BMC Complement Altern Med. 2001;1(5).
16. Linde K, Jonas WB, Melchart D, Willich S. The methodological quality of ranomized controlled trials of homeopathy, herbal medicines and acupuncture. Int J Epidemiol. 2001;30(3):526-31.
17. Moher D, Soeken K, Sampson M, Ben-Porat L, Berman B. Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine. BMC Pediatr. 2002;2(3).
18. Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. In: Council MR, editor. London, United Kingdom: MRC; 2000.
19. Campbell M, Fitzpatrick R, Haines A, Kinmouth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321:694-6.
20. Verhoef MJ, Vanderheyden LC, Fønnebø V. A Whole Systems Research Approach to Cancer Care: Why Do We Need It and How Do We Get Started? Integr Cancer Ther. 2006;2(4):287-92.
21. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001:April 14;Volume 357:1191 – 4.
22. Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
23. Ioannidis JPA, Evans SJW, Gøtzsche PC, O’Neill RT, Altman DG, Schulz K, et al. Better Reporting of Harms in Randomized Trials: An Extension of the CONSORT Statement. Ann Intern Med. 2004;141:781-8.
24. Gagnier JJ, Boon H, Rochon P, Moher D, Barnes J, Bombardier C, et al. Reporting Randomized, Controlled Trials of Herbal Interventions: An Elaborated CONSORT Statement. Ann Intern Med. 2008;144:364-7.
25. Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, for the CONSORT Group. Methods and Processes of the CONSORT Group: Example of an Extension for Trials Assessing Nonpharmacologic Treatments. Ann Intern Med. 2008;148:W-60-W-6.
26. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.
27. MacPherson H, Altman DG, Hammerschlag R, Youping L, Taixiang W, White A, et al. Revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA): Extending the CONSORT Statement. PLoS Medicine. 2010;7(6):e1000261.

Be the first to know when a new issue is online. Subscribe today.