1 Shalkree

Effectiveness Of Psychotherapy Research Paper

College of William & Mary, P.O. Box 8795, Williamsburg, VA 23187-8795, USA

Received 18 June 2014; Revised 15 September 2014; Accepted 15 October 2014; Published 6 November 2014

Copyright © 2014 Glenn Shean. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Despite the growing influence of lists of empirically supported therapies (ESTs) there are concerns about the design and conduct of this body of research. These concerns include limitations inherent in the requirements of randomized control trials (RCTs) that favor those psychotherapies that define problems and outcome in terms of uncomplicated symptoms. Additional concerns have to do with criteria for patient selection, lack of integration with research on psychotherapy process and effectiveness studies, limited outcome criteria, and lack of controls for experimenter bias. RCT designs have an important place in outcome research; however it is important to recognize that these designs also place restrictions on what and how psychotherapy can be studied. There is a need for large scale psychotherapy outcome research based on designs that allow for inclusion of process variables and the study of the effects of those idiographic approaches to therapy that do not lend themselves to RCT designs. Interpretative phenomenological analysis may provide a useful method for the evaluation of the effectiveness of idiographic approaches to psychotherapy where outcome is not understood solely in terms of symptom reduction.

1. Introduction

There are several issues related to the external validity of lists of empirically supported psychotherapies (ESTs) that should be considered before broad policy changes are instituted based on this body of research. First, the design requirements of randomized control trial (RCT) studies do not allow for adequate understanding of the multifaceted nature of many mental health problems or the complex interplay of individual differences, interpersonal processes, and range of potential outcomes that are inherent in psychotherapy practice. Second, the EST literature is based on studies characterized by unacknowledged sampling issues, methodological constraints, researcher bias, and limited outcome criteria [1–3].

2. Background

A white paper issued by a Task Force of the American Psychological Association in 1995 stated that “in order to remain competitive in the mental health services marketplace psychologists must provide evidence of therapeutic efficacy in the form of lists of empirically supported psychotherapies (ESTs) for specific disorders” [4]. The APA paper specified RCTs as the ideal design for studies of psychotherapy efficacy, in which “a treatment is manualized and demonstrated to be more effective than other treatments or placebo” [4]. A large number of efficacy studies have been published that are the basis for lists of ESTs [5]. These lists are the basis for proposals to differentiate between “psychological treatments” and “psychotherapy” [6]. As proposed by Barlow and Carl, for example, [6], the term “psychological treatment” would be restricted to ESTs designed to address specific diagnoses within a health-care context. “Psychotherapy” in turn would be limited to references to processes and procedures designed to resolve issues such as interpersonal and relationship problems and personal growth concerns and administered outside of the health-care system. Given the broad policy and social implications of this proposal, it is worthwhile to more closely examine the external validity of lists of ESTs.

2.1. RCT Designs

Double-blind randomized control trial designs are widely considered to be the “gold standard” for treatment efficacy studies [7]. RCTs are the most direct way of determining whether a cause-effect relation exists between treatment and outcome [8]. RCT designs include the following features [8]. (1) The first item is random allocation of participants to intervention and control groups, in order to minimize allocation bias and to balance known and unknown prognostic factors in the assignment of treatments. (2) Both patients and clinicians must remain unaware of which treatment (e.g., placebo control or a previously tested treatment positive control study) was given until the study is completed. (3) All intervention groups are treated identically except for the experimental treatment. (4) Patients are analyzed within the group to which they were allocated. (5) Analyses are designed to estimate the size of the difference in predicted outcomes between intervention and control groups. Valid studies using RCT designs also require representative samples of adequate size to support planned statistical analyses and close matching of treatment and control group patients based on defined criteria.

2.1.1. RCT Advantages and Limitations

The practice of patient selection based on uncomplicated DSM based symptoms, common to most RCT design EST studies, allows for enhanced matching of treatment and control groups and clarity of outcome criteria. On the other hand, this requirement restricts the population studied to a portion of the individuals who seek psychotherapy services, since many patients seek psychotherapy services for issues that transcend specific symptoms, and comorbidity of symptoms is common. The use of therapy manuals in RCT studies allows for enhanced control over the experimental treatment under study but restricts the types of psychotherapeutic interventions that can be studied. The requirements of RCT designs in effect help to ensure the reliability of reported effects but limit the manner in which problems, treatments, and outcome criteria are defined, understood, and assessed [3, 9–11]. The effects of therapy moderators are assumed to be randomized in RCT design studies based on the assumption that randomization of group assignment takes into account potential effects of known moderators of therapeutic effects such as level of distress, social impairment, comorbidity, self-reflectiveness, readiness for change, openness to experience, level of engagement, ability to verbalize feelings, social support, and coping [1–3, 7, 10, 11]. Differences in dispositional traits that are correlated with DSM diagnoses (e.g., negative affect, submissiveness, low affect tolerance, and introversion) are also assumed to be randomized in RCT designs [2, 3]. The assumption that randomization of patient assignment controls for the effects of therapy moderators and process variables is valid only if study sample sizes are large enough to allow for adequate levels of power for valid statistical comparisons. Many EST studies are based on relatively small sample sizes, justified by power estimates based on questionable effect sizes reported in previous non-double-blind psychotherapy efficacy studies. These estimates are likely to be inflated by the effects of “therapist allegiance.”

In contrast to most manualized psychotherapies now listed as ESTs, idiographic psychotherapies emphasize the importance of process variables as moderators of therapeutic effectiveness and tailor the application of therapeutic strategies and techniques to the particular characteristics and circumstances of each individual patient. These approaches cannot be manualized in a manner that can be applied in the same manner across patients. The goals of idiographic therapies are typically described in terms of changes that are more in line with the concept of recovery than symptom reduction. Idiographic therapies emphasize the multifaceted origins and nature of mental disorders and understand recovery as an ongoing process that requires flexibility on the part of the therapists, as well as responsibility, commitment, and motivation to change on the part of the patient. The goals of idiographic therapies are not limited to symptom reduction and include enhanced connection of memories, emotions, changes in patterns of problematic experiences, and relating in ways that enable patients to live more satisfying and effective symptom-free lives, to have richer and more satisfying relationships, and to find new ways to become more effective agents in their lives [1, 7, 11, 12].

2.2. Issues Related to ESTs
2.2.1. Sampling Practices

The practice of limiting RCT psychotherapy studies to populations with a single diagnosis and few if any comorbid problems does not mirror the realities of most clinical practice. Individuals who meet single uncomplicated symptom criteria are estimated to represent no more than about 20% of the overall population seeking psychotherapy services [2, 3, 13]. Surveys indicate that at least one-third to one-half of people seeking mental health treatment do not meet criteria for any one diagnostic category [2, 14]. When specific symptoms are the initial focus of treatment about one-half of the patients add new target complaints or change their complaints over the course of treatment [10, 14]. It is estimated that between 40% and 70% of individuals who present for treatment in clinical practice settings are excluded from EST studies because they do not meet the restrictive criteria required by RCT designs [3, 14]. Finally, many EST studies are conducted in university affiliated related clinics [13]. Participants are often enrolled in studies of behavioral or cognitive-behavioral therapy (CBT). The individuals selected for the study are not likely to be blind to the treatment condition and likely were referred because of prior information about the lead therapist’s reputation. In short, the patient samples in university affiliated studies are not representative of the population at large nor are they randomly selected.

2.2.2. Length of Therapy

The average number of sessions for published EST studies is sixteen [2, 3]. The assumption that mental health problems can be resolved in an average of 16 or fewer sessions is not consistent with evidence for a significant psychotherapy dose-response relationship [3, 9] or with field studies of CBT for depressed patients that report an average of 69 sessions [3]. In contrast to the EST literature on treatment for depression, the research team conducted by the National Institute of Mental Health Treatment of Depression Collaborative Research Program concluded that 16 weeks of cognitive-behavioral or interpersonal therapy is insufficient for most patients to achieve full recovery and lasting remission [15]. Follow-up assessments of participants in the NIMH study, conducted at 6, 12, and 18 months after treatment, indicated that the percent of patients who were rated as recovered following the end of treatment and remained well during follow-up did not differ between the four treatment groups: about 30% for cognitive-behavioral therapy, interpersonal therapy, imipramine plus clinical management, and placebo plus clinical management. Long-term studies of patients treated by ESTs indicate high rates of relapse and seeking of alternative therapies [3, 15]. Evidence indicates that short-term therapies produce benefits more quickly than more individually focused therapies but long-term therapy is superior to short-term therapies at three- and five-year follow-up [16, 17].

2.2.3. Therapy Manuals

The use of manuals in RCT outcome studies is based on the assumption that psychotherapy can be formulated as a standardized set of procedures that can be applied across matched individuals without significant variation related to differences between either therapists or patients. There is logic in attempting to describe as systematically as possible the various principles and strategies involved in each therapeutic approach but it is important to acknowledge that this requirement limits the scope and range of problems that can be treated as well as the therapies that can be studied. The use of therapy manuals biases psychotherapy outcome research in favor of those therapies that can be operationalized [13, 14, 18]. Idiographic therapies do not lend themselves to manualization or standardized application across patients [19].

Finally, evidence indicates therapists within treatments vary significantly in outcomes so that some people respond to certain types of psychotherapy provided by some practitioners and others do not, and moderators of treatment effectiveness for one outcome indicator may not be the same as moderators of treatment for another [7, 11]. A more fruitful direction for future research would be to design studies that seek to identify the characteristics of patients that are associated with positive responses to different approaches to psychotherapy, with adequate safeguards for researcher bias.

2.2.4. Therapist Allegiance

It is often not practical to conduct true double-blind RCT studies of psychotherapy since it is difficult for both researchers and patients to remain blind to the therapeutic approaches included in the study. Efforts have been made to control for this source of error variance in non-double-blind RCT psychotherapy studies but evidence consistently indicates that the “same” therapy performs better in studies conducted by researchers who are committed to the approach than it does in studies conducted by others [20, 21]. Therapist allegiance accounts for about 69% of the variance in psychotherapy outcome studies [21]. Double-blind studies of psychotherapy are admittedly difficult to conduct; however, controls for the effects of therapy allegiance are not that difficult to implement. Luborsky and colleagues have published guidelines for how to control for the effects of therapist allegiance but these suggestions are ignored in the EST literature [20, 21]. The guidelines suggested by Luborsky et al. represent minimal standards for valid psychotherapy outcome research. (1) Comparative treatment studies should be conducted using raters of varied theoretical persuasion, and researchers who have minimal allegiances to the approaches studied should be involved in conducting the study. (2) Therapists for each treatment mode should be selected and supervised by those who represent the same treatment mode, and therapists should be assigned to each mode of treatment on the basis of ratings of their effectiveness. (3) Outcome criteria developed based on the input of the therapists of all persuasions under study and long-term functional follow-up evaluations should be conducted using consistent outcome criteria. (4) If all else fails efficacy studies include researcher/therapist allegiance as a variable in all analyses.

2.2.5. Idiographic Psychotherapy Effectiveness and Process

Numerous psychotherapy studies have been published that support the effectiveness of idiographic approaches but these studies are ignored in lists of ESTs [6, 22–24]. Meta-analytic analyses of comparative outcome studies also indicate that large differences in effectiveness are not observed among the major approaches to psychotherapy (e.g., cognitive-behavioral, interpersonal, behavioral activation, psychodynamic, problem solving, or social skills training) for many problems [25]. Researchers have identified practitioner related moderators that contribute to therapeutic outcome across approaches to therapy, including the quality of the therapeutic alliance, fostering a sense of hope and self-efficacy in the patient, and that encourage relevant emotional expression in combination with meaningful cognitive and emotional processing and integration [26–29]. Process variables, such as the quality of the therapeutic relationship estimated to account for between 15 and 30% of the variance in outcome; therapist techniques about 15%; expectancy (hope), therapist credibility about 15%; and environmental and patient characteristics (e.g., readiness for change, openness, engagement, active participation, and ability to verbalize feelings) about 40%, are ignored in RCT studies [12, 29].

3. Conclusion

The use of RCT designs for the evaluation of psychotherapy effectiveness works best for those symptom focused approaches to psychotherapy that can be manualized and applied in essentially the same manner across patients who share a certain uncomplicated diagnosis. RCT designs are ideal when cause and effect relationships can be clearly defined, interventions can be applied in a similar manner across patients, and outcome criteria are applicable across all study participants. RCT designed studies have resulted in useful lists of ESTs that can be applied effectively to address specific DSM symptoms. On the other hand the requirements of RCT designs limit how problems are understood, psychotherapy can be practiced, and outcome can be defined and assessed. These design requirements disadvantage idiographic approaches to psychotherapy. Research is needed on the effectiveness of idiographic approaches to psychotherapy that are characterized by broad descriptions of patients’ problems and allow for flexibility in how therapeutic processes are described, applied, and assessed in individual cases and indicators of recovery are defined and measured over extended follow-up intervals [19]. Not everyone seeking psychotherapy services is interested to, able to, or motivated to commit to the demands and relative ambiguity of idiographic therapies, just as not everyone seeks targeted approaches to therapy for the resolution of specific uncomplicated symptoms. RCT design requirements do not allow for research on the effectiveness of idiographic psychotherapies, nor do they allow researchers to address the full range of problems for which people seek psychotherapy.

Advances in the application of phenomenological psychology appear to hold promise for the study of the effects of idiographic psychotherapies [30]. Phenomenological psychology is rooted in attempts to construct a philosophical science of consciousness that includes both hermeneutics (theory of interpretation) and symbolic interactionism (posits that individual meanings are of central concern and are only accessible through an interpretive process) [30, 31]. Methods have been developed that allow for reliable idiographic understandings of patients, their social realities, and what it means to live with symptoms in a particular situation. IPA allows us to situate and validate observations developed through discourse, subjective engagement, and careful and sympathetic attention to how another person cares for and about certain “things,” how they are involved and/or distressed by them, and how they are involved and tied up with certain aspects of their lives [31]. These methods lack the potential for clear delineation of cause and effect relationships in outcome research but hold promise for research on the effects of idiographic therapies in particular because they allow for (1) descriptions of the patients’ concerns and cares and their orientation to the world and (2) contextualization of phenomenological claims within the patient’s social, cultural, and physical environments in ways that make sense of the mutually constitutive relationship between “person” and “world” within a psychological framework that describes what this means to this person in this context [31].

IPA, in combination with methodological safeguards against researcher bias, can allow for more appropriate ways of evaluating idiographic approaches to psychotherapy and broaden our understanding of psychotherapy process while minimizing the risks of ungrounded theoretical relativism and authoritarianism. RCT designs are the “gold standard” for determining cause and effect relationships in studies of treatment effectiveness but it is important to recognize that not all approaches to psychotherapy can be adequately described, understood, and evaluated within RCT design requirements. Alternative research approaches are necessary in order to evaluate the effectiveness of idiographic approaches to psychotherapy.

Conflict of Interests

The author declares that there is no conflict of interests regarding the publication of this paper.

How effective is psychotherapy, really?

Though Sigmund Freud and his followers regularly published case histories describing how patients could be cured of their psychological problems through therapy, exactly how treatment effectiveness could be measured was rarely discussed.   Even as psychotherapy became more popular and new kinds of treatment were developed, few therapists seemed interested in putting actual science into psychotherapy to bring it up to the same standard as regular medicine.

Then along came Hans Eysenck and psychotherapy would never be quite the same again…

A German-born psychologist who spent most of his life in Great Britain,  Eysenck is likely best remembered for his work on intelligence, personality, and genetics.   He is less well-known as a major critic of psychoanalysis despite this being where he most influential.   In 1952,  he published one of the first comprehensive reviews looking at research studies evaluating psychoanalysis and other psychotherapies.   Based on the limited number of studies available, Eysenck concluded that only 44% of psychoanalysis patients receiving Freudian treatment showed any kind of real improvement over a five-year period.  Though the numbers were somewhat higher for eclectic therapy (64%), the improvement rate was still well below the 72% rate for patients treated in hospitals or by general practitioners. 

While only focusing on treatment of neurosis, Eysenck argued for an inverse correlation between psychotherapy and recovery.  In other words, the more psychotherapy you got, the less likely you were to recover.   As Eysenck put it, his data “fail to prove that psychotherapy, Freudian or otherwise, facilitates the recovery of neurotic patients.”   Despite the controversy that erupted, Eysenck followed up his 1952 paper with a 1961 study having essentially the same results.

Not that this challenge went unanswered for long.   Hans Herman Strupp, a German-born psychoanalyst who was one of the pioneers of psychotherapy research published his own critique of Eysenck’s research in 1963.    Strupp’s critique, titled “The Outcome Problem in Psychotherapy Revisited”, blasted Eysenck’s methodology and conclusions and offered his own recommendations for effective psychotherapy research.   Predictably, Eysenck wrote his own rebuttal to Strupp's comments which he branded as “irrelevant, incompetent and immaterial.”    This was then followed by Strupp’s response in which he pointed out that “The controversy about the value of psychotherapy has been with us for some time, and it is not likely to be resolved by argument or counterargument.”

In the 50 years since the famous Eysenck-Strupp debate, there have been thousands of research studies looking at psychotherapy effectiveness though many of the basic questions raised by Eysenck and Strupp are still difficult to answer.  While we know more about what does or doesn’t work as far as psychotherapy is concerned, the question of how to measure treatment outcome is still controversial.   

According to Bruce E. Wampold, professor and chair of counseling psychology at the University of Madison-Wisconsin, the impact of the Eysenck-Strupp debate continues to influence psychotherapy researchas well as public perception of psychotherapists in general.   In a recent article published in the journal Psychotherapy, Wampold weighs the current legacy of the debate using a “spaghetti western” model presenting his opinions on both Eysenck and Strupp in terms of “the good, the bad, and the ugly” effect they have had on psychotherapy research.

The Good

Hans Eysenck’s research forced psychotherapists to ask awkward questions largely neglected up to that time.  During the 1950s, psychotherapy usually meant Freudian psychoanalysis although other types of therapy were available by then as well.  Since psychoanalysis was mainly practiced by medical doctors, the public assumed that it was established medical procedure and just as valid anything else supported by medicine.   By introducing psychotherapy outcome research, both Eysenck and Strupp set the standard that later researchers needed to follow.  Simply assuming that psychotherapy worked based on case histories was no longer enough and outcome studies using randomized trials with control groups became more common.

The Bad

Eysenck and Strupp were not impartial over what worked in psychotherapy.  They both had a bias since Eysenck was a behaviour therapist while Strupp was a psychoanalyst.  Hans Eysenck’s research were largely intended to show that behaviour therapy was more effective than psychoanalysis.  Still, he never put  behaviour therapy to the test the way he had with psychoanalysis and eclectic psychotherapy.  Strupp’s criticism of Eysenck’s research was largely aimed at defending psychoanalysis while arguing against Eysenck’s own bias.

The Ugly

Part of the “ugly” surrounding the psychotherapy debate was the tendency for therapists to define what was healthy rather than letting the patient decide.  Well into the 1960s, both Strupp and Eysenck advocated “treating” homosexuality to make them fit their own view of “normal” behaviour.   Though reparative therapy is no longer accepted, any type of treatment that lets the therapist impose his or her own view of “normal” on a patient, whether the patient agrees or not, is going to be a problem. 

As for Eysenck, he seemed determined to show that behaviour therapy was superior to all other forms of therapy and often lashed out at critics with personal attacks, no matter how solid their facts were.  Even up to his death in 1997, he largely dismissed the numerous research studies showing that other forms of psychotherapy could be as effective as behavior therapy.  

As Bruce E. Wampold points out, the fifty years since the Eysenck-Strupp debate has seen a tremendous boom in research articles examining the psychotherapy process and  measuring treatment effectiveness.  Despite this progress, there continues to be strong resistance to the type of evidence-based treatment that Eysenck and Strupp both endorsed.   Finding which treatments work best for specific conditions such as posttraumatic stress disorder has led to new debates that can be just as bitter as the debates of the past.    

Despite these disagreements, there is still a general consensus that some treatments based on scientific psychological principles are better than others.   Psychotherapists have also come to recognize that actual evidence is more important than opinion in determining what works and what does not.   For that reason, therapists need to examine the therapy process and treatment outcome carefully to make sure patients were actually helped by the treatment they received.    

Ethical therapists should do no less.

Leave a Comment


Your email address will not be published. Required fields are marked *