Outcome Data

"The data are always friendly" - Irene Elkin  

Following are a sample of my clinical outcome data since 2011.  I support the "open data" movement towards making clinical outcome data available to consumers, payers, policy-makers, and researchers (after identifying information has been removed from the data).

A detailed description of the outcome data process is available in Atlantic Monthly article "What your therapist doesn't know".

Below is a poster that describes a representative sample of my psychotherapy outcome data from two years of practice at the University of Alaska Fairbanks Student Health and Counseling Center.  This poster was created by Data Visualization expert Andy Kirk


Tony Rousmaniere Therapy Outcome Data

2011 Private Practice data (San Francisco & Palo Alto, California)

  • Total # of clients in dataset: 42
  • Avg # of sessions: 8.58
  • Total # of clients with more than one session: 38 (90%)
  • Single session clients: 4 (10%)
  • Start in clinical range (of clients with 2+ sessions): 25 (66%)

Clinical Outcomes

  • Ended Year in Clinical Change: 22 (88% of clients starting in clinical range)
  • Ended Year in Clinical Recovery: 20 (80% of clients starting in clinical range)
  • Ended Year in Clinical Deterioration: 2 (8% of clients starting in clinical range)
  • Average change in ORS score: 7.52  (clients starting in clinical range)
  • Cohen's D effect size for clients starting in clinical range: 1.22.  (Pre-tx std dev=6.14, mean intake ORS=22.27, mean last session ORS=29.79, n=38)

Duration of Treatment

  • Avg # of sessions for low distress clients: 5.27
  • Avg # of sessions for high distress clients: 10.74
  • Avg # of sessions for clinical change to occur: 3.27

  

2012-2013 University Counseling Center data  (University of Alaska Fairbanks)

Outcome Questionnaire-45 (OQ-45) data

  • Total # of clients in dataset: 102
  • Avg # of sessions:  4.60
  • Total # of clients with more than one session: 73 (72%)
  • Single session clients: 29 (28%)
  • Start in clinical range (of clients with 2+ sessions): 53 (73%)

Clinical Outcomes

  • Ended Year in Clinical Change: 24 (45% of clients starting in clinical range)
  • Ended Year in Clinical Recovery:  13 (25% of clients starting in clinical range)
  • Ended Year in Clinical Deterioration: 3 (6% of clients starting in clinical range) 
  • Average change in OQ score: 15.06  (clients starting in clinical range)
  • Cohen's D effect size for clients starting in clinical range: .82.  (pre-tx std dev=18.25, mean intake OQ=86.30, mean last session OQ=71.24, n=53)           

Duration of Treatment

  • Avg # of sessions for low distress clients: 4.24
  • Avg # of sessions for high distress clients: 4.78


Outcome Rating Scale (ORS) data

  • Total # of clients in dataset:  89
  • Avg # of sessions: 5.37
  • Total # of clients with more than one session: 65 (73%)
  • Single session clients: 24 (27%)
  • Start in clinical range (of clients with 2+ sessions): 42 (65%)

Clinical Outcomes

  • Ended Year in Clinical Change: 20 (48% of clients starting in clinical range)
  • Ended Year in Clinical Recovery:  11 (26% of clients starting in clinical range)
  • Ended Year in Clinical Deterioration: 4 (6% of clients starting in clinical range)   
  • Average change in ORS score: 4.84  (clients starting in clinical range)
  • Cohen's D effect size for clients starting in clinical range: .97.  (pre-tx std dev=4.98, mean intake ORS=15.63, mean last session ORS=20.47, n=42)

Duration of Treatment

  • Avg # of sessions for low distress clients: 5.61
  • Avg # of sessions for high distress clients: 5.24
  • Avg # of sessions for clinical change to occur: 2.60
  • Maximum # of sessions for clinical change: 5.00


2014-2015:  I do not have data for these years because my counseling center at the University of Alaska Fairbanks switched to a different outcome measurement program (the Counseling Center Assessment of Psychological Symptoms - CCAPS) which does not facilitate analysis of outcome data at the clinician level.  We started using the CCAPS because (a) it integrated better with our electronic records software, (b) was easier for counselors and medical staff to use for consultations, and (c) was designed specifically for university counseling centers.  


2015-2018 Private Practice data (Seattle) - Preliminary

I moved to Seattle and started a private practice in the Fall of 2015. Following are the data from my practice from Fall 2015 to Winter 2018. The dataset is small because I work part-time as a therapist, some clients have opted-out of having their data included in this analysis, and I do not include data from clients who start treatment outside the "clinical range" (see below for an explanation.) These results should be considered preliminary because there are not sufficient data for a statistically reliable gauge of my work at my Seattle private practice. Ideally, there should be data from at least 30 clients to provide statistical reliability. These data will be updated annually.

Notes: Since 2015 my primary professional focus has gradually shifted towards more writing, research, leading workshops and webinars, and providing consultation for agencies and training programs. This means I have worked with fewer new clients and had less time for my own professional development (clinical training, expert consultation, and deliberate practice.) Research from the science of expertise suggests it is reasonable to expect that my clinical effectiveness will plateau and decline over time given this career trajectory.

The outcome measure I used from 2015-2016 was the Behavioral Health Measure-20 (BHM-20). Since January 2016 I have used the Outcome Questionnaire-45.

  • Total # of clients with data: 42
  • Clients who started in clinical range (below "clinical cutoff"): 28 (66%) 
  • Clients for whom outcome data is not reliable due to starting outside clinical range: 14 (33%)
  • Range of sessions: 1 to over 100 
  • Clients with more than one session: 37 (88%) 
  • Clients who did not return after first session: 5 (12%) 
  • Average # of sessions: 10 (estimated)

Clinical Outcomes

  • Clients who met the BHM-20 or OQ-45 "recovery" or “improved” benchmark: 17 (77% of the 22 clients who started in clinical range and had two or more sessions.)
  • Clients who showed no clinically significant change on the BHM-20 or OQ-45: 6 (27% of clients starting in clinical range and had two or more sessions.)
  • Clients who deteriorated: 1 (5% of the 22 clients who started in clinical range and had two or more sessions.)

Outcome Measurement and Client Response

I have used multiple outcome measures:

  • The Outcome Rating System (ORS), an ultra-brief measure with four items that is free to use.  Information about the ORS is here and here.   Clients completed the ORS at the beginning of each session, on a laptop computer or with paper/pencil.  It takes about 30 to 60 seconds to complete. 
  • The Outcome Questionnaire-45 (OQ-45).  Information about the OQ-45 is here.  Clients completed the OQ at the beginning of each session on a computer in the waiting room.  It takes about 3-5 minutes to complete.
  • The Behavioral Health Measure (BHM 20).  Information about the BHM-20 is available here.  Clients completed the BHM-20 at the beginning of each session, on a laptop computer or with paper/pencil.  It takes about 2 minutes to complete.

Consent for tracking outcomes is obtained at the beginning of treatment and is voluntary for clients.  Consent includes agreement for the outcomes to be used for research and publication, after being de-identified.

Based on my subjective judgment, I consider the ORS, BHM-20, and OQ-45 results to be valid for about 75% of my cases.  For some clients I think the measures under-represent the amount of positive change, especially for clients who started treatment with less clinical distress.  For some clients the measures over-represent the amount of positive change, in my judgement.  

Client Impressions of Outcome Measures

When I asked clients their thoughts about the outcome measurement, most clients were curious about their results but said they gave more weight to my subjective clinical impression of their case.  A few clients were skeptical of the validity of standardized outcome measurements, and a few clients refused to complete the measures because of this concern.  Some clinicians have reported that they do not collect outcome data because of concerns about harming the therapeutic alliance.  I did not find this to be the case, but rather found that discussing outcomes with some clients strengthened the therapeutic alliance.  Research suggests that clinician attitudes towards the outcome measures strongly affect clients’ reactions to the measures.

What About Change for "Low-Distress" Clients?

There are no outcome measures that are good at assessing the amount of change in psychotherapy for clients who start therapy with low levels of distress, termed the "clinical range" in the psychotherapy research literature.  For this reason, I do not have reliable quantitative data on the amount of change for low-distress clients.  

Limitations of the Data and Outcome Measurement

It is difficult to assess how good these results are, because there are no available data from therapists working in the same locations where I worked.  Likewise, it is not possible to compare outcome data from my different work locations because the client populations are different.  Another validity concern is that I do not have follow-up outcome data, so I do not know if gains from treatment are lasting. 

Therapy outcome measures are not perfect.  Multiple clients report negative change on outcome measures when in fact their symptoms improved.  Likewise, paper forms can never describe the rich complexities of growth and healing in therapy.  Given these limitations, I still think it is important for therapists to track their clients' outcomes with unbiased measures.  While the measures should never be used alone for clinical decisions, they are one important source of information regarding client progress or deterioration, and are can be helpful for assessing overall therapist effectiveness, when combined with a range of other assessment tools.


Popular outcome measures

Online forums and communities that discuss outcome measures and provide training

Software to help measure outcomes



Books on measuring psychotherapy effectiveness

Feedback-Informed Treatment in Practice: Reaching for Excellence. (2017). Prescott, D., Miller, S.D., & Maeschalck, S. Washington, DC: APA Books.

Outcomes Management, Reimbursement, and the Future of Psychotherapy. G.S. (Jeb) Brown & Takuya Minami.  (2010).  In: The heart and soul of change: Delivering what works in therapy (2nd ed.). Duncan, Miller, Wampold, & Hubble,  (Eds.); pp. 267-297. Washington, DC, US: American Psychological Association.

The heroic client: A revolutionary way to improve effectiveness through client-directed, outcome-informed therapy. (2004). Duncan,B.L., Miller, S.D., & Sparks, J.A.  San Francisco: Jossey-Bass.

Developing and Delivering Practice-Based Evidence: A Guide for the Psychological Therapies. (2010). Barkham, M., Hardy, G.E., & Mellor-Clark, J.  New York, NY:  Wiley.