Evaluation of social, mental health, and public health programs are challenging, for many reasons. Our readings for this week discuss and highlight some of these challenges. Which challenges are especially important to address, in evaluating these programs?
SPECIAL ARTICLE
New trends in assessing the outcomes of mental
health interventions
GRAHAM THORNICROFT, MIKE SLADE
King’s College London, Health Service and Population Research Department, Institute of Psychiatry, Denmark Hill, London SE5 8AF, UK
Assessing the outcomes of interventions in mental health care is both important and challenging. The aim of this paper is to advance the
field of outcomes research by proposing a taxonomy of the decisions that clinicians and researchers need to consider when evaluating
outcomes. Our taxonomy has eight components, framed as decisions: Whose outcome will be considered? Which scientific stage is being
investigated? What outcome domain(s) matter? What level of assessment will be used? Will clinical and/or recovery outcomes be assessed?
Whose perspective will be considered? Will deficits and/or strengths be the focus? Will invariant or individualized measures be preferred?
We propose a future focus on understanding what matters most to people using mental health services, and on the use of measures rated by
service users as the primary approach to evaluating outcome.
Key words: Outcomes research, assessment measures, mental health services research
(World Psychiatry 2014;13:118–124)
Assessing the outcomes of interventions in mental health
care is both important and challenging. It is important
because producing significant outcomes, i.e., health gains
attributable to an intervention (1), is the main goal of mental
health services. Other important attributes of an intervention, such as accessibility, acceptability, efficiency and costeffectiveness, need only to be considered where the intervention produces significant outcomes. Assessing outcomes
is also challenging, because choosing methods, outcome
domains and outcome measures all involve the balancing of
conceptual, ethical and clinical considerations (2,3).
The aim of this paper is to propose a taxonomy of the
decisions that clinicians and researchers need to consider
when evaluating outcomes. Our taxonomy has eight components, each of which involves making explicit underpinning assumptions. We therefore frame these components as
decisions.
DECISION 1: WHOSE OUTCOME?
It might be thought that the outcome for the patient is of
primary importance, but the needs of at least three other
stakeholder groups also need to be considered.
First, the patient’s informal carers – their friends and family – often have substantially more contact with the patient
than mental health staff, which may have powerful consequences. A UK study estimated that 4.8% of carers had
terminated employment and 15.5% took a mean of 12.5
days off work per year as a result of their carer role (4).
Carers also provide emotional and practical support that
otherwise would be required from mental health services –
estimated for people with schizophrenia being looked after
by family as involving 5.6 hours per day (5). Informal carers
will have their own perspectives on valued outcomes for the
patient and for themselves (6). Evaluating the impact of
their caring role on their mental and physical health may be
118
a cost-effective element of an evaluation strategy, and carerfocussed measures are available (7-9).
Second, the wellbeing of staff may be considered as an
outcome, for two specific reasons. The clinical rationale is
that there is now robust evidence (10) that “parallel processes” exist in mental health services – the experience of
staff within the system influences how they work with people using services. If services are for example to promote
hope and empowerment, then staff need to experience hope
and empowerment in their work role. The economic rationale is that providing mental health services is expensive,
and the primary cost driver is human resources. A workforce with low morale, high sickness rates and poor
performance is an inefficient investment (11). For both
these reasons, monitoring outcomes such as staff wellbeing
and morale might be justified.
Finally, members of the public largely fund mental health
systems in many countries, whether directly through health
insurance or indirectly through taxation. Therefore the public have a legitimate interest in the return on their investment (12). Outcomes of interest to the public might include
reductions in antisocial behaviour such as aggressive begging, or of “odd” behaviour such as shouting at voices.
For the remainder of this paper, we focus on outcomes
for patients.
DECISION 2: WHICH SCIENTIFIC STAGE?
Developed as an analogue of the phases of pharmacological product development, recent formulations have considered phases of complex psychosocial interventions (13), or
more generally the phases of the translational medicine continuum (14), shown in Figure 1.
In this conceptualization, outcomes may vary according
to the scientific stage of enquiry. At Phase 0 (scientific discovery) the key outcome may be the formulation of a new
World Psychiatry 13:2 – June 2014
Figure 1 Phases of the translational medicine continuum
aetiological pathway or candidate risk factor for a disorder.
At Phase 1 (early human trials) the key outcomes of interest
are likely to be tolerability of the new intervention and
dose-response. In Phase 2 (early clinical trials) the critical
outcome issue is estimating the effect size of the intervention – in other words, is there an early indication that the
intervention is effective, and how effective? At Phase 3 (late
clinical trials) the key outcome is more specific – exactly
how effective is the intervention among routine populations
with the condition of interest? Finally, in Phase 4 (implementation) the outcome focus becomes more pragmatic,
namely how far can Phase 2 and 3 benefits be replicated in
routine clinical practice?
DECISION 3: WHAT OUTCOME DOMAIN?
An outcome domain is a conceptually distinct component of outcome. A systematic review identified seven categories of outcome domains: wellbeing, cognition/emotion,
behaviour, physical health, interpersonal, societal and services (15). Choosing the outcome domain or domains to
evaluate should be a separate and prior decision to choosing the outcome measure (16). In our experience, this distinction is often not maintained, with the more common
starting point being identification of measures. Conflating
the choice of outcome domain with choosing the outcome
measure leads to three problems: inconsistency, unimportance and unfairness.
When evaluating outcome for a specific intervention, it
remains common to under-specify the intended mechanisms of action and the causal pathway from intervention to
outcome. This is despite the scientific consensus that evaluation should involve identification of the theory base for an
intervention (13). The absence of a testable model means
that the rationale for the choice of outcome domain cannot
be stated, and so the choice is likely to reflect current clinical assumptions about “what matters”. There has been a
general movement from using service data (e.g., readmission
rates) towards clinical outcomes (e.g., symptomatology) and
more recently towards health-related quality of life as clinical endpoints in outcomes research. However, the result is
that the evidence base remains poor for interventions targeting some important outcome domains, such as hope and
empowerment (17). Explicitly identifying, with a rationale,
the choice of outcome domain will increase theoretical
coherence between intervention and outcome.
When evaluating whole-system interventions such as service models or when introducing outcome assessment into
routine clinical settings, consideration of outcome domains
is also an important first step. Mental health systems need to
meet many goals, including patient benefit, harm minimization, public protection, and value-for-money. The choice of
outcome domains sends a clear message about the relative
balance of these goals, and hence is an influence on organizational culture. It is one means by which an organization
communicates what is important, in other words its “core
business”.
Finally, explicitly identifying the outcome domain reduces
the extent to which success is unfairly judged in relation to
aspects of a patient’s life which are outside the control of
the mental health service. Social determinants of mental
ill-health such as poverty and social inequality are wellestablished (18), and as services in general cannot influence wider social determinants, measures of health-related
quality of life may be insufficiently sensitive. An alternative
approach is to identify more proximal outcome domains,
such as symptomatology or recovery support.
DECISION 4: WHAT LEVEL OF ASSESSMENT?
It is important to have clarity about the level of assessment, from the individual intra-psychic level (e.g., symptoms) through the inter-personal and immediate social
environment (e.g., carers, social networks) to the broader
environmental level (e.g., stigma). For example, in relation
to interventions related to stigma and discrimination associated with mental illness, one can assess the outcomes of
a national programme, such as the Time to Change campaign
119
in England (19), using whole population surveys (20), or in
terms of sub-populations such as journalists (21), or in terms
of the outcomes rated by individual mental health services
users (22), all of which can be seen as valid and indeed complementary outcome measures.
DECISION 5: CLINICAL OR RECOVERY OUTCOMES?
Outcome assessment internationally remains primarily
focussed on traditional clinical outcomes such as symptomatology, social disability and service use (e.g., admission
rates). The four most commonly used measures assess
social disability (Health of the Nation Outcome Scale,
HONOS (23)), symptoms (Clinical Outcomes in Routine
Evaluation – Outcome Measure, CORE-OM (24); Outcome Questionnaire-45, OQ-45 (25)), and needs (Camberwell Assessment of Need, CAN (26)). These are mandated
for national or large regional use in Australia (27), Canada
(28), England (29), Netherlands (30) and New Zealand (1).
These measures have in common that they assess clinical
outcomes.
Internationally there is an emerging consensus that services should be recovery-oriented (31). Recovery has been
defined as “a deeply personal, unique process of changing
one’s attitudes, values, feelings, goals, skills, and/or roles”
and “a way of living a satisfying, hopeful, and contributing
life even within the limitations caused by illness” (32). International best practice is emerging (33), and it is becoming
clear that organizational transformation is needed to develop a recovery orientation (34). Some dimensions of transformation include a greater emphasis on the biomedical ethical imperative of promoting autonomy (35), a changed
workforce (36), a greater emphasis on patient choice, and,
most relevantly, different goals of mental health care. The
challenge is summarized by Repper and Perkins (37):
“Traditional yardsticks of success – the alleviation of symptoms and discharge from services – are replaced by questions about whether people are able to do the things that
give their lives meaning and purpose, irrespective of whether their problems continue and whether or not they continue to need help and support”. The challenge is to measure
recovery as an outcome in a way which is both aggregatable
and meaningful.
How might this be done? A systematic review of recovery
frameworks identified five key recovery processes: connectedness (social inclusion, community integration), hope and
optimism, development of a positive identity, meaningfulness
in life, and empowerment – the CHIME Framework (38). If
the goal of a mental health system is to promote recovery,
then these recovery outcomes are the appropriate domains to
target. New measures are becoming available (39).
One proposal is that outcome assessment should measure valued social roles which reinforce social identity, and
individual goals which contribute to personal identity (40).
120
Valued social roles include employee, partner, family
member, friend, citizen, free (i.e. non-detained) person, etc.
Their value is relatively invariant – most (but of course not
all) people want a job, a relationship, contact with their family, some close friends, the ability to exercise citizenship
rights such as voting, not to be held in hospital or prison,
etc. Assessment tends to be quantitative and dichotomous
(or at least on an ordinal scale, such as unemployed – voluntary work – part-time work – full-time work), and hence
easy to aggregate with little loss of meaning. The primary
advantage of these outcome measures is that they are based
on normal social values, and so avoid illness-related lowering of expectations either by staff, in an effort to be realistic,
or by patients with internalized stigmatizing beliefs about
what they can expect in life (41). Since most valued social
roles occur outside the mental health system, they orientate
the actions of the service towards increasing integration and
participation by the person into his/her social environment,
rather than encouraging a decontextualized and servicefocussed view of the person. Their primary disadvantage is
their invariance – some people get along very well in life
without friends, or a partner, or a job.
Individual goals differ from person to person. No standardized measure will have items such as “swim with
dolphins” or any of the other idiosyncratic goals individuals
set and attain on their recovery journey. Any attempt to
squeeze personal identity into predefined boxes can be justifiably criticized for its loss of meaning. This does not, of
course, mean that personal goals should not be included in
outcome evaluation – they remain central, despite the difficulties in assessing individual goal attainment. Rather, as
McNamara (42) put it, “the challenge is to make the important measurable, not the measurable important”. So, an
overall outcome evaluation strategy might measure two
things. First, objective quality of life indicators, such as adequacy of housing, friendship, safety, employment and close
relationships. Second, progress towards personal goals.
DECISION 6: WHOSE PERSPECTIVE?
Assuming that the outcome for the patient is the main
focus, the question remains of whose perspective is used.
Two perspectives have primarily been used to evaluate
outcome.
First, and in our view most central, is the patient perspective. An emerging distinction in relation to patient-rated
measures is between assessment oriented towards the experience of using mental health services and systems – patient
rated experience measures (PREMs) – and assessment capturing direct health gain – patient rated outcome measures
(PROMs), especially using patient-generated PROMs (PGPROMs) (43). A range of PROMs exist, spanning both clinical and recovery outcomes (44-47). The development of
PREMs is earlier stage, and has primarily focussed on satisfaction and experience of care. The main limitation of
World Psychiatry 13:2 – June 2014
PREMs is that they may reduce the focus on a “life beyond
illness”. People who use mental health services long-term
can live in a “virtual institution”, in which key aspects of
identity (social network, sense of self, housing, etc.) are all
indexed on the mental illness (48). PREMs such as satisfaction are a normative judgment influenced by the person’s
reference group, so in people using mental health services
positive ratings may be obtained because of an atypical reference group. This vulnerability of PREMs to being rated
positively because of lowered expectations means that mental health systems should as far as possible evaluate success
using outcome rather than experience measures.
Second, and perhaps the traditional focus in mental
health systems, is the perspective of the clinician (49-51).
Staff-rated measures exist for most outcome domains. This
perspective has been called the “objective” assessment and
the patient rating called the “subjective” assessment, but in
fact staff assessments are themselves prone to bias due for
example to professional training (52), and some studies have
found patient rather than staff assessments to be more
reliable (53). The reality is that both staff and patient
perspectives are influenced by a range of factors, and both
provide useful and complementary information on outcome. The relative balance given to the two perspectives
should be based on scientific, ethical, professional and pragmatic considerations.
We now consider outcome assessment from the patient’s
perspective.
DECISION 7: DEFICITS OR STRENGTHS?
The World Health Organization (WHO) declares that
health is “a state of complete physical, mental and social
well-being and not merely the absence of disease or
infirmity” (54). However, creating health-oriented rather
than illness-oriented services has proved rather more difficult than the clarity of this declaration would suggest. In relation to outcome, the substantial majority of measures used in
research and practice are focussed on mental illness – they
assess amelioration of undesirable experiences such as symptoms or cognitive problems, reduction in risk factors such as
stress, or attainment of an adequate level of functioning.
Very few assess mental health, such as the use of strengths
(55), the development of protective factors such as resilience,
or the attainment of positive wellbeing (56).
Some argue that mental health is a distinct construct. The
Complete State Model of Mental Health posits that mental
health and mental illness lie on orthogonal spectrums (57).
People with mental illness range from those who are
“floundering” (when mental health is absent), through those
experiencing moderate mental health, to those who are
“struggling” (when mental health is present) as they work
towards “flourishing” (high mental health, low mental illness). Epidemiological studies of adults (n53,032) (57) and
adolescents (n51,234) (58) confirm that mental health and
mental illness according to these definitions co-exist in the
general population.
An alternative view is that mental health is better understood as lying on a single spectrum with positive mental
health at one end and negative mental health at the other.
Measures based on this approach have been developed,
such as the Subjective Happiness Scale (59), which includes
items such as “Compared to most of my peers, I consider
myself. . .”, with Likert ratings from 1 (less happy) to 7 (more
happy). Some wellbeing measures include only positively
worded items, which are compatible with both understandings of mental health. Examples include the WHO-5 WellBeing Index (e.g., “I have felt cheerful and in good spirits”)
(60) and the Warwick-Edinburgh Mental Well-Being Scale
(WEMWBS) (e.g., “I’ve been feeling useful”) (61).
More generally, Vaillant identifies six models of mental
health (62). The first model, being “above normal”, relates to
superior functioning in a wide range of activities, such that
life’s problems never get out of hand. The goal of the second
model, positive psychology, is intervention to maximize
positive qualities, such as self-efficacy. A recent systematic
review has identified indicators of wellbeing in psychosis
(63), which are now being used to inform a new intervention based on positive psychology principles (64). The third
model involves maturity, shown by attainment of developmental tasks such as identity, intimacy, generativity and
integrity (65). The fourth model is emotional or social intelligence, i.e., the ability to read other people’s emotions. Subjective wellbeing, i.e., the experience of positive mental
health, is the fifth model, and the last model is resilience,
which is linked to the adaptive value of coping mechanisms.
DECISION 8: INVARIANT VERSUS INDIVIDUALIZED?
Normal practice in outcome evaluation is to use standardized measures, for which key psychometric criteria
have been established as adequate. More recent attention
has enlarged the focus from the usual reliability and validity
concerns to also consider feasibility and clinical relevance
(66). However, standardized measures have the feature of
invariance – the same outcome domain is assessed for each
patient. The advantage of this approach is that it allows
statements about the impact of an intervention or service on
a specific outcome domain, such as symptomatology. The
emerging important disadvantage, however, is that the outcome domain may or may not be important to the patient.
We learn from the reports of people who use services that
recovery is very individual, varying greatly from person to
person (67). As well as symptomatic or functional improvement, the tipping point towards starting to develop an identity as a person in recovery can be developing a supportive
relationship with a mental health worker who treated them
as a person not a patient (68), or non-clinical changes such
as spiritual growth (69). This variation highlights the need
for caution about viewing improvement in any single domain
121
as universally important, and the outcome evaluation challenge of capturing individual importance using standardized
assessments.
One technology that can be used to personalize evaluation is goal attainment scaling (GAS) (70). This approach
involves patients prospectively identifying a personallyimportant goal and associated progress indicators on a typically five-point scale, using these indicators to assess progress at outcome evaluation, and then standardizing the
results to allow aggregation. GAS has been used to identify
and then evaluate a valued outcome in randomized controlled trials, primarily in rehabilitation medicine (71) and
with older adults (72). Two systematic reviews have investigated this use of GAS. In relation to pharmacy practice, the
conclusion was that GAS demonstrated high reliability, variable validity, excellent responsiveness, and was a useful
methodology for evaluating effectiveness (73). In relation to
physical rehabilitation, GAS was described as a sound measure, with reliability and sensitivity needing further investigation (74). Concern has been raised about sensitivity to
subtle changes, responsiveness, inter-rater reliability, validity (content and construct), scaling non-linearity and lack of
uni-dimensionality (75). For example, agreement on progress between a patient’s therapist and an independent assessor is low (76). To these concerns, we would add that
administration burden can be high, and that the GAS score
(77) is not intuitive to interpret.
A new approach to address some of these issues is called
the Personal Primary Outcome (PPO) list. Designed for use
in randomized controlled trials and other evaluations, the
PPO list comprises several outcome domains, each of which
is (invisibly) linked to a relevant standardized outcome
measure. At baseline, the patient chooses the outcome
domain that is most closely linked to his/her goal in using
mental health services, and then he/she completes the associated measure. The measure is re-administered at followup. The PPO list approach is currently being evaluated as a
methodology for trials (78).
A second approach is to develop a standardized measure
where items are selected according to patient preference.
An example is the INSPIRE measure (downloadable at
www.researchintorecovery.com/inspire) of recovery support, where for each item about support from a mental
health worker, respondents are first asked if the item matters to them, and only if it does are they asked to rate support from the worker (47). The INSPIRE score therefore
reflects the respondent’s preferences, yet produces a quantitative score which can be used for monitoring change over
time or can be aggregated with the scores of others.
CONCLUSION: WHAT OUTCOMES REALLY MATTER
FOR SERVICE USERS?
Perhaps the most important insight developed in the last
decade is that it is the point of view of the patient or service
122
user that is the most important in deciding which outcomes
to assess, and in making the actual outcome ratings. We
know, for example, that quality of life is not closely related
to users’ needs as rated by staff of mental health services, but
is closely associated with unmet needs as rated by service
users (79,80). It follows that the emerging literature reporting service user views on measures (44,81) and developing
new measures (82) is of paramount importance. New measures, such as the Recovery Star (83), can be independently
evaluated (84) and incorporated into clinical practice (85).
An additional advantage of making service user rated outcomes a principal focus is that it side-steps the issue that has
bedevilled services in recent years, namely how to incentivize staff to make frequent, complete and valid outcome ratings on a long-term sustainable basis.
If we were unwisely to try to predict the central issues in
mental health outcome measurement over the next decade,
then we propose a relentless attention to the detail of what
matters most to service users, as rated by service users.
Acknowledgements
The authors receive support from the National Institute
for Health Research (NIHR) Biomedical Research Centre
for Mental Health at South London and Maudsley NHS
Foundation Trust and King’s College London. The views
expressed are those of the authors and not necessarily
those of the NHS, NIHR, or the Department of Health.
The two authors contributed equally to this work.
References
1. Trauer T. Outcome measurement in mental health: theory and
practice. Cambridge: Cambridge University Press, 2010.
2. Thornicroft G, Tansella M. Mental health outcome measures, 3rd
ed. London: Royal College of Psychiatrists, 2010.
3. Thornicroft G, Becker T, Knapp M et al. International outcomes
in mental health. Quality of life, needs, service satisfaction, costs
and impact on carers. London: Gaskell, 2006.
4. Mangalore R, Knapp M. Cost of schizophrenia in England. J Ment
Health Policy 2007;10:23-41.
5. Andrews A, Knapp M, McCrone P et al. Effective interventions in
schizophrenia: the economic case. A report prepared for the
Schizophrenia Commission. London: Rethink Mental Illness,
2012.
6. Wallcraft J, Amering M, Freidin J et al. Partnerships for better
mental health worldwide: WPA recommendations on best practices in working with service users and family carers. World Psychiatry 2011;10:229-36.
7. Harvey K, Catty J, Langman A et al. A review of instruments
developed to measure outcomes for carers of people with mental
health problems. Acta Psychiatr Scand 2008;117:164-76.
8. Vella S-L, Pai N. The measurement of burden of care in serious
mental illness: a qualitative review. Aust N Z J Psychiatry 2013;
47:222-34.
9. Dare A, Hardy J, Burgess P et al. Carer outcome measurement in
mental health services: scoping the field. Brisbane: Australian
Mental Health Outcomes and Classification Network, 2008.
World Psychiatry 13:2 – June 2014
10. Maben J, Peccei R, Adams M et al. Exploring the relationship
between patients’ experiences of care and the influence of staff
motivation, affect and wellbeing. Final report. Southampton:
NIHR Service Delivery and Organization Programme, 2012.
11. Kakuma R, Minas H, van Ginneken N et al. Human resources for
mental health care: current situation and strategies for action.
Lancet 2011;378:1654-63.
12. Saxena S, Thornicroft G, Knapp M et al. Resources for mental
health: scarcity, inequity, and inefficiency. Lancet 2007;370:87889.
13. Campbell NC, Murray E, Darbyshire J et al. Designing and evaluating complex interventions to improve health care. BMJ 2007;
334:455-9.
14. Thornicroft G, Lempp H, Tansella M. The place of implementation science in the translational medicine continuum. Psychol
Med 2011;41:2015-21.
15. Slade M. What outcomes to measure in routine mental health
services, and how to assess them: a systematic review. Aust N Z J
Psychiatry 2002;36:743-53.
16. Slade M. Routine outcome assessment in mental health services.
Psychol Med 2002;32:1339-43.
17. Slade M, Leamy M, Bird V et al. Mental health services and
recovery. In: Thornicroft G, Ruggeri M, Goldberg D (eds).
Improving mental health care: the global challenge. Chichester:
Wiley, 2013:40-56.
18. Tew J, Ramon S, Slade M et al. Social factors and recovery from
mental health difficulties: a review of the evidence. Br J Soc Work
2012;42:443-60.
19. Henderson C, Thornicroft G. Stigma and discrimination in mental illness: Time to Change. Lancet 2009;373:1928-30.
20. Evans-Lacko S, Malcolm E, West K et al. Influence of Time to
Change’s social marketing interventions on stigma in England
2009–2011. Br J Psychiatry 2013;202 (Suppl. 55):s77-88.
21. Thornicroft A, Goulden R, Shefer G et al. Newspaper coverage of
mental illness in England 2008–2011. Br J Psychiatry 2013;
202(Suppl. 55):s64-9.
22. Corker E, Hamilton S, Henderson C et al. Experiences of discrimination among people using mental health services in England
2008–2011. Br J Psychiatry 2013;202(Suppl. 55):s58-63.
23. Wing JK, Beevor AS, Curtis RH et al. Health of the Nation Outcome Scales (HoNOS). Research and development. Br J Psychiatry 1998;172:11-8.
24. Evans C, Mellor-Clark J, Margison F et al. CORE: Clinical Outcomes in Routine Evaluation. J Ment Health 2000;9:247-55.
25. Lambert MJ, Burlingame GM, Umphress V et al. The reliability
and validity of the Outcome Questionnaire. Clin Psychol Psychother 1996;3:249-58.
26. Phelan M, Slade M, Thornicroft G et al. The Camberwell Assessment of Need: the validity and reliability of an instrument to
assess the needs of people with severe mental illness. Br J Psychiatry 1995;167:589-95.
27. Australian Mental Health Outcomes Classification Network.
Adult national outcomes & casemix collection standard reports,
1st ed, Version 1.1. Brisbane: Australian Mental Health Outcomes Classification Network, 2005.
28. Slade M. An evidence-based approach to routine outcome assessment. Adv Psychiatr Treat 2012;18:180-2.
29. HM Government. No health without mental health. Delivering
better mental health outcomes for people of all ages. London:
Department of Health, 2011.
30. Drukker M, Bak M, Campo JA et al. The cumulative needs for
care monitor: a unique monitoring system in the south of the
Netherlands. Soc Psychiatry Psychiatr Epidemiol 2010;45:47585.
31. Slade M, Amering M, Farkas M et al. Uses and abuses of recovery:
Implementing recovery-oriented practices in mental health systems. World Psychiatry 2014;13:12-20.
32. Anthony WA. Recovery from mental illness: the guiding vision of
the mental health system in the 1990s. Psychosoc Rehabil J 1993;
16:11-23.
33. Le Boutillier C, Leamy M, Bird VJ et al. What does recovery
mean in practice? A qualitative analysis of international
recovery-oriented practice guidance. Psychiatr Serv 2011;62:
1470-6.
34. Farkas M, Gagne C, Anthony W et al. Implementing recovery oriented evidence based programs: identifying the critical dimensions. Community Ment Health J 2005;41:141-58.
35. Beauchamp T, Childress J. Principles of biomedical ethics. Oxford:
Oxford University Press, 2001.
36. Shepherd G, Boardman J, Burns M. Implementing recovery. A
methodology for organisation change. London: Sainsbury Centre
for Mental Health, 2010.
37. Perkins R, Repper J. Social inclusion and recovery. London: Bailliere Tindall, 2003.
38. Leamy M, Bird V, Le Boutillier C et al. Conceptual framework for
personal recovery in mental health: systematic review and narrative synthesis. Br J Psychiatry 2011;199:445-52.
39. Bellack A, Drapalski A. Issues and developments on the consumer recovery construct. World Psychiatry 2012;11:156-60.
40. Slade M. Measuring recovery in mental health services. Isr J Psychiatry Relat Sci 2010;47:206-12.
41. Lasalvia A, Zoppei S, Van Bortel T et al. Global pattern of experienced and anticipated discrimination reported by people with
major depressive disorder: a cross-sectional survey. Lancet 2013;
381:55-62.
42. Slade M, Priebe S. Choosing methods in mental health research.
Hove: Routledge, 2006.
43. Trujols J, Portella MJ, Iraurgi I et al. Patient-reported outcome
measures: are they patient-generated, patient-centred or patientvalued? J Ment Health 2013;22:555-62.
44. Law H, Morrison A, Byrne R et al. Recovery from psychosis: a
user informed review of self-report instruments for measuring
recovery. J Ment Health 2012;21:193-208.
45. Burgess P, Pirkis J, Coombs T et al. Assessing the value of existing
recovery measures for routine use in Australian mental health
services. Aust N Z J Psychiatry 2011;45:267-80.
46. Shanks V, Williams J, Leamy M et al. Measures of personal recovery: a systematic review. Psychiatr Serv 2013;64:974-80.
47. Williams J, Leamy M, Bird V et al. Measures of the recovery orientation of mental health services: systematic review. Soc Psychiatry Psychiatr Epidemiol 2012;47:1827-35.
48. Priebe S, Turner T. Reinstitutionalisation in mental health care.
BMJ 2003;326:175-6.
49. Johnson D. A compendium of psychosocial measures. New York:
Springer, 2010.
50. National Institute for Mental Health in England. Outcomes compendium. Birmingham: National Institute for Mental Health in
England, 2008.
51. Rush AJ, First M, Blacker D. Handbook of psychiatric measures,
2nd ed. Washington: American Psychiatric Press Inc, 2007.
52. Slade M. Assessing the needs of the severely mentally ill: cultural
and professional differences. Int J Soc Psychiatry 1996;42:1-9.
53. Slade M, Leese M, Taylor R et al. The association between needs
and quality of life in an epidemiologically representative sample
of people with psychosis. Acta Psychiatr Scand 1999;100:149-57.
54. World Health Organization. Promoting mental health. Concepts,
emerging evidence, practice. Geneva: World Health Organization, 2004.
55. Bird V, Le Boutillier C, Leamy M et al. Assessing the strengths of
mental health service users – systematic review. Psychol Assess
2012;24:1024-33.
56. Slade M. Mental illness and well-being: the central importance of
positive psychology and recovery approaches. BMC Health Serv
Res 2010;10:26.
123
57. Keyes CLM. Mental illness and/or mental health? Investigating
axioms of the complete state model of health. J Consult Clin Psychol 2005;73:539-48.
58. Keyes CLM. Mental health in adolescence: is America’s youth
flourishing? Am J Orthopsychiatry 2006;76:395-402.
59. Lyubomirsky S, Lepper HS. A measure of subjective happiness:
preliminary reliability and construct validation. Social Indicators
Res 1999;46:137-55.
60. Bech P, Olsen RL, Kjoller M et al. Measuring well-being rather
than the absence of distress symptoms: a comparison of the SF-36
Mental Health subscale and the WHO-Five Well-Being Scale. Int
J Methods Psychiatr Res 2003;12:85-91.
61. Tennant R, Hiller L, Fishwick R et al. The Warwick-Edinburgh
Mental Well-being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes 2007;5:63.
62. Vaillant GE. Positive mental health: is there a cross-cultural definition? World Psychiatry 2012;11:93-9.
63. Schrank B, Bird V, Tylee A et al. Conceptualising and measuring
the well-being of people with psychosis: systematic review and
narrative synthesis. Soc Sci Med 2013;92:9-21.
64. Schrank B, Riches S, Bird V et al. A conceptual framework for
improving well-being in people with a diagnosis of psychosis. Epidemiol Psychiatr Sci (in press).
65. Erikson E. Identity: youth and crisis. New York: Norton, 1968.
66. Slade M, Thornicroft G, Glover G. The feasibility of routine outcome measures in mental health. Soc Psychiatry Psychiatr Epidemiol 1999;34:243-9.
67. Leamy M, Bird V, Le Boutillier C et al. A conceptual framework
for personal recovery in mental health: systematic review and
narrative synthesis. Br J Psychiatry 2011;199:445-52.
68. Topor A, Borg M, Mezzina R et al. Others: the role of family,
friends, and professionals in the recovery process. Am J Psychiatr
Rehabil 2006;9:17-37.
69. Pargament KI, Lomax JW. Understanding and addressing religion
among people with mental illness. World Psychiatry 2013;12:26-32.
70. Kiresuk TJ, Smith A, Cardillo JE. Goal attainment scaling: applications, theory and measurement. Hillsdale: Lawrence Erlbaum,
1994.
71. Turner-Stokes L, Baguley IJ, De Graaff S et al. Goal attainment
scaling in the evaluation of treatment of upper limb spasticity
with botulinum toxin: a secondary analysis from a double-blind
124
placebo-controlled randomized clinical trial. J Rehabil Med 2010;
42:81-9.
72. Rockwood K, Fay S, Jarrett P et al. Effect of galantamine on verbal repetition in AD: a secondary analysis of the VISTA trial.
Neurology 2007;68:1116-21.
73. Vu M, Law A. Goal-attainment scaling: a review and applications
to pharmacy practice. Res Social Adm Pharm 2012;8:102-21.
74. Hurn J, Kneebone I, Cropley M. Goal setting as an outcome measure: a systematic review. Clin Rehabil 2006;20:756-72.
75. Steenbeek D. Goal attainment scaling in paediatric rehabilitation
practice. PhD thesis, 2010.
76. Thamar J, Bovend’Eerdt T, Dawes H et al. Agreement between
two different scoring procedures for goal attainment scaling is
low. J Rehabil Med 2011;43:46-9.
77. Rockwood K, Howlett S, Stadnyk K et al. Responsiveness of goal
attainment scaling in a randomized controlled trial of comprehensive geriatric assessment. J Clin Epidemiol 2003;56:736-43.
78. Slade M, Bird V, Le Boutillier C et al. REFOCUS Trial: protocol
for a cluster randomised controlled trial of a pro-recovery intervention within community based mental health teams. BMC Psychiatry 2011;11:185.
79. Slade M, Leese M, Cahill S et al. Patient-rated mental health
needs and quality of life improvement. Br J Psychiatry 2005;187:
256-61.
80. Slade M, Leese M, Ruggeri M et al. Does meeting needs improve
quality of life? Psychother Psychosom 2004;73:183-9.
81. Crawford MJ, Robotham D, Thana L et al. Selecting outcome
measures in mental health: the views of service users. J Ment
Health 2011;20:336-46.
82. Evans J, Rose D, Flach C et al. VOICE: developing a new measure
of service users’ perceptions of inpatient care, using a participatory methodology. J Ment Health 2012;21:57-71.
83. McKeith J, Burns S, Onyemaechi I et al. The Recovery Star: user
guide, 2nd ed. London: Mental Health Providers Forum, 2010.
84. Killaspy H, White S, Taylor TL et al. Psychometric properties of
the Mental Health Recovery Star. Br J Psychiatry 2012;201:65-70.
85. Dickens G, Weleminsky J, Onifade Y et al. Recovery star: validating user recovery. The Psychiatrist 2012;36:45-50.
DOI 10.1002/wps.20114
World Psychiatry 13:2 – June 2014