Healthcare Delivery System Quality Improvement Plan Paper

QUALITY ASSESSMENT AT TRIDENT INTERNATIONAL
HOSPITAL
PHUONG-TRANG PHAM
DATE: 6/12/2022
OBJECTIVES
 To discuss the quality assessment and why it is necessary.
 To discuss the quality assessment tools and its importance.
 To discuss the two outcome assessment tools that can help to assess the quality of ED at TIH.
 To discuss the importance of patient experience and timeliness of care and its impact on the wait time at ED at
TIH.
INTRODUCTION
 I have been selected to serve on the Trident International Hospital (TIH) Evaluation Team. In recent months, the
emergency department (ER) of TIH has been rebuilt and reorganized.
 The team will conduct a quality evaluation to determine if the improvements have reduced the amount of time
patients wait in the emergency room.
 I was asked to do research and build two tools for evaluating outcomes that might be utilized in this process as
part of the early development of the assessment strategy. This will be carried out as part of the evaluation
process.
QUALITY ASSESSMENT
 The ultimate objective of quality assessment (QA) in primary care is to promote the continual improvement of
medical care through the evaluation of structure, process, and outcome indicators.
 QA works toward this objective by reviewing the delivery of many components of patient care.
 Improving the entire quality and performance of the healthcare environment can result in more dependable, cost-
efficient, and durable healthcare processes for providers. This can assist them in achieving their objective of
enhancing care delivery and patient outcomes (Park et al., 2016).
QUALITY ASSESSMENT TOOLS OR OUTCOME MEASUREMENT FOR
TIH
 The purpose of measuring, reporting, and comparing healthcare outcomes is to achieve the Quadruple Aim, which
consists of the following:

Enhance the overall care provided to patients.

The health of individuals should be improved.

Reduce the cost of providing each individual with medical care.

Assist doctors and employees in avoiding occupational fatigue (Naeini, 2019).
 The survey to obtain patient experience and to check timeliness of patient care are the two outcome
measurement tools that will be used for TIH to assess the improvement in wait time in the ED.
PATIENT EXPERIENCE
 Patient-reported outcome measures, are a type of patient experience outcome measure. It evaluate the patient’s
experience and view of healthcare.
 This information can give us a more accurate way to measure how satisfied patients are, as well as current
information that can be used to improve local services and help us respond faster to problems we find.
 A patient may be asked to fill out a satisfaction survey about their care, with answers ranging from scale of 1 to 5.
 The patient satisfaction can also be used as a way to measure how well care is getting better (Health Catalyst,
2018).
CONTINUED…
 The survey can be conducted at TIH to assess the patient satisfaction level or to assess the patient experience
level for the healthcare services at ED Emergency department.
 The wait time at ED is reduced by bringing new improvements. And to assess this quality improvement, the use of
patient experience is the best outcome measurement tool to asses the quality of services provided at ED.
 Patients can better inform us about their experience at ED or if the waiting time has been reduced or not.
TIMELINESS OF CARE
 Using timeliness of care outcome measures, patient accessibility to care is evaluated. Overcrowding in the
emergency department has been associated with an increase in;

inpatient mortality

length of hospital stay, and

patient costs (Health Catalyst, 2018).
CONTINUED…
 Timeliness of care is the best outcome measurement tool that can be used for assessing the quality improvement
at ED of TIH.
 By using this tool, we can check if the timeliness of care is effective or not. Mostly, the hospitals uses this tool for
assessing the timeliness of care outcome at ED.
 TIH management has implemented some changes to reduce the wait time at ED and it can be measured by
measuring the timeliness of care factor.
OUTCOME OF CHANGES
 The outcome of changes brought by the management of TIH for the ED is measured by using two outcome
measurement tools which includes the survey of patient experience and timeliness of care.
 Both the tools are useful to assess if the wait time has actually reduced at ED by the management of TIH.
 Without these tools, the outcome of this change cannot be measured properly.
CONCLUSION
 The use outcome measurement tools are effective to asses the quality of healthcare in healthcare organizations.
 The use of survey of patient experience and timeliness of care are the two best outcome measurement tools to
asses the reduction in wait time at ED of TIH.
REFERENCES
 Park, G. W., Kim, Y., Park, K., & Agarwal, A. (2016). Patient-centric quality assessment framework for healthcare
services. Technological Forecasting and Social Change, 113, 468-474.
 Naeini, E. K., Azimi, I., Rahmani, A. M., Liljeberg, P., & Dutt, N. (2019). A real-time PPG quality assessment approach
for healthcare Internet-of-Things. Procedia Computer Science, 151, 551-558.
 Health Catalyst. (2018, October 31). The Top 7 Healthcare Outcomes Measures. Health Catalyst;
www.healthcatalyst.com. https://www.healthcatalyst.com/insights/top-7-healthcare-outcome-measures
QUALITY ASSESSMENT AT TRIDENT INTERNATIONAL
HOSPITAL
DATE: 6/12/2022
OBJECTIVES
 To discuss the quality assessment and why it is necessary.
 To discuss the quality assessment tools and its importance.
 To discuss the two outcome assessment tools that can help to assess the quality of ED at TIH.
 To discuss the importance of patient experience and timeliness of care and its impact on the wait time at ED at
TIH.
INTRODUCTION
 I have been selected to serve on the Trident International Hospital (TIH) Evaluation Team. In recent months, the
emergency department (ER) of TIH has been rebuilt and reorganized.
 The team will conduct a quality evaluation to determine if the improvements have reduced the amount of time
patients wait in the emergency room.
 I was asked to do research and build two tools for evaluating outcomes that might be utilized in this process as
part of the early development of the assessment strategy. This will be carried out as part of the evaluation
process.
QUALITY ASSESSMENT
 The ultimate objective of quality assessment (QA) in primary care is to promote the continual improvement of
medical care through the evaluation of structure, process, and outcome indicators.
 QA works toward this objective by reviewing the delivery of many components of patient care.
 Improving the entire quality and performance of the healthcare environment can result in more dependable, cost-
efficient, and durable healthcare processes for providers. This can assist them in achieving their objective of
enhancing care delivery and patient outcomes (Park et al., 2016).
QUALITY ASSESSMENT TOOLS OR OUTCOME MEASUREMENT FOR
TIH
 The purpose of measuring, reporting, and comparing healthcare outcomes is to achieve the Quadruple Aim, which
consists of the following:

Enhance the overall care provided to patients.

The health of individuals should be improved.

Reduce the cost of providing each individual with medical care.

Assist doctors and employees in avoiding occupational fatigue (Naeini, 2019).
 The survey to obtain patient experience and to check timeliness of patient care are the two outcome
measurement tools that will be used for TIH to assess the improvement in wait time in the ED.
PATIENT EXPERIENCE
 Patient-reported outcome measures, are a type of patient experience outcome measure. It evaluate the patient’s
experience and view of healthcare.
 This information can give us a more accurate way to measure how satisfied patients are, as well as current
information that can be used to improve local services and help us respond faster to problems we find.
 A patient may be asked to fill out a satisfaction survey about their care, with answers ranging from scale of 1 to 5.
 The patient satisfaction can also be used as a way to measure how well care is getting better (Health Catalyst,
2018).
CONTINUED…
 The survey can be conducted at TIH to assess the patient satisfaction level or to assess the patient experience
level for the healthcare services at ED Emergency department.
 The wait time at ED is reduced by bringing new improvements. And to assess this quality improvement, the use of
patient experience is the best outcome measurement tool to asses the quality of services provided at ED.
 Patients can better inform us about their experience at ED or if the waiting time has been reduced or not.
TIMELINESS OF CARE
 Using timeliness of care outcome measures, patient accessibility to care is evaluated. Overcrowding in the
emergency department has been associated with an increase in;

inpatient mortality

length of hospital stay, and

patient costs (Health Catalyst, 2018).
CONTINUED…
 Timeliness of care is the best outcome measurement tool that can be used for assessing the quality improvement
at ED of TIH.
 By using this tool, we can check if the timeliness of care is effective or not. Mostly, the hospitals uses this tool for
assessing the timeliness of care outcome at ED.
 TIH management has implemented some changes to reduce the wait time at ED and it can be measured by
measuring the timeliness of care factor.
OUTCOME OF CHANGES
 The outcome of changes brought by the management of TIH for the ED is measured by using two outcome
measurement tools which includes the survey of patient experience and timeliness of care.
 Both the tools are useful to assess if the wait time has actually reduced at ED by the management of TIH.
 Without these tools, the outcome of this change cannot be measured properly.
CONCLUSION
 The use outcome measurement tools are effective to asses the quality of healthcare in healthcare organizations.
 The use of survey of patient experience and timeliness of care are the two best outcome measurement tools to
asses the reduction in wait time at ED of TIH.
REFERENCES
 Park, G. W., Kim, Y., Park, K., & Agarwal, A. (2016). Patient-centric quality assessment framework for healthcare
services. Te c hno lo g ic al F o re c as ting and S o c ial C hang e , 113, 468-474.
 Naeini, E. K., Azimi, I., Rahmani, A. M., Liljeberg, P., & Dutt, N. (2019). A real-time PPG quality assessment approach
for healthcare Internet-of-Things. P ro c e dia C o mpute r S c ie nc e , 151, 551-558.
 Health Catalyst. (2018, October 31). The To p 7 He althc are Outc o me s Me as ure s . Health Catalyst;
www.healthcatalyst.com. https://www.healthcatalyst.com/insights/top-7-healthcare-outcome-measures
Week 3 Assignment Overview
Healthcare Delivery Systems and Quality
Assessing the quality of care is not new in health care. The rapid growth
of the managed-care industry in the U.S. has led to a variety of
definitions and perceptions of quality (Cowing, Davino-Ramaya,
Ramaya, & Szmerekovsky, 2009). Today, several well-established
agencies and organizations address improving health care quality and
patient safety through a process known as continuous quality
improvement (Cowing, Davino-Ramaya, Ramaya, & Szmerekovsky,
2009).
To fully understand relevant performance measures in the health care
industry, it is essential to consider each of the three key players in the
service triad: the health care organization, the clinician (team of
physicians, nurses, medical assistants, and office staff), and the patient.
Each of these three entities has a unique, but interrelated, perspective
on the needs associated with health care performance (Cowing, DavinoRamaya, Ramaya, & Szmerekovsky, 2009).
Healthcare Delivery System Goals
According to Buchbinder & Shanks (2017), sustainable and effective
healthcare systems work to balance these three goals:
1. Appropriate access to necessary healthcare services.
2. Assurance of quality workforce, services, and institutions.
3. Acceptable cost to society.
Why is Quality Improvement Important to Healthcare?
•
•
•
Better monitoring and assessment of performance and using findings
to guide improvement activities.
Applying organizational strategies, methods, and tools for continued
quality improvement.
Identifying and analyzing problems and generating solutions.
Performance Measurement
Performance measurement is the ongoing monitoring and reporting of
program accomplishments, particularly progress toward pre-established
goals (Buchbinder & Shanks, 2017). Performance measures may
address the type or level of program activities conducted (process), the
direct products and services delivered by a program (outputs), or the
results of those products and services (outcomes) (Buchbinder &
Shanks, 2017).
(Buchbinder & Shanks, 2017)
Quality Improvement (QI)
QI is the use of a deliberate and defined improvement process which is
focused on activities that are responsive to community needs and
improving population health. It refers to a continuous and ongoing effort
to achieve measurable improvements in the efficiency, effectiveness,
performance, accountability, outcomes, and other indicators of quality in
services or processes which achieve equity and improve the health of
the community (Buchbinder & Shanks, 2017). It is important to note that
quality assurance and quality improvement are not the same.
(Buchbinder & Shanks, 2017)
Buchbinder, S. B. & Shanks, N. H. (2017). Introduction to health care
management. 3rd Ed. Jones & Bartlett Learning: Burlington, MA.
Cowing, M., Davino-Ramaya, C. M., Ramaya, K., & Szmerekovsky, J.
(2009). Health Care Delivery Performance: Service, Outcomes, and
Resource Stewardship. The Permanente Journal, 13(4), 72–78.
Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2911834/
GOVERNMENT, LAW, AND PUBLIC HEALTH PRACTICE
Defining and Assessing Quality Improvement Outcomes:
A Framework for Public Health
Anita W. McLees, MA, MPH, Saira Nawaz, PhD, Craig Thomas, PhD, and Andrea Young, PhD
We describe an evidencebased framework to define
and assess the impact of quality improvement (QI) in public
health. Developed to address
programmatic and researchidentified needs for articulating
the value of public health QI in
aggregate, this framework proposes a standardized set of
measures to monitor and improve the efficiency and effectiveness of public health programs
and operations.
We reviewed the scientific
literature and analyzed QI initiatives implemented through the
Centers for Disease Control
and Prevention’s National Public Health Improvement Initiative
to inform the selection of 5
efficiency and 8 effectiveness
measures.
This framework provides
a model for identifying the
types of improvement outcomes targeted by public
health QI efforts and a means
to understand QI’s impact
on the practice of public
health. (Am J Public Health.
2015;105:S167–S173. doi:10.
2105/AJPH.2014.302533)
AT A TIME WHEN TAXPAYER
resources are scarce, government
agencies are expected to deliver
on broader missions while reducing operating costs.1—6 As stewards
of public funds, agencies must
implement programs and deliver
services as effectively and efficiently as possible on the basis of
the best evidence available. Federal programs are required to engage in rigorous measurement and
evaluation and use the findings to
facilitate continuous improvement
and understand the value of services and programs for improved
accountability and decision making.7 This approach relies on the
adoption of valid measures that
track progress toward goals,
identify areas for improvement,
and assess achievement of
outcomes.7,8
In the public health field, quality improvement (QI) is an increasingly recognized approach to
maximizing the effectiveness of
services while minimizing costs. As
defined by Riley et al., public
health QI “refers to a continuous
and ongoing effort to achieve
measurable improvements in the
efficiency, effectiveness, performance, accountability, outcomes,
and other indicators of quality
in services or processes which
achieve equity and improve the
health of the community.”9(p6)
To date, several initiatives have
promoted the use of QI among
public health agencies with the
goals of reaching these outcomes
and building the evidence base.
Tools, such as the National Public
Health Performance Standards,
and initiatives, such as the Turning
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
Point Performance Management
Collaborative and the Robert
Wood Johnson Foundation–supported Multi-State Learning
Collaborative, represent some of
the earliest efforts that encouraged
health departments to adopt performance management and QI
methods as a strategy to strengthen
public health systems.10—12
More recently, new initiatives
aimed at integrating QI into the
practice of public health have
included the Robert Wood Johnson
Foundation—funded Communities of Practice for Public Health
Improvement, which serves as
a forum for public health agencies
to exchange best practices related
to QI,13 and the Centers for Disease Control and Prevention’s
National Public Health Improvement Initiative (NPHII), through
which 73 state, tribal, local, and
territorial public health agencies
are funded to achieve public
health standards and adopt and
institutionalize cross-cutting performance management and QI
approaches to improve the accountability, efficiency, and effectiveness of their public health
programs and services.14,15 Most
recently, the establishment of
the Public Health Accreditation
Board and its release of version
1.0—and subsequently version
1.5—standards and measures
have driven public health agencies to integrate performance
management into their daily practice. The Public Health Accreditation Board has further highlighted
QI as an important aspect of the
performance management system,16 supporting the Turning
Point initiative, which includes QI
as a core component of its performance management framework as
a demonstrated means to manage
change and make improvements
based on data.11
As a result of these efforts, the
body of evidence for public health
QI is growing, with a focus on
the extent to which public health
agencies have adopted QI and the
kinds of QI processes and tools
implemented.6,12,17—21 However,
conceptualizing and assessing
outcomes resulting from the
implementation of public health
QI has proven challenging, in large
part because of the diversity of
public health contexts12,20 and the
scarcity of evidence-based measurement methods.5,22,23 Only
recently have researchers and
practitioners begun to describe or
assess outcomes of public health
QI in a way that has the potential
to demonstrate the impact of this
work on public health organizations and the public health system
more broadly.5,22,24,25 Recent
studies have described the role
of public health QI in addressing
service and program processes as
well as operational processes.5,22
Another study highlighted that
McLees et al. | Peer Reviewed | Government, Law, and Public Health Practice | S167
GOVERNMENT, LAW, AND PUBLIC HEALTH PRACTICE
certain characteristics of QI initiatives correlate with an increased
likelihood of attaining stated objectives, including clarity around
select measurement parameters
such as time frames, baselines, and
targets.24 Although progress has
been made, these studies have
acknowledged that the evidence
base for what works in public
health QI is still growing and
standardized measures for improvement initiatives targeting
operational or programmatic
efficiency and effectiveness are
lacking. To improve performance,
public health practitioners and researchers need to clarify what we
hope to achieve and continue to
build the evidence base for what
works.5,22,25
In recognition of this need,
the Public Health Services and
Systems Research national
research agenda has focused
attention on the following QI
research questions:
d
d
What measures provide the
most valid and reliable indicators of the implementation and
impact of QI strategies in public
health settings?
What types of QI strategies have
the largest effects on the effectiveness, efficiency, and outcomes of public health strategies
delivered at local, state, and national levels?26
To advance the science and
practice of QI outcome measurement, we conceptualized a framework that proposes and defines
a standardized way to assess public health QI outcomes related to
efficiency and effectiveness. The
primary purposes of this QI measurement framework are to (1)
support public health agencies’
efforts to achieve demonstrable
outcomes, (2) provide a means to
aggregate the impact of individual
QI initiatives, and (3) advance
the science and practice of this
emerging field.
METHODS
We based our identification of
specific outcomes and the development of a standardized measurement approach on an iterative
process that used both theoretical
and grounded approaches, including a review by Centers for
Disease Control and Prevention
evaluators (A. W. M., C. T., and
A. Y.) and a contractor (S. N.) of
both the existing literature and
data collected by the Centers for
Disease Control and Prevention
on QI initiatives reported by
NPHII awardees.
We identified peer-reviewed
journal articles in PubMed by
means of a title—abstract search.
The search terms public health and
quality improvement were applied
together and in combination with
each of these additional terms:
outcomes, efficiency, effectiveness,
and evaluation. After removing
duplicates, we identified 147
articles. We conducted an additional PubMed search, applying
the terms measurement and public
health in the title—abstract field
and quality improvement as a text
word. This search resulted in 34
articles. Once all duplicates were
removed, 170 articles remained.
Evaluators reviewed abstracts for
these 170 articles and removed
those that focused on health care
settings, accreditation, or laboratory services, resulting in 35
articles that directly addressed the
topic of public health QI. We
identified an additional 3 articles
through a manual review of a table
of contents (volume 16, issue 1, of
the Journal of Public Health Management and Practice), resulting in
a final total of 38 articles.
Many studies documented the
process of QI, including workforce
development,27,28 establishing
a culture or environment conducive to QI,12,29—31 integrating
QI within a broader framework,20,32,33 or describing QI
implementation.17—19,34—37 Other
articles referred to QI outcomes
without specifying them.24,38
Many articles described various
types of public health QI
efficiency-related outcomes, including cost reductions22,31,39—42
and time savings.5,9,21,22,40,41,43
Effectiveness-related outcomes
were also described, including increased reach of, or access to, programs and services5,22,39,41,43,44;
improved quality of data,5,43,45
programs, or services21,31,41,45; increased customer or client satisfaction5,42,44; changes to organizational structure31; increased
preventive behaviors5,22,41—44,46;
and reduced disease incidence or
prevalence.22,43,45 The review also
highlighted the need for a robust
measurement system23,31,33,38,47
to accompany the articulation of
outcomes.
Additional inputs included reports by the Institute of Medicine
that focused on performance
measurement, public health, and
health care quality, as well as the
US Department of Health and
Human Services’ national framework for public health quality. In
Crossing the Quality Chasm,48 the
S168 | Government, Law, and Public Health Practice | Peer Reviewed | McLees et al.
Institute of Medicine highlighted
dimensions of improvement in the
personal health care delivery system that are also relevant to public
health, including a focus on quality, timeliness, and cost of administrative and clinical or servicedelivery processes. Other Institute
of Medicine reports49,50 emphasized the importance of measurement and of maximizing the efficiency and effectiveness of public
health services and strategies as
a means to make progress toward
population health outcomes. The
US Department of Health and
Human Services’s framework
identified efficiency and effectiveness as critical public health system characteristics and core components of successful QI.8,51 Both
the Institute of Medicine reports
and the Department of Health
and Human Services framework
provided conceptual guidance
for the organization of the QI
measurement framework, yet
neither source provided specific
guidance on operationalizing
concepts in a manner that would
facilitate measurement of discrete
QI initiatives.
To ensure the framework’s
relevance to current practice, we
also conducted a grounded review
of measures for QI initiatives
reported by 74 NPHII awardees to
the Centers for Disease Control
and Prevention at the end of the
2nd program year. We conducted
this review to (1) determine the
extent to which awardees’ efforts
aligned with outcomes found in
the literature, (2) identify additional outcomes to consider for
inclusion in the framework, and
(3) identify potential measurement
challenges. The review confirmed
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
GOVERNMENT, LAW, AND PUBLIC HEALTH PRACTICE
the relevance of efficiency-related
outcomes such as cost and time
savings, the importance of a focus
on health outcomes, and the
need for a series of outcomes
associated with business processes
or program or service delivery
improvements, such as standardization and enhancements to services
or systems, and a focus on reducing
steps associated with various processes. The review also highlighted
measurement challenges, including
the lack of baseline values or consistent units of measurement.
An initial version of the QI measurement framework was used by
73 awardees during the 3rd year of
the NPHII program (September 30,
2012—September 29, 2013) to test
its relevance to and utility for their
efforts. This testing resulted in a
more grounded and refined measurement framework by revealing
additional nuances to existing
outcomes, and new outcomes, that
were subsequently incorporated
into the final version. For example, recognizing that several
awardees engaged in QI efforts to
support work by other public
health system partners, we defined a new outcome to capture
how broadly QI products or
practices are disseminated.
Defining Public Health Quality
Improvement Outcomes
The QI measurement framework (Table 1) defines outcomes
for 2 key constructs—efficiency
and effectiveness—and provides
standardized-measure language
for each outcome. Specifically, 5
efficiency outcomes and 8 effectiveness outcomes were developed. We used 2 primary criteria
in the selection and definition of
these outcomes: (1) applicability to
a wide variety of public health
processes, programs, or services,
and (2) relevance to public health
agencies’ differing contexts and
stages of familiarity with QI.
By definition, efficiency outcomes typically reflect reductions
in the amount of resources
required to implement activities
resulting from a QI initiative.
Efficiency outcomes included in
the framework are time saved,
reduced number of steps, revenue
generated from billable services,
costs saved, and costs avoided.
Compared with the other efficiency outcomes, reduced number
of steps is process focused but
is the first step to realizing other
efficiency gains and may be
a more realistic outcome for
agencies new to QI.
Three outcomes track efficiencies
based on dollar amounts. Revenue
generated captures increases in resources, particularly revenue,
resulting from expansion of coverage
or increases in productivity. For example, if a QI initiative results in
timely and accurate billing for services or more productive service delivery, a public health agency might
experience increases in revenue. The
costs-saved outcome focuses on investments made by the public health
agency in labor, resources, and
overhead to achieve monetary
returns. Finally, costs avoided captures future costs that are offset by
current investments in efficiencies.
These offsets might occur because of
improved allocation of staff or current investments in automation.
Effectiveness outcomes include
results associated with improved
service or program delivery or improved implementation of organizational processes to achieve agency
or program goals. The 8 effectiveness outcomes are increased customer or staff satisfaction; increased
TABLE 1—Efficiency and Effectiveness Outcomes in the Quality Improvement Measurement Framework
Outcome
Description of Associated Measure
Efficiency
Time saved
Time to complete a specific process or deliver a specific service
Reduced no. of steps
No. of steps required to complete a specific process or delivery of a specific service
Revenue generated from billable services
Revenue generated by changing the implementation of a billable process or service
Costs saved
Costs avoided
Cost to complete a specific process or deliver a specific service
Cost avoided because of changes in a specific process or delivery of a specific service
Effectiveness
Increased customer or staff satisfaction
Percentage of customers or staff who report being satisfied or extremely satisfied with a specific service or process
Increased reach to a target population
Percentage of target population that has been offered, received, or completed a specific public health service or program
Dissemination of information, products, or
Percentage of individuals or public health partner organizations reached through
evidence-based practices
dissemination of information, products, or evidence-based practices
Quality enhancement of services or programs
Description of issue or improvement opportunity and its resolution for a specific service or program
Quality enhancement of data systems
Organizational design improvements
Description of issue or improvement opportunity and its resolution for a specific data or health information system
Description of improvements to organizational operations, business processes, or service or program
Increased preventive behaviors
Percentage of preventive or health-promoting behavior or early indicators of preventive behaviors in a target population
Decreased incidence or prevalence of disease
Percentage of individuals with disease in the target population
delivery resulting from specific organizational redesign efforts
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
McLees et al. | Peer Reviewed | Government, Law, and Public Health Practice | S169
GOVERNMENT, LAW, AND PUBLIC HEALTH PRACTICE
reach to a target population; dissemination of information, products,
or evidence-based practices; quality
enhancement of services or programs; quality enhancement of data
systems; organizational design improvements; increased preventive
behaviors; and decreased incidence
or prevalence of disease. Each of
these outcomes may be short or
long term with respect to the time
frame required to demonstrate improvements. They are intended
to represent a range of potential
improvements that are feasibly
achieved by a broad array of public
health organizations and within
myriad different programs or service delivery settings.
The existing literature has
emphasized the need to link
QI initiatives to programmatic
successes or increased equity
in service delivery, such as
increased reach to a target population, and health outcomes, such
as increased preventive behaviors
(or, alternatively captured in this
outcome, reduced risk factors) and
decreased incidence or prevalence
of disease.5,8,12 However, in the
early stages of QI efforts, public
health agencies may not yet be
able to detect improvements in
these outcomes. Therefore, we
included a range of outcomes to
highlight more immediate QI
successes. Dissemination of information, products, or evidencebased practices tracks results of
public health agencies’ efforts to
share products with or provide
other forms of technical assistance
to their community or regional
partners. Quality enhancement of
services tracks standardization of
services, adoption of evidencebased practices, and compliance
with established policies with the
goal of improved service delivery,
and quality enhancement of data
systems captures improvements in
data systems’ accuracy, functionality, and standardization. Finally,
improved effectiveness may result
as public health organizations reorganize or adjust their service
delivery models for more effective
use of human resources. These
changes are captured under organizational design improvements.
Within the framework, each
outcome is defined independently
for purposes of clarity and simplicity, recognizing that any given
QI initiative may address multiple
outcomes either within or across
the constructs of efficiency and
effectiveness. Also, public health
agencies may identify other outcomes of interest. To increase
the framework’s usability, the outcomes are accompanied by a series of steps to consider at the
outset of any QI initiative. First,
practitioners are asked to determine what they hope to achieve
if their QI initiative is successful:
increased efficiency, increased effectiveness, or both. On the basis
of the response to this first question, practitioners can identify the
specific outcome of interest. Any
initiative may have a primary
intended outcome as well as additional intended benefits or outcomes that should be considered.
The framework provides a series
of guiding questions for consideration when deciding on outcomes,
notably, Is the outcome relevant?
Does it reflect the intent of the
initiative given the problem or
opportunity being addressed? Is
the outcome achievable given
the available resources and the
given time period? Is the outcome
measurable? Are data sources
available?
Quality Improvement
Measurement Framework
The framework has been
implemented in the field for 2
years. For the framework to be
relevant and useful, it had to
improve the consistency of
measurement of efficiency and
effectiveness outcomes while
simultaneously acknowledging
and respecting the diversity of
public health agencies and their
QI initiatives. To this end, the
framework guides practitioners
through an approach to developing measures that is both standardized and customizable to individual agency priorities.
Standardizing measurement of
public health quality improvement
outcomes. Each QI initiative is
unique to each jurisdiction’s needs
and context. Therefore, the measurement approach uses a standard
set of generic measures (Table 1)
that address each of the framework’s key outcomes associated
with efficiency and effectiveness.
This approach allows each organization to tailor the measures to the
aims of its specific programmatic,
service-oriented, or process-oriented
QI initiative and facilitates a consistent approach to measurement despite the wide array of QI efforts.
To ensure common interpretation and application of this generic
measurement language, the framework incorporates additional guidance regarding the calculation of
these measures and considerations
for other contextual information.
Specifically, for each outcome and
associated measure, the framework
S170 | Government, Law, and Public Health Practice | Peer Reviewed | McLees et al.
includes (1) a definition of the
intended outcome and further clarification of the measure itself, including sample measures; (2) specific information about what should
be considered when establishing
baseline and target values for the
measure and what should be captured after implementation of the QI
initiative; (3) guidance on how to
calculate the measurement specifications, such as the numerator and
denominator, start and stop time, or
criteria to consider for qualitative
measures; and (4) when applicable,
additional information that may
provide context to the measure
itself.
Given that some public health
QI initiatives are more conducive
to quantitative measurement than
others,5,24 the framework includes
a combination of quantitative and
qualitative measures, depending
on the outcome. For example,
measures for increased customer
or staff satisfaction would be
quantitative, specifically, the percentage of customers who were
satisfied or extremely satisfied
with a service. For the time-saved
outcome, the measure would represent start and stop times to
calculate the average time taken
to complete a service or activity.
An example of an outcome with
a qualitative measure is quality
enhancement of services. For this
outcome, a baseline may describe
gaps in effectiveness resulting
from variability in services, and
the postimplementation value
would reflect gains achieved because of standardization or policy
implementation.
Implementation of the framework.
After the first 1.5 years of implementation among 73 NPHII
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
GOVERNMENT, LAW, AND PUBLIC HEALTH PRACTICE
TABLE 2—Examples of Quantitative and Qualitative Measures Using the Quality Improvement Measurement Framework
Associated Outcome
Example Measure
Quantitative
Efficiency
Time saved
No. of days from award letter to contract start date
Costs saved
Cost per unit of pharmaceuticals wasted in 4 Public Health Center pilot sites
Reduced steps in process
No. of steps in accessing the most recent data from (state-based) information technology system
Revenue generated from billable services
Revenue generated by reducing the no. of preventable denials for claims submitted for clinical service by health department
Costs avoided
Cost of processing applications by the Environment Unit (through move from paper to online processing)
Effectiveness
Customer and staff satisfaction
Increased preventive behaviors
% of nursing staff satisfied or extremely satisfied with the protocols in their refugee clinic
% of babies born at hospital to moms with diabetes who are exclusively breastfed in the hospital
Decreased incidence and prevalence
% of individuals exposed to syphilis who are designated as unable to locate
Dissemination of Information
% of participating programs that are submitting meaningful measures to the agency Dashboard report
Organizational design improvements
% of job descriptions across the division that align with standard (responsibilities) domains identified for each job class
Qualitative
Effectiveness
Quality enhancement, services
Uniform standard policy and procedures for disposal of sharps used in HIV testing will be in place within 6 mo of development
Quality enhancement, systems
Extent to which health departments’ databases are compliant with standards for collection of race, ethnicity, and gender data
awardees, 97.3% of awardees (71
of 73) submitted measures for at
least 1 QI initiative. This yielded
693 measures for 357 QI initiatives because several of the initiatives addressed more than 1
outcome and therefore resulted
in development of more than 1
measure. A variety of data sources
informed measures, including but
not limited to process maps, customer satisfaction surveys, vital
records, programmatic data, and
electronic health records.
The most commonly addressed
outcomes were quality enhancement of services (18.2%; n = 126),
time saved (17.7%; n = 123),
and increased customer or staff
satisfaction (11.4%; n = 79).
The outcomes least frequently
addressed were revenue generated
from billable services (0.7%; n = 5),
costs avoided (0.9%; n = 6), and
costs saved (1.2%; n = 8).
NPHII awardees reported both
quantitative and qualitative
measures. Of all measures, 83%
(n = 575) tracked quantifiable improvements, and 16.7% (n = 116)
tracked improvements qualitatively. The remaining measures
(0.3%; n = 2) were somewhat
ambiguous and difficult to categorize. Quality enhancement of services had the highest percentage
of qualitative measures (5.6%;
n = 39), followed by quality
enhancement of systems (4.2%;
n = 29). Examples of quantitative
and qualitative measures are
provided in Table 2.
DISCUSSION
This standardized framework
represents 1 approach to operationalizing and defining measures
for QI efficiency and effectiveness
outcomes that can be applied in
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
a broad array of public health
contexts. The framework identifies
measures relevant to a range of
public health programs, services,
and operational processes. Although initially originated as a
framework for specific QI initiatives, the close linkage between QI,
especially at the organization level,
and performance management38
allows it to be useful within, or
considered a part of, performance
management efforts. For example,
these outcomes and measures may
be used to assess changes related
to agency priorities or captured
within an agency’s performance
management system. Whether
used specifically in the context
of discrete QI projects or embedded in broader performance
management efforts, this framework provides a unique balance
between standardization and
customization through a focus on
outcomes without prescribing
specific processes to achieve
them and generic measures that
can be tailored to agency- or
program-specific initiatives and
contexts.
According to the Public Health
Services and Systems Research
national research agenda and recent literature on the science of QI,
the field of public health QI has
grown in both visibility and attention, presenting opportunities for
innovative approaches to practice
and research.5,22,24—26 To advance the science and practice of
public health QI, the field needs
more studies that use valid and
reliable instruments and draw
conclusions from representative
samples.52 The field can be advanced by establishing a standardized set of QI measures that can
be used to support individual
project aims as well as systemwide
McLees et al. | Peer Reviewed | Government, Law, and Public Health Practice | S171
GOVERNMENT, LAW, AND PUBLIC HEALTH PRACTICE
initiatives. This framework has the
potential to advance the dialogue
around these needs by (1) presenting a parsimonious measurement
model for collecting data on QI
efficiency and effectiveness outcomes at the program and agency
levels; (2) testing the face validity of
the framework through implementation in the field across a variety of
state, local, tribal, and territorial
health departments; and (3) identifying the types and frequency of QI
approaches used to improve public
health programs’ and services’ efficiency and effectiveness.
This framework is unique in its
articulation of a standard set of
outcomes and measures uniquely
applicable to public health QI that
are responsive to needs identified
in the literature and reflect current
public health practice. However,
a review of resulting measures and
other information on QI initiatives
is critical to determine whether
other core outcomes of public
health QI need to be considered.
Similarly, an analysis of timeframes required for achievement
of various outcomes may help inform
improvements to guide the application of various outcomes and expectations to achieve results. Additional
analysis of data derived from implementation of the framework will
further test the validity and reliability
of the measurement constructs across
varying QI initiatives, programs, and
organizations, as well as build an
understanding of how context affects
its use. Further research can build on
the measurement framework and
explore how it may be used to understand the impact of QI across
multiple contexts and over time.
The framework is intended to be
a living document that can expand as
understanding of the science and
practice of QI in public health progresses, ultimately contributing to the
“so what” of public health QI. j
About the Authors
Anita W. McLees, Craig Thomas, and
Andrea Young are with the Division of
Public Health Performance Improvement,
Office for State, Tribal, Local, and Territorial Support, Centers for Disease Control
and Prevention, Atlanta, GA. Saira Nawaz
is with Carter Consulting, Atlanta, GA.
Correspondence for this article should be sent
to Anita W. McLees, Centers for Disease Control
and Prevention, 1600 Clifton Road, NE, MS
E-70, Atlanta, GA 30333 (e-mail: zdu5@cdc.
gov). Reprints can be ordered at http://www.
ajph.org by clicking the “Reprints” link.
This article was accepted December 24,
2014.
Note. The findings and conclusions
presented here are those of the authors and
do not necessarily represent the official
position of the Centers for Disease Control
and Prevention.
Contributors
A. W. McLees led the conceptualization
and development of the framework described in the article and drafted and
revised large sections of the article.
S. Nawaz assisted with the development
of the framework, analyzed data, and
drafted sections of the article. C. Thomas
provided conceptual input into the
framework’s development and provided
substantive review of and input into
the drafting and revision of the article.
A. Young contributed to revisions to the
framework and provided substantive review of and input into the drafting
and revision of the article. All authors
approved the final version of the article.
Acknowledgments
We acknowledge Laura Hsu and Cassandra Frazier from the Division of Public
Health Performance Improvement, Office
for State, Tribal, Local, and Territorial
Support, at the Centers for Disease Control
and Prevention for their work on the
validation and cleaning of the performance
measures data presented in this article.
Human Participation Protection
No human participants were involved in
this work. Institutional review board approval was not required.
References
1. White House. Opportunity for all:
building and using evidence to improve
results. Available at: http://www.
whitehouse.gov/sites/default/files/omb/
budget/fy2015/assets/fact_sheets/
building-and-using-evidence-tostrengthen-results-in-government.pdf.
Accessed March 27, 2014.
2. Maylahn C, Fleming D, Birkhead G.
Health departments in a brave new world.
Prev Chronic Dis. 2013;10:E41.
3. National Association of County and
City Health Officials. Local health department job losses and program cuts:
findings from January/February 2010
survey. Available at: http://www.naccho.
org/topics/infrastructure/lhdbudget/
upload/Job-Losses-and-Program-Cuts-510.pdf. Accessed May 5, 2014.
4. Booz Allen Hamilton. Effectiveness
and efficiency, reimagining government:
how US federal leaders can modernize
their agencies to meet critical mission
objectives and customer expectations.
Available at: http://www.boozallen.com/
media/file/E3-Reimagining-Government.
pdf. Accessed May 12, 2014.
5. Dilley JA, Bekemeier B, Harris JR.
Quality improvement interventions in
public health systems. Am J Prev Med.
2012;42(5 suppl 1):S58—71.
6. Randolph GD, Lea CS. Quality improvement in public health: moving from
knowing the path to walking the path. J
Public Health Manag Pract. 2012;18(1):4– 8.
7. US Department of Health and Human Services. Performance.gov website.
Available at: http://www.performance.
gov/agency/department-health-andhuman-services#overview. Accessed May
12, 2014.
8. US Department of Health and
Human Services. Public health quality.
Available at: http://www.hhs.gov/
ash/initiatives/quality/quality/
improvequality2010.pdf. Accessed
May 12, 2014.
9. Riley WJ, Moran JW, Corso LC,
Beitsch LM, Bialek R, Cofsky A. Defining
quality improvement in public health. J
Public Health Manag Pract. 2010;16
(1):5—7.
10. Centers for Disease Control and
Prevention. National Public Health
Performance Standards. Available at:
http://www.cdc.gov/nphpsp. Accessed
March 27, 2014.
11. Public Health Foundation. Performance management: turning point.
S172 | Government, Law, and Public Health Practice | Peer Reviewed | McLees et al.
Available at: http://www.phf.org/
focusareas/performancemanagement/
toolkit/Pages/PM_Toolkit_About_the_
Performance_Management_Framework.
aspx. Accessed September 12, 2014.
12. Joly BM, Shaler G, Booth M, Conway
A, Mittal P. Evaluating the multi-state
learning collaborative. J Public Health
Manag Pract. 2010;16(1):61—66.
13. National Network of Public Health
Institutes. Community of practice and
public health improvement. Available at:
http://nnphi.org/program-areas/
accreditation-and-performanceimprovement/programs. Accessed May
12, 2014.
14. Centers for Disease Control and
Prevention. National Public Health Improvement Initiative. Available at: http://
www.cdc.gov/stltpublichealth/nphii.
Accessed May 14, 2014.
15. Thomas CW, Pietz H, Corso L,
Erlwein B, Monroe J. Advancing accreditation through the National Public
Health Improvement Initiative. J Public
Health Manag Pract. 2014;20(1):
36—38.
16. Public Health Accreditation Board.
Public Health Accreditation Board Standards and Measures version 1.5. Available at: http://www.phaboard.org/
wp-content/uploads/SM-Version-1.5-Boardadopted-FINAL-01-24-2014.docx.pdf.
Accessed May 7, 2014
17. Beitsch LM, Leep C, Gulzar S, Brooks
RG, Pestronk RM. Quality improvement
in local health departments: results of the
NACCHO 2008 survey. J Public Health
Manag Pract. 2010;16(1):49—54.
18. Leep C, Beitsch LM, Gorenflo G,
Solomon J, Brooks RG. Quality improvement in local health departments: progress, pitfalls, and potential. J Public Health
Manag Pract. 2009;15(6):494—502.
19. Madamala K, Sellers K, Beitsch LM,
Pearsol J, Jarris P. Quality improvement
and accreditation readiness in state public
health agencies. J Public Health Manag
Pract. 2012;18(1):9—18.
20. Corso LC, Lenaway D, Beitsch LM,
Landrum LB, Deutsch H. The national
public health performance standards:
driving quality improvement in public
health systems. J Public Health Manag
Pract. 2010;16(1):19—23.
21. Harrison LM, Shook ED, Harris G,
Lea CS, Cornett A, Randolph GD. Applying the model for improvement in a local
health department: quality improvement
as an effective approach in navigating the
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
GOVERNMENT, LAW, AND PUBLIC HEALTH PRACTICE
changing landscape of public health
practice in Buncombe County, North
Carolina. J Public Health Manag Pract.
2012;18(1):19—26.
22. Riley W, Lownik B, Halverson P,
et al. Developing a taxonomy for the
science of improvement in public health.
J Public Health Manag Pract. 2012;18
(6):506—514.
23. Derose SF, Schuster MA, Fielding JE,
Asch SM. Public health quality measurement: concepts and challenges. Annu Rev
Public Health. 2002;23:1—21.
24. Beitsch LM, Carretta H, McKeever J,
Pattnaik A, Gillen S. The quantitative
story behind the quality improvement
storyboards: a synthesis of quality improvement projects conducted by the
multi-state learning collaborative. J
Public Health Manag Pract. 2013;19
(4):330—340.
25. McLees AW, Thomas CW, Nawaz S,
Young AC, Rider N, Davis M. Advances in
public health readiness and quality improvement: evaluation findings from the
National Public Health Improvement Initiative. J Public Health Manag Pract.
2014;20(1):29—35.
26. Consortium from Altarum Institute,
Centers for Disease Control and Prevention, Robert Wood Johnson Foundation,
National Coordinating Center for Public
Health Services and Systems Research. A
national research agenda for public health
services and systems. Am J Prev Med.
2012;42(5 suppl 1):S72—S78.
29. Joly BM, Booth M, Shaler G, Conway
A. Quality improvement learning collaboratives in public health: findings from
a multisite case study. J Public Health
Manag Pract. 2012;18(1):87—94.
30. Joly BM, Booth M, Mittal P, Shaler G.
Measuring quality improvement in public
health: the development and psychometric testing of a QI maturity tool. Eval
Health Prof. 2012;35(2):119—147.
31. Riley WJ, Parsons HM, Duffy GL,
Moran JW, Henry B. Realizing transformational change through quality improvement in public health. J Public Health
Manag Pract. 2010;16(1):72—78.
32. DeAngelo JW, Beitsch LM, Beaudry
ML, Corso LC, Estes LJ, Bialek RG. Turning point revisited: launching the next
generation of performance management
in public health. J Public Health Manag
Pract. 2014;20(5):463—471.
33. Landrum LB, Baker SL. Managing
complex systems: performance management in public health. J Public Health
Manag Pract. 2004;10(1):13—18.
34. Carman AL, Timsina LR, Scutchfield
FD. Quality improvement activities of
local health departments during the
2008-2010 economic recession. Am J
Prev Med. 2014;46(2):171—174.
35. Erwin PC. The performance of local
health departments: a review of the literature. J Public Health Manag Pract.
2008;14(2):E9—E18.
38. Beitsch LM, Yeager VA, Moran J.
Deciphering the imperative: translating
public health quality improvement into
organizational performance management
gains. Annu Rev Public Health. 2014;
Published online ahead of print on December 10, 2014.
36. Gunzenhauser JD, Eggena ZP,
Fielding JE, Smith KN, Jacobson DM,
Bazini-Barakat N. The quality improvement experience in a high-performing
local health department: Los Angeles
County. J Public Health Manag Pract.
2010;16(1):39—48.
28. Davis MV, Vincus A, Eggers M, et al.
Effectiveness of public health quality improvement training approaches: application, application, application. J Public
Health Manag Pract. 2012;18(1):E1—E7.
37. Scutchfield FD, Zuniga de Nuncio
ML, Bush RA, Fainstein SH, LaRocco MA,
Anvar N. The presence of total quality
management and continuous quality improvement processes in California public
based on performance results: Washington State’s experience. J Public Health
Manag Pract. 2010;16(1):24—31.
46. Livingood WC, Sabbagh R,
Spitzfaden S, et al. A quality improvement
evaluation case study: impact on public
health outcomes and agency culture. Am J
Prev Med. 2013;44(5):445—452.
39. Beitsch LM, Grigg CM, Mason K,
Brooks RG. Profiles in courage: evolution
of Florida’s quality improvement and
performance management system.
J Public Health Manag Pract. 2000;6(5):
31—41.
47. Mays GP, Halverson PK. Conceptual
and methodological issues in public
health performance measurement: results
from a computer-assisted expert panel
process. J Public Health Manag Pract.
2000;6(5):59—65.
40. Davis MV, Cornett A, Mahanna E,
See C, Randolph G. Advancing quality
improvement in public health departments through a statewide training
program. J Public Health Manag Pract.
2014; Published online ahead of print
on October 16, 2014.
48. Institute of Medicine. Crossing the
quality chasm: a new health system for
the 21st century. Available at: http://
iom.edu/Reports/2001/Crossing-theQuality-Chasm-A-New-Health-Systemfor-the-21st-Century.aspx. Accessed
March 5, 2012.
41. Riley W, Parsons H, McCoy K,
et al. Introducing quality improvement
methods into local public health departments: structured evaluation of a statewide pilot project. Health Serv Res.
2009;44(5 pt 2):1863—1879.
42. Weir E, d’Entremont N, Stalker S,
Kurji K, Robinson V. Applying the balanced scorecard to local public health
performance measurement: deliberations
and decisions. BMC Public Health.
2009;9:127.
43. Derose SF, Asch SM, Fielding JE,
Schuster MA. Developing quality indicators for local health departments: experience from Los Angeles County. Am J Prev
Med. 2003;25(4):347—357.
27. Cornett A, Thomas M, Davis MV,
et al. Early evaluation findings from
a statewide quality improvement training
program for local public health departments in North Carolina. J Public Health
Manag Pract. 2012;18(1):43—51.
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
health clinics. J Public Health Manag Pract.
1997;3(3):57—60.
44. Tumlinson K, Speizer IS, Curtis SL,
Pence BW. Accuracy of standard measures
of family planning service quality: findings from the simulated client method.
Stud Fam Plann. 2014;45(4):443—470.
45. Mason M, Schmidt R, Gizzi C,
Ramsey S. Taking improvement action
49. Institute of Medicine. For the public’s
health: the role of measurement in action
and accountability. Available at: http://
www.iom.edu/reports/2010/for-thepublics-health-the-role-of-measurementin-action-and-accountability.aspx.
Accessed March 5, 2012.
50. Institute of Medicine. For the public’s
health: investing in a healthier future.
Available at: http://www.iom.edu/
Reports/2012/For-the-Publics-HealthInvesting-in-a-Healthier-Future.aspx.
Accessed April 15, 2012.
51. Honoré PA, Wright D, Berwick DM,
et al. Creating a framework for getting
quality into the public health system.
Health Aff (Millwood). 2011;30(4):737–745.
52. Harris JK, Beatty KE, Barbero C, et al.
Methods in public health services and
systems research: a systematic review. Am
J Prev Med. 2012;42(5 suppl 1):S42—S57.
McLees et al. | Peer Reviewed | Government, Law, and Public Health Practice | S173
International Journal of Pediatrics and Adolescent Medicine (2015) 2, 7e11
Available online at www.sciencedirect.com
H O S T E D BY
ScienceDirect
journal homepage: http://www.elsevier.com/locate/ijpam
QUALITY IMPROVEMENT
How (and why) do quality improvement
professionals measure performance?
Donald E. Lighter*
The University of Tennessee, The Institute for Healthcare Quality Research and Education, 2120 River
Sound Dr., Knoxville TN 37922, United States
Received 29 October 2014; accepted 20 December 2014
Available online 27 March 2015
KEYWORDS
Quality improvement;
Measurement;
Performance
improvement;
Medicare;
Shared savings;
Payment
Abstract The era of value-based care has engulfed healthcare delivery systems around the
world. Pediatricians are especially challenged by constrained resources for providing care to
our vulnerable population, and methods for achieving value for children through improved
quality and reduced cost of care are crucial for success. This paper examines the use of measures to determine the two components of the value proposition: quality and cost. The implications for adopting Lean Six Sigma as an improvement paradigm are reviewed, and the case
for using these concepts is detailed with examples of measures used in health systems in the
United States and several other countries.
Copyright ª 2015, King Faisal Specialist Hospital & Research Centre (General Organization),
Saudi Arabia. Production and hosting by Elsevier B.V. This is an open access article under
the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
1. Measurement in healthcare
Pediatricians in nearly every country around the world are
becoming even better at managing patients with fewer
financial resources, and Lean Six Sigma (LSS) is approach
that has been shown to be effective at increasing efficiency
while concurrently improving quality. The first paper in this
series [1] described the paradigm of LSS and how the
approach simultaneously addresses the cost and quality of
* Tel.: þ1 865 974 1772.
E-mail address: dlightermd@cyberce.net.
Peer review under responsibility of King Faisal Specialist Hospital & Research Centre (General Organization), Saudi Arabia.
care. This paper describes the metrics that quality professionals use to determine performance and how these
measures have been translated into practice in the United
States.
2. Background
A maxim in quality improvement (QI) that has been
attributed to many iconic figures in the field is “You can’t
manage what you don’t measure”. For physicians, that
axiomatic statement is almost intuitive because one of the
goals for diagnosis and treatment is to have accurate test
results to determine a patient’s clinical condition and to
determine what therapeutic interventions might be effective. Quality improvement professionals have the same
http://dx.doi.org/10.1016/j.ijpam.2015.03.003
2352-6467/Copyright ª 2015, King Faisal Specialist Hospital & Research Centre (General Organization), Saudi Arabia. Production and hosting
by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
8
D.E. Lighter
goal: to understand a process quantitatively so that
worthwhile interventions might be applied to discern and
improve performance. Once those interventions are
applied, the QI professional uses the metric to determine
the effect of the intervention, just as physicians measure
the effect of therapeutic interventions by repeating a test
after treatment. Then, to ensure that a process does not
deteriorate into its previous state of poor performance, the
QI professional monitors the measure over time and tracks
the performance trend. The role of measurement in quality
improvement is every bit as important as lab and imaging
studies in clinical medicine.
Avedis Donabedian (1919e2000) was a Lebanese-born
physician and health policy researcher at the University of
Michigan’s School of Public Health who is credited with the
earliest work in health care quality management. The
Donabedian model [2] was published in 1980 and provided
the early framework for performance measurement and
improvement in health care. As shown in Fig. 1, the model
divided healthcare services into three major categories:
Structure, Process, and Outcomes. Nearly every quality
measure can be characterized by one of these classifications. Table 1 provides some examples of each of these
types of measures. The measurement systems that have
been developed in the United States and many other
countries are organized into these three categories.
A fundamental principle in the development of effective
measures can be represented as SMART criteria, which are
shown in Fig. 2. These criteria provide the basis for development of effective measures for each of the Donabedian
model categories and are important to ensuring that a
measure will be acceptable to both providers and payers.
Developing measures using the SMART criteria involves the
following:
 Specific e the measure must address a specific goal or
process step and be as narrowly defined as possible.
Thus, a process metric should endeavor to encompass a
single step in the process, e.g., giving a child an immunization at a well-child visit. An outcome measure is
usually broader, but typically is designed to quantify an
important result of care, e.g., return to full function in
activities of daily living.
Figure. 1
Table 1
groups.
Measures categorized into Donabedian model
Donabedian Standardized metric
category
Structure
Process
Outcome
Availability of a computerized tomography
scanner
Number of hospital beds
Number of examining rooms in a clinic
Medication availability
Staffing availability
Emergency medical services equipment
Beta blockers after a myocardial infarction
Assessment of pediatric body mass index (BMI)
Percentage of two-year old children with
completed vaccinations
Human Papillomavirus vaccination for female
adolescents
Lead screening in children
Appropriate treatment for children with viral
upper respiratory infection
Death or mortality rate
Quality adjusted life years
Activities of daily living
Complications of diagnosis or treatment
Patient satisfaction
 Measurable e the metric must have an operational
definition that clearly states the data to be collected
and how that data are analyzed to create the final
measure. If no data are available, then this criterion
cannot be met.
 Achievable e any measure must have an achievable
level of performance. If providers do not control a
particular process, then for them, the measure and its
related performance goals may not be achievable,
leading only to frustration and a sense of unfairness.
 Relevant e metrics used in performance improvement
must be relevant to those involved in the process, most
often the physician and the patient or family. If the
measure is deemed trivial by stakeholders, then the
The Donabedian model.
Quality improvement professionals measure performance
Figure. 2
probability is high that it will be ignored by providers and
patients.
 Time specific e measures must have time limits that are
reasonable. For example, a measure that takes a year to
collect enough data for analysis will likely be disregarded by stakeholders, while one that can be used
daily or even weekly will garner the attention needed to
promote improvement.
Using SMART criteria and the Donabedian Model, many
professional organizations and payer entities have created
performance metrics for healthcare that are being used
daily to assess the quality of care.
2.1. Value-based metrics
A major change has occurred in the United States over the
past four years since the passage of the Patient Protection
and Affordable Care Act (PPACA), often called Obamacare.
The concept of value-based purchasing has become
important not just in government financed health care but
also for commercial insurance companies. Value-based
purchasing (VBP) is a concept based on the business paradigm of value, which is defined as:
9
SMART Criteria.
measures with their operational definitions. Most new
quality measures are sanctioned by the NQF through a
rigorous process of approval and updating, with the organization sponsoring the measure remaining responsible for
the certification and update processes.
Although this approach to measurement of healthcare
performance has led to considerable dissatisfaction among
providers, it has become well established and is now serving
as the basis for new payment models in the United States,
such as bundled payments and capitation [4]. These payment modalities are still in their infancy in many areas of
the U.S., but payers are rapidly adopting them for their
insurance plans.
3. Example case study
Quality
ValueZ
Cost
In the United States, quality measures now have an impact
on payments by private insurers or federal and state governments through Medicare and Medicaid. Some of the new
payment plans for hospitals and physicians relate quality
measures to either regular payments or to extra incentive
payments. For example, the Medicare Shared Savings Plan
(MSSP) was designed to integrate quality with cost
containment through a unique approach to sharing some of
the money saved by the imposition of cost targets. The
MSSP program works as follows:
The value of healthcare services is directly related to
the quality of those services and inversely related to the
cost. This concept is sometimes termed the “value proposition”, and it serves as the foundation for nearly all human
transactions. Thus, physicians now are being evaluated for
quality, as well as cost of services, despite the argument
that many providers have made for decades that health
care is too complex to be measured. In fact, payers have
determined that quality and cost can and will be measured,
and over the past three decades payers and accrediting
organizations, such as The Joint Commission, have created
metrics for both cost and quality. Thus, the measures noted
in Table 1 have been used for at least 20 years in the United
States; some of them have been used even longer. The
effect of establishing this set of measures for a growing
number of diagnoses has changed the way that medicine is
paid for, as well as what medical practices are deemed
acceptable and appropriate. The National Quality Forum
[3] (NQF) has become the central repository for healthcare
measures in the U.S., and the site catalogs hundreds of
 A medical provider entity (e.g., an integrated care delivery network of physicians, hospitals, and other providers) is given responsibility to provide care to a
population of patients within a defined geographic
region.
 The payer (presently the Center for Medicare and
Medicaid Services or CMS) determines a cost target for
the population based on a discount from historic cost
trends.
 A number of quality measures are chosen by the payer
based on the diseases that are inherent in the population
(i.e., the “disease burden”).
 The provider works to meet the cost target through
better care management, including preventive care and
directing patients to lower cost, high quality services.
 At the end of a fixed time period, currently one year, the
cost of care and the performance on the selected quality
measures are measured and a quality score is
calculated.
10
D.E. Lighter
Figure. 3
Shared savings scenario.
 Based on the quality metric performance, the provider
shares in the cost savings with the provider up to
50e60%.
Fig. 3 illustrates this scenario. The figure demonstrates a
scenario in which care for a population of patients is
determined to be increasing at the trend shown in the
upper line. The payer determines that the lower line is the
trend that should be achieved by the physicians and hospitals providing care and selects several quality measures.
The difference between the two lines is the total savings
expected by the payer. If the provider group achieves the
target levels for all of the selected quality measures, then
the group will share 50% of the savings with the payer,
which is equivalent to receiving a “bonus” payment equal
to half of the savings that the group achieved. Here are
some examples of numbers to illustrate how this system
works:
 A population of 10,000 patients is selected at the
beginning of the shared savings agreement, and the
payer examines current cost trends based on historical
data from the past five years of health services use by
the patient population. Based on those trends, the
overall cost of care is expected to be USD 10 million by
period 4 of the agreement.
Table 2 Shared savings distribution based on quality
measure performance.
If the percentage of
measures meeting
the threshold is.
The percentage of savings
shared by the provider
group is
50%e59%
60%e69%
70%e79%
80% or above
20%
30%
40%
50%
 The payer uses statistical approaches and the disease
burden of the population to make predictions of target
costs of USD 8 million at period 4, a reduction of 20% of
the costs or USD 2 million.
 Based on the disease burden of the population, the
payer selects quality measures for the five most prevalent conditions:
– Congestive heart failure
– Acute myocardial infarction
– Pediatric asthma
– Neonatal intensive care unit
– Chronic obstructive pulmonary disease
 Thresholds were established for each of the measures in
the five disease categories that required high level
performance. The provider group’s performance on the
measures determined the amount of the shared savings
the group would share as shown in Table 2.
 At the end of period 4, the provider group reports its
quality measure results and meet 72% of the threshold
targets. From the table, that level of performance would
earn 40% of the USD 2 million in savings, or USD 800,000.
Thus, quality metric performance has a significant
impact on the revenues returned to the provider group.
Since performance measurement has become such an
important factor in the healthcare industry, a basic understanding of quality measurement is important for clinicians and physician leaders.
4. Measurement is key to delivering value
A LSS program is value based, as discussed in the first paper
in this series. The Lean approach promotes process efficiency by removing non-value-added work and streamlining
process flow, and Six Sigma has methods and tools that
induce effectiveness, reduce errors, and improve safety.
The value proposition requires that these features of LSS be
demonstrated using objective metrics, which is the key to
effective implementation of a LSS culture. LSS metrics are
Quality improvement professionals measure performance
Table 3
Type of
measure
Efficiency and effectiveness measures.
Example measures
Efficiency
Turnaround time for lab results
Turnaround time for x-ray procedure reports
Time for pre-physician prep during office visit
Cost of a procedure
Time required to prepare for a procedure
Effectiveness Accuracy of lab results
Diagnostic accuracy of x-ray interpretations
Complication rates from a procedure
Medical error rates by physician
Medication errors
Immunization rates
Tobacco use rates among adolescents
usually divided into two major categories: those related to
efficiency and those related to effectiveness. Examples of
some of these measures are listed in Table 3. A common
characteristic of efficiency measures is the inclusion of
time as a factor, with a goal of reducing time to the lowest
possible level. On the other hand, effectiveness measures
tend to be more outcome oriented and relate to safety and
patient harm.
Characterizing measures by efficiency and effectiveness
clearly fits into the value-based purchasing model. Cost is
11
linked to efficiency, while quality relates to effectiveness.
LSS thus provides the foundation for achieving value for
patients. Using the LSS approach, quality practitioners have
the ability to address both of the key aspects of the value
proposition, and this system of improvement will serve both
clinicians and QI professionals with the tools and methods
to achieve the high levels of performance as health care
around the world moves into an era of accountable care.
Conflicts of Interest
None.
References
[1] Lighter Donald E. The application of Lean Six Sigma to provide
high-quality, reliable pediatric care. IJPAM 2014;1(1):8e10.
[2] Donabedian A. Explorations in quality assessment and monitoring: the definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Press; 1980.
[3] National Quality Forum, [accessed October, 2014] at http://
www.qualityforum.org.
[4] Damberg CL, Sorbero ME, Lovejoy SL, Martsolf G, Raaen L,
Mandel D. Measuring success in health care value-based purchasing programs: findings from an environmental scan, literature review, and expert panel discussions, Rand Corporation,
[accessed October 2014] at http://aspe.hhs.gov/health/
reports/2014/HealthCarePurchasing/rpt_vbp_findings.pdf.