HS 159 Health Program Planning Impact Evaluation Plan Matrix Template

Impact Evaluation Plan MatrixObjective:
Evaluation Questions
Description of Data to be
Collected
Quantitative:
Data Sources
Data Collection Methods
Data Analysis Methods
Methods
Quantitative:
Instruments
Qualitative:
Methods
Instruments
Qualitative
HS 159 Spring 2020
April 20, 2020
Image: https://www.usu.edu/ccampis/media/Evaluation.jpg
Agenda and Reminders
— Welcome, Announcements, & Questions
— Feedback on PC8, PC12
— Evaluation Overview
— Evaluation Activity
— Where you should be – what you are working on
— G&O Matrix revisions
— Theoretical Framework revisions
— Strategies and Activities revisions
— Implementation Plan Draft
— Implementation Barriers and Facilitators Draft
Words of
Encouragement
•
•
•
•
—
—
—
—
—
—
We’ve made it this far – keep going – finish strong – we’ve
got this
Keep trying, don’t give up and don’t be afraid to ask for
help
Start early, don’t procrastinate, time management is key
We are all in this together – work with each other for
success
Don’t stress – do your best. The class will go by quick so
chill & relax
Remember you are not in this class alone
Be gentle on yourself – there’s only one you
Keep pushing yourself out of your comfort zone
Act as if what you do makes a difference – IT DOES
We can do it!
3
PC 8 Feedback
— 1 Page Problem Statement Draft
— Remove emotional language – this is academic writing
“sadly”
— Pay attention to:
— Grammar, syntax, spelling, sentences that begin with a
number must be spelled out: ”Five people”
— Once you spell out an organization and refer to the
acronym – you should use the acronym throughout the rest
of your proposal: Example: “Centers for Disease Control
and Prevention (CDC)
— Minimum 5 references including 2 peer reviewed journal
articles (published within the last 5 years)
PC 12 Feedback
— Be sure to:
— Clearly introduce and explain each strategy
— Explain each activity that falls under each strategy
— What will participants be doing? Who do activities target?
— Refer to best practices, and cite them
— Pay attention to:
— grammar, syntax, spelling
— Subject verb agreement
— Consistent use of tense
— 3rd person narrative throughout
— Be careful when using declarative statements “will lower
depression rates” “will create change”
PC 12 Feedback
— Strategies & Activities Feedback continued
— Avoid superlatives, “in order to”, “not only this”
— Etc. is not used in academic writing – write clearly and
concisely
— Paraphrase – demonstrate understanding of concepts; no
direct quotes and check with Turnitin –if you are not
receiving a Turnitin score, contact ecampus for help to
resolve this issue
What is Evaluation?
vA process…”to determine the significance, worth, or
condition of (someone or something) by careful
appraisal and study.” (Miriam Webster, 2014)
vA judgment about the value, importance, or quality
of something after considering it carefully (Macmillian,
2013)
Careful…..thoughtful…..pre-determined set of standards…
This Photo by Unknown Author
is licensed under CC BY-NC-SA
Common Evaluations
v Job performance
v Customer satisfaction surveys
v Peer evaluations:
v
v
v
Evaluations of your school work:
v
v
v
v
Team Presentations/Group Work
Peer Review Activities (in class or Canvas)
Grades on homework/papers
Instructor feedback/comments
GPA
Evaluations of your professors:
v
v
SOTES
Rae My Professor

Evaluations you may have had
experience with….
v Job performance
v Peer evaluations for classwork: HS25 Team
Presentations
vEvaluations of your school work: Grades on
papers and homework assignments
vEvaluations of your professors: SOTES
Why Evaluate Programs?
vProvide feedback on how well objectives were
accomplished.
vDirect the planner towards areas for continuous
improvement
vProve that good and necessary things were done
(outcome)
vDocument how they were done (process)
vTransfer best practices with clear evaluation evidence
Coley, 2014
General Categories of Evaluation
Informal evaluation
vHas absence of breadth and depth because of the
lack of systematic procedures and formally
collected evidence
Formal evaluation
vHas “systematic, well-planned procedures” and
controls extraneous variables
11
Program Evaluation
v“…the systematic collection of information
about the activities, characteristics, and
outcomes of programs to
o make judgments about the programs,
o improve program effectiveness, and/or
o inform decisions about future program
development.”
EVIDENCE-BASED EVALUATIONS
(DHHS, 2005)
Framework for Program Evaluation
Utility – ensure
that an
evaluation will
serve the
information
needs of
intended users.
Feasibility
ensure that an
evaluation will
be realistic,
prudent,
diplomatic and
frugal.
Propriety –
conducted legally,
ethically and with
due regard for the
welfare of those
involved in the
evaluation, as well
as those affected by
its results.
http://www.cdc.gov/eval/framework/
Accuracy – reveal
and convey
technically
adequate
information about
the features that
determine worth
or merit of the
program being
evaluated.
Some areas to evaluate?
v
Implementation:
Is the program being put into place as intended?
v
Effectiveness:
Is the program reaching its goals and objectives?
v
Attribution Ability
Is there a clear link between your program activities
and progress on the objectives and ultimate goal?
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTGqyreV0A78vMMJOTXXGc03C44OL5NH6wHSHkIra8nzDk-NXWc
Types of Evaluations
Formative Evaluation
vTo obtain information to develop
or improve a program
vOccurs during program planning or implementation
Summative Evaluation
vTo determine whether a program produced its
intended effects
vOccurs after program implementation or at the end of
the program
Process Evaluation
(used in Formative Evaluations)
How program achieved results:
vAssessment of the procedures used in conducting
programs
vGather feedback during program operation to
assess if changes need to be made
v Improve later programs with similar focus
vLevel of staff effort to achieve results
vLevel of outreach to priority population
vLevel of participant satisfaction
Coley, 2014
Impact Evaluation
(used in summative evaluations)
Impact Evaluation connects to Impact Objectives.
Remember, what we call impact
Coley calls short & midterm outcome.
vTo measure the extent to which the program has caused
the intended short-term changes
vImprovement in behavioral, environmental,
predisposing, reinforcing, or enabling factors
vWhat and how much was achieved?
vChanges in participants or conditions
vResults of activities on recipients
Outcome Evaluation
Outcome Evaluation connects to Outcome Objectives
Coley calls these “long term outcomes”
Used in Summative Evaluations
vTo measure the extent to which the program has
caused the intended long-term changes, such as:
v Improvement in health status or condition
v Positive change to quality of life
v Long term continuation of behaviors
v Policy regulation and enforcement
v Proxy measures – if you can’t measure the actual thing,
measure something else that is shown to be correlated
with outcomes
Where do they all fit?
Planning
Start of
Implementation
Formative
End of
Implementation
Summative
Process
Impact
19
Outcome
Evaluation Questions
Four main categories:
Planning and implementation issues
v How well was the program or initiative planned out, and
how well was that plan put into practice?
Assessing attainment of objectives
v How well has the program or initiative met its stated
objectives?
Impact on participants
v How much and what kind of a difference has the program or
initiative made for its targets of change?
Impact on the community
v How much and what kind of a difference has the program or
initiative made on the community as a whole?
CTB, 2013
Determine the Type of Evidence
Examples:
vAgency records
vProgress reports
vTime allocation records
vMeeting agendas and minutes
vVisitor logs
vQuestionnaires
vInterview notes
vParticipant surveys
vDocumentation of role plays
vIntake and exit interviews
vPre and post test Surveys
vGovernment Records Review
vMedical chart review
Coley, 2014
Data Analysis Methods
Depends on the type of information needed
to answer the evaluation questions.
Quantitative:
Qualitative:
Descriptive
Statistics
Counts and
Frequency
Content
Analysis
Means and
Medians
Recurring
Themes
Rates/ratios of
change
http://www.qualitativedataanalysis.net/wp-content/uploads/2016/11/principles-of-data-analysis-online.jpg
22
Evaluation Framework
For your proposal . . .
— Your evaluation plan will identify two impact objectives
and describe how, if you were funded, you would
determine the degree to which those objectives had been
met.
— Use the evaluation template provided for indicating the
selected objectives, indicators, and data sources.
— Each chosen impact objective requires a separate
completed evaluation template.
— Let’s All Try it Now!
Based on your objective complete this
chart…..
Impact Objective:
Evaluation
Question
Description of
Data to be
Collected
Data
Sources
Data
Collection
Methods
Data Analysis
Methods
Quantitative:
Instruments:
Quantitative:
Qualitative:
Methods:
Qualitative:
Evaluation Process Evaluation
Approach
Why
Did I deliver what I was
supposed to?
Summative Evaluation
Did anything change?
Evaluation Who? What? Where?
Questions When? How Many?
Focus
Data
Analysis
To what extent…did the
implementation of ______
result in a change in
________?
Involves counts: # of
Involves measuring change:
people recruited, # of
extent of change in
people participated, # of awareness, knowledge,
materials distributed, # of attitudes, behavior,
services (e.g. classes)
environment, health status
delivered
Descriptive – counts
Descriptive and Inferential
Example
Impact Objective:
Following the educational training session, 85% of participants will be able to
describe three new strategies in care and communication that can decrease the
potential of elder abuse at ABC Skilled Nursing Facility.
Evaluation
Question
Description of Data
to be Collected
How many
participants were
able to describe 3
strategies in care
and
communication?
Quantitative:
Data Sources Data Collection Data Analysis
Methods
Methods
Skilled
The percentage of
Nursing
participants who can Facility staff
describe 3 new
(participants)
strategies in care and who
communication.
participated in
How confident do
the
participants feel in Qualitative:
educational
Participants’
views
on
their utilizing
training
the strategies in care and
strategies in the
communication and their session.
future?
What barriers and
facilitators can
they identify
confidence in their ability
to use those strategy.
Participants identification
on barriers and
facilitators
Methods:
Quantitative:
Instruments:
Qualitative:
Descriptive statistics
identifying the
•Conduct pre
and post-test
percent of
participants who
•Interview
participants and were able to
describe 3 new
record
responses
strategies
Summary of
•Pre-Post test recurring themes
•Semifrom participant
interviews regarding
structured
interview guide their views on the
strategies and their…
Example
Impact Objective:
Following the presentation of the public transit accessibility findings and recommendations, local
leaders will be able to explain three ways, that they did not know before, in which accessible and
reliable transportation impacts the daily lives of people with disabilities in the community.
Evaluation
Question
Description of Data
to be Collected
Data Sources
Quantitative:
How many local
leaders were able
to explain, at least
three news
impacts of
accessible and
reliable
transportation on
daily activities for
people with
disabilities?
The percentage of
local leaders who
can effectively
identify and explain
the impact of
accessible
transportation.
Qualitative:
Personal experiences,
What are the
views, and verbal
participants views
commitment to
on the need to
ensure accessible ensuring that
transportation
transportations?
accessibility is
improved.
Data Collection Data Analysis
Methods
Methods
Methods:
Local leaders
and disability
advocates
attending
presentation
•Conduct prepost survey with
county
supervisors,
transit board,
disability
advocates
attending
presentation.
•Follow up
interviews.
Quantitative:
Descriptive statistics
identifying the
percent of
participants who
were able to explain
three impacts.
Qualitative:
Summary of
Instruments:
recurring themes
from participant
•Pre-post
interviews regarding
Survey
their views and
•Semicommitment to next
structured
interview guide steps.
Additional Resources
— CDC, (n.d.). A framework for program evaluation.
https://www.cdc.gov/eval/framework/index.htm
For Wednesday 4/22/20
— Evaluation Plan – download PC14 to work
on in class
— Preparation for class:
— Review Coley, chapter 7 (repeat from today)
— Community toolbox, chapter 36, section 5
— Submit PC14: Implementation Plan Draft by 2:45pm
Impact Objective:
Evaluation
Questions
Description of
Data to be
Collected
Data Sources
Data Collection
Methods
Data Analysis
Methods
Quantitative:
Methods:
Quantitative:
Qualitative:
Instruments:
Qualitative:
Elements of a Process Evaluation
Student Name:
Health Issue: Lung Cancer
Priority Population & Community: Vietnamese Immigrants in Santa Clara County
Goal: To reduce the rate of lung cancer among Vietnamese immigrants in Santa Clara
Activity: To educate members of the priority population and community about the risk factors of lung
cancer and the reason why they should change their behaviors and seek early cancer screening.
Process
Impact
Outcome
– By May 30th, 2020, program
director will be hired 10
community health nurses to
provide educational services to
the Vietnamese immigrants in the
county.
– By December 30th, 2020, 60% of
the Vietnamese Immigrants will
be able to seek help and do cancer
screening more frequently.
-By May 30th, 2023, the incidence
of lung cancer among Vietnamese
immigrants in Santa Clara will
reduce by 40% compared to the
rate from the beginning of the
program.
– By June 1st, 2020, staff will
ensure that all 10 community
health nurses confirmed their
arrival/presence for the upcoming
program.
– By May 30th, 2020, county’s
staff will print 400 copies of the
education manuals and have
enough learning materials for the
people who will participate in the
education program. (about 300400 participants)
– By May 30th, 2020, staff will
ensure there is different languages
such as Vietnamese version and
English version for the education
manuals with in dept information
about the program.
– By July 30th, 2020, the
community health nurses will
develop customize programs such
– By May 30th, 2021, 50% of
Vietnamese immigrants will meet
the minimum required physical
activity (about 10-15 minute/day)
compared to the start of the
program. (there will be an app for
them to log in and put how long
for what activity they do
according to the customize
routines that the nurses/trainers
have created, so the nurses can
follow up and collect data)
– By July 30th, 2021, 70% of
Vietnamese immigrants will be
able to adapt to their customized
healthy diets on daily basis. (same
as the physical activity one)
– By May 30th, 2022, 40% of
cigarette smokers among
Vietnamese immigrants will
reduce.
as physical activities and healthy
diet plans for different groups
(people are at risk of getting lung
cancer and people are having the
lung cancer) within the priority
population.
– By June 1st, 2020, the staff will
ensure the information about this
program will show on media and
radio channels which have the
Vietnamese Immigrants watch the
most to reach a wide range of
audience.
-By June 1st, 2020, staff will
ensure the confirmation of the
location rental for the program
(Yerba Buena High School Gym)