Program Evaluation Methods and Results
January 2020
Source: http://www.artsforlifenc.org/art-gallery/
Prepared by:
Emma Olson, MSW, MPH, Director of Partnerships and Evaluation
Camden Spade, MPH, Research Assistant
Caitlin Turbyfill, Research Assistant
2
Table of Contents
BACKGROUND .................................................................................................................................. 3
PERFORMANCE MEASURES ............................................................................................................... 3
KEY FINDINGS ................................................................................................................................... 4
APPENDICES ...................................................................................................................................... 6
APPENDIX A: Review of Research Relevant to Arts for Life ..........................................................................6
APPENDIX B: Summary Memo of Stakeholder Survey .................................................................................8
APPENDIX C: Arts for Life Survey Protocol ..................................................................................................9
APPENDIX D: Arts for Life Survey .............................................................................................................. 11
APPENDIX E: Arts for Life Survey Reponses ............................................................................................... 13
3
BACKGROUND
In 2019, Arts for Life (AFL) partnered with the Culture of Results (COR) Initiative of the North Carolina
Center for Health and Wellness to complete a program evaluation. The COR team applied an
empowerment model of evaluation to iteratively develop performance measures and draft a survey to
evaluate AFL’s programs. The purpose of this evaluation was to understand how AFL services impact
key customers and to support AFL in growing their programs, improving services, and building
sustainability.
Culture of Results uses Results-based Accountability (RBA), an evidence-based framework for program
planning and evaluation. This report follows the RBA format of reporting on headline performance
measures: How much did we do?, How well did we do it?, Is anyone better off?
The desired results of the program evaluation were to: 1) Better understand short- and long-term
impact of programs on key customers using quantitative and qualitative information, 2) Produce
product(s) that can be used to communicate outcomes to funders and 3) Prepare evidence that
positions Arts for Life within the field of arts in health.
PERFORMANCE MEASURES
Through background research on AFL programs and a search of scientific research in the field, COR
identified potential performance measures to include in this evaluation. A summary of the background
research and literature review that informed this list of measures can be found in Appendix A. The
following possible performance measures were identified prior to seeking stakeholder input:
4
By means of a survey (Arts for Life Performance Measures Identification and Prioritization) shared with
board members, staff, volunteers, clinical partners, family members/patients, donors, funders, and
other stakeholders, performance measures were identified and prioritized in order of importance.
Stakeholders also offered other key measures they felt were missing. The survey received a total of 39
responses representative of the AFL community. Results of the stakeholder survey can be reviewed in
greater detail in Appendix B.
The following were the final prioritized performance measures that informed the program evaluation:
These measures were used to develop the Arts of Life Survey that was delivered to program
participants. The protocol for collecting survey data is outlined in Appendix C. The Arts for Life survey
and all responses can be found in Appendices D and E respectively.
KEY FINDINGS
How much did we do?: Numbers Served & Frequency of
Services Received
In total, 487 survey responses were received between September
2019 and January 2020. Nearly 63% of respondents were patients
and 21.5% were siblings of patients. The remaining respondents
were parents, friends, staff, and other family members. Most
participants (74.7%) were between 5-12 years old. Nearly 60%
Are you a patient or a family member?
5
were white, 21.9% African-American/Black and 10.4% Hispanic. More participants were female (62.1%)
than male (37.7%). About half of all respondents (49%) were from Asheville: Rueter and the Winston
Salem-7
th
floor program locations, with the other half being split the remaining 10 locations. Almost all
participants (93.2%) were active in the craft or art studio.
For many respondents (39%), this was the first-time making art or music with Arts for Life. One fourth
of respondents participated once in a while, and 35.5% participate often or all of the time.
How well did we do it?: Customer
Satisfaction & Inclusivity
A majority of respondents (82.9%)
always enjoyed doing art projects with
Arts for Life while being patients or
family members/friends of patients.
Over 80% of respondents felt that
making art or music, always made
them feel included and respected.
Is anyone better off?: Various Outcomes
Many participants (76.6%) reported higher satisfaction with clinical care, stating art or music always
makes being at the hospital or clinic better. More than half of patients (62.1%) had improved health
outcomes agreeing that “When I make art or music, my body always starts feeling better.” More than
half (58.1%) reported lower anxiety, always starting to feel less
nervous or scared and 82.4% always starting to feel happier
when making art or music
A large majority (79.7%) of participants said making art or music
always let them express themselves. When feeling bad, 61.1%
always agree that they can start doing something that will make
them feel better. Most participants (73.1%) always enjoyed
spending time with other people while making art or music
All respondents had positive comments in reaction to the final survey item,
“Why do you do art or music with us?”. For a complete list of open-ended responses, please see the
“Arts for Life Survey Responses” Excel sheet attached. Exceptional comments have been pulled out
below:
“Because it makes me feel happy and welcome. Doing art always makes me feel
better. I can forget everything that happened. I really like doing art.”
“It helps me not be as scared or nervous about being here or at a doctor’s office/clinic.”
“My body feels more happy when I do art. It's always something I can do to make
myself feel better. Because it makes me want to stay at the art table and not leave.”
“It gives me something to do when I'm hurting. And it's real fun.”
I enjoy doing these art projects.
When I make art or music, I feel
included and respected.
When I make art or music, it makes being at the
hospital /clinic better.
6
APPENDICES
APPENDIX A: Review of Research Relevant to Arts for Life
By: Cat Turbyfill, MPH Candidate, NCCHW Research Assistant &
Emma Olson, MPH, LMSW, NCCHW Director of Partnerships and Evaluation
Methods
The Culture of Results team conducted research to learn about the effectiveness and impact of art
therapy on clinical patients. We primarily reviewed two systematic reviews studying various impacts of
art therapy using validated tools and conducted additional research based on primary findings. All
reviewed articles were scientific studies in clinical applications of art therapy.
Key Findings
Significant findings from these reviews included:
1) There is a strong positive association between art therapy and anxiety; it is worth noting that
different types of anxieties were measured (e.g., hospital anxiety, treatment anxiety, etcetera).
2) Impacts on depression were present but not as strong (i.e., statistically insignificant).
3) Though often researched, there were no significant correlations between art therapy and
quality of life in the systematic reviews. Other independent studies (those not included in
aforementioned reviews) revealed conflicting findings on impact of patient’s QoL – revealing
QoL may have a positive impact in some settings. As a caveat, in one study patients reported
anxiety and depressive symptoms as critically important to treatment, while QoL was rated as
important in comparison.
4) Lastly, one study revealed positive associations in measures of coping and general symptoms
of the patients; but both of these measures were not included in the two reviews and evidence
is limited.
5) Other impact measures included: mood states (positive and negative moods),
symptoms/functional assessment, blood pressure, heart rate, social behavior/social
desirability, body image, pain, coping, benefit finding, spiritual wellbeing, shoulder range of
motion, social functioning, social interaction, socialization, problem-solving abilities, locus of
control, apathy, cognitive function, memory, self-esteem, hostility, emotional content, self-
image, somatic/psychological symptoms, distress, general well-being, health-related QoL,
academic achievement, adjustment (defined by: withdrawal, troubling thoughts,
delinquency, attention, aggressive behavior psychosomatic difficulties, anxiety / depression,
7
social problems), resistance, simple response (composed of agreement, appropriate request,
and recounting), and cognitive exploration, affective exploration, insight, therapeutic change.
In addition, one study looked at factors that affect the intervention impact. Process measures for
predicting impact included: therapists’ and children’s session impression (feeling and meaning) and
therapists’ and children’s strength of bonding (agreement on tasks, agreement on goals, and bonding
with the therapist).
Validated tools for AFL performance evaluation have been compiled by the Research Assistant to
match all performance measures listed. There are a number of studies assessing art therapies but
these prove difficult to compare as populations differ across characteristics, as do outcome measures,
and instruments used.
Conclusions
This research illustrates various types of impact of art therapy on patient health and wellbeing. These
can be considered potential changes to measure in the Arts for Life program evaluation. As the impact
of art therapy on anxiety shows the strongest correlation, the AFL team should seriously consider
including possible measures of anxiety change. Depression and quality of life should also be considered
in particular.
However, the methods of these studies may differ substantially from those that will be utilized in the
program evaluation. Arts for Life may not focus on one type of impact and thus the use of a validated
tool could be challenging. In addition, the sample population may differ. For example, there will be
great variability in patient diagnosis or conditions, as well as where the patient is in the trajectory of
the treatment plan or illness. This variability could lead to differences in the impact or effectiveness of
the intervention. Additional challenges may also include not having a comparison group or pre-
assessment available.
Table of Findings: https://drive.google.com/drive/u/0/recent
8
APPENDIX B: Summary Memo of Stakeholder Survey
RESPONSE
The Performance Measure Identification and Prioritization Survey received a total of 39 responses.
Responses were representative of AFL community comprising of Board, staff (including employed
Doctors), volunteers, clinical partners, family members/patients, donors and funders.
KEY FINDINGS
The majority of respondents (86.8%) chose: “Patients feel better physically after…” as the most
important PM to capture health outcomes. “Improved general symptoms of illness and/or side effects
of treatment…” had a similarly high response rate of 81.6%, while 50% selected “Reduced need for
medication”.
There was overwhelming consensus on how to measure satisfaction: 100% selected “Improvement of
healthcare experience/quality of care…”
Attitudes/opinions/feelings had much greater variance in responses across most measures.
“Expressing self and emotions through art” received the greatest consensus at a rate of 94.9%.
“Anxiety about their treatment and/or the hospital” also received majority response (87.2%).
Most responses were varied regarding skills/behaviors performance measures. The highest response
rate of 87.2% fell on “Coping skills with effects of illness and/or treatments in patients”.
Performance measures for circumstances also varied in responses. “Social support/reduced isolation”
received the greatest response rate of 87.2%
NEXT STEPS
Our team will discuss best methods to collect data with AFL leadership. The internal survey revealed
most respondents (87.2%) agreed with survey design to capture PMs from AFL customers. The other
top two responses included observations (69.2%) and focus groups (59%).
Most headline measures appear that they could easily be captured via self-reported survey and/or
interviews with parents/caregiver while child completes art project. Our team recommends using a
survey format for ease of use and analysis. For instance, Google Forms (as used in the stakeholder
survey) provides a platform that also executes the statistical analysis of responses. While Google’s
analytics may not be as sophisticated as other software, it provides AFL with a simplistic and cost-
efficient tool. Our team believes this will enable AFL to better support their evaluation efforts, ensure
continuity, and foster sustainability in future/on-going evaluations.
To further elucidate what PMs are best suited for AFL, our team will update the Performance Measure
grid to indicate majority responses (>85%), responses under consideration (50%-84%), and some
interest (<50%). This will be sent to AFL staff to discuss further in the next meeting and determine
possible indicators for survey prototype.
9
APPENDIX C: Arts for Life Survey Protocol
Screening and Participant Eligibility
First, screen for appropriate respondents.
ASK PARTICIPANTS:
1) How old are you? (Must be 5 years or older). [If under 5, continue art project as usual.]
2) Have you filled out a survey with us recently-sometime this year? (Must answer no.)
[If YES: Great, thanks for doing that already – we really appreciate your help.] - END.
[If NO: Continue below]
Information about the Purpose and Directions
If s/he is under the age of 18 years, locate the parent/guardian of the child. Ask, Can you show me who
bought you here today?
ASK/TELL GUARDIANS:
Can your child fill out a brief survey that we can use to evaluate our program? The survey will probably
take about 3 to 5 minutes to complete, but your child can take as much time as necessary. There are
relatively no risks to participating—we don’t anticipate any questions will be upsetting, and the
answers are anonymous. The survey will help us understand our impact, continue to bring art supplies
to this hospital, and make activities even better.
If possible, your child should complete the survey independently. If s/he needs help, you can read the
questions, but try not to influence your child’s answers. We want him/her to be honest about his/her
experiences here.
Your child can ask us questions at any point, skip a question if s/he is having trouble coming up with an
answer, or stop if s/he wants to. Do you or your child have any questions? Do you consent to
participating?
IF GUARDIAN IS NOT IMMEDIATELY PRESENT, STAFF SHOULD PROCEED WITH SURVEY- TELL THE
PARTICIPANT:
Can you please fill out a brief survey that we can use to evaluate our art projects? The survey will
probably take about 3 to 5 minutes to complete, but you can take as much time as necessary. The
survey will help us understand what we do, continue to bring art supplies to this hospital, and make
activities even better.
There are no right or wrong answers – it is not a test. We just want you to be honest about how you
feel. You won’t be asked for your name and no one will know what answers you gave. We don’t think
any questions will be upsetting, but you can skip any you don’t want to answer or stop if you chose to.
We really want to know what you think. Can you help us with this important project?
AFTER THEY CONSENT ASK:
Would you like to fill the survey out on an iPad or a piece of paper?
Get the appropriate item, complete the first portion, then ask:
10
Would you like to complete the rest on your own or with help from your parent/guardian or me?
Hand the survey to them or support them in completing it.
If they are completing it independently, stay nearby for questions.
If they are completing an electronic version, ask them to press submit when they finish. If they are
completing a hard copy, ask them to put the finished survey in a nearby envelop.
Thank them for their time.
If they are curious, they can learn about highlighted results of the survey in the next Arts for Life
annual report.
11
APPENDIX D: Arts for Life Survey
Program Evaluation Survey 2019
Directions: Circle the option that best describes the participant.
General Information
1. What is the program location?
a) Asheville: Zeis
b) Asheville: Reuter
c) Asheville: Inpatient
d) Charlotte: Playroom
e) Charlotte: Bedside
f) Durham: Clinic Waiting
Room
g) Durham: VDH
h) Durham: PBMT
i) Durham: Inpatient
j) Winston-Salem: 9PHO
Clinic
k) Winston-Salem: 9PHO
Inpatient
l) Winston-Salem: 7
th
Fl.
2. What type of activity are you doing?
a) Art Studio
b) Craft Studio
c) Music
d) Creative Writing
e) Art Kit
3. Are you a patient or a family member?
a) Patient b) Sibling c) Other______
4. How old are you?
a) 5-8 b) 9-12 c) 13-15 d) 16-18 e) 18 and above
5. What is your gender?
a) Male b) Female c) Other: _______________________________
6. What is your race/ethnicity?
a) African American/Black
b) Asian
c) Middle Eastern
d) Hispanic/Latino
e) Native American
f) Pacific Islander
g) White/Non-Hispanic
h) Multiracial
7. How often have you made art or music with us?
a) It is my first-time b) Once in a while c) Often d) All the time
Turn over the page and pass it to the participant.
12
Program Evaluation Survey 2019
Directions: Circle the option that best describes you.
Program Information
1. I enjoy doing these art projects.
Never Rarely Sometimes Always
2. When I make art or music, it makes being at the hospital/clinic better.
Never Rarely Sometimes Always
3. When I make art or music, I feel included and respected.
Never Rarely Sometimes Always
4. When I make art or music, my body starts feeling better.
Never Rarely Sometimes Always
5. When I make art or music, I start feeling less nervous.
Never Rarely Sometimes Always
6. When I make art or music, I start feeling happier.
Never Rarely Sometimes Always
7. Making art or music lets me express myself.
Never Rarely Sometimes Always
8. When I’m feeling bad, I can usually start doing something that will make me feel better.
Never Rarely Sometimes Always
9. I enjoy spending time with other people while I make art or music.
Never Rarely Sometimes Always
10. Why do you do art or music with us?
Comments: __________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
DRAFT
13
13
APPENDIX E: Arts for Life Survey Reponses
GENERAL INFORMATION
DRAFT
14
14
DRAFT
15
15
PROGRAM INFORMATION
DRAFT
16
16
DRAFT
17
17
DRAFT
18
18
Why do you do art or music with us?
See “Arts for Life Survey Responses” sheet attached for all participant comments.
