Plan, Evaluate and Improve Your Health Literacy Interventions and Activities Sabrina Kurtz-Rossi, M.Ed. Health Literacy Consultant April 10, 2013
Objectives After attending the session, participants will be able to: Conceptualize a health literacy program evaluation Write measurable health literacy outcome objectives Identify indicators to assess both process and outcomes Identify user-friendly evaluation resources for future use Olney, C. and Barnes, Susan (2006). Planning and Evaluating Health Information Outreach. NN/LM Outreach Evaluation Resource Center. Seattle, WA.
Ten Attributes of Health Literate Health Care Organizations Brach C, Keller D, Hernandez LM, et al. Ten Attributes of Health Literate Health Care Organizations. Washington DC: National Academy of Sciences. 2012
Case Study Providence, RI Dorcas Place Adult and Family Learning Center Computer Lab Western Maine Auburn Public Library Computer Training Class
Research vs. Evaluation Research: To draw causal links between observed phenomena and to add to the knowledge base of the field Evaluation: To demonstrate effectiveness, and determine what works and what does not for the purpose of program improvement Boulmetis, J and Dutwin, P (2005). The ABCs of Evaluation. Jossey-Bass. San Francisco, CA
Health Literacy Program / Evaluation Planning Step 1: Know Your Audience Step 2: Identify Goals and Objectives Step 3: Develop Program (and evaluation) Plan Step 4: Implement and Evaluate Step 5: Apply and Share Results
Step 1: Know Your Audience Start with an Assessment Identify needs in context Guide your program Build on already existing capacity Identify champions within your organization Methods Informal discussion, observation Formal Key informant interviews, focused group discussions, survey
Attribute 1: In a Health Literate Organization leadership makes health literacy integral to its mission, structure, and operations.
Case Study Methodology: Leaders at community-based organizations interviewed. Semi- structured interview protocol designed to explore community confidence in using the Internet to find health information and local resources to support access. All interviews conducted in person. Data were coded and analyzed into themes. Key Findings: Maine has a growing senior population and community-based organizations support public health efforts. Public schools, adult education, and libraries offer computer access and support (n=9). Latinos get their health information from sources they know and trust, including the radio. A network of minority health programs offer computers access, training and support (n=10).
Step 2: Identify Goals & Objectives Goal -- Why you are doing what you are doing and for whom Example: Increase use of health literacy interventions Improve community access to reliable online health information
Attribute 2: A Health Literate Organization integrates health literacy into planning, evaluation, patient safety, and quality improvement.
Case Study Focused Health Information Outreach Project Goal: Increase access to health information among communities experiencing health disparities National Network of Libraries of Medicine New England Region (NN/LM NER) http://nnlm.gov/ner/http://nnlm.gov/ner/
Step 3: Develop Program(and Evaluation) Plan Logic Model: A planning tool connecting activities to outcomes If you have these resources ResourcesActivities/OutputsOutcomes and conduct these activities you will get these outcomes
Why a Logic Model? Provides a framework for planning Helps organize your thoughts Clarifies intended outcomes Links activities to outcomes Acts as a communication tool ActivitiesResources OutputsOutcomes
Attribute 3: A Health Literate Organization prepares its workforce.
Case Study Goal: Increase access to health information among communities experiencing health disparities ResourcesActivitiesOutputsOutcomes Increase knowledge of NLM resources Increase use of NLM resources Increase community experts to help others Number of trainings held Number trained Types of materials developed Number of materials distributed Develop training materials Distribute materials at community events Consultant Outreach coordinators NLM materials NLM funding Conduct trainings Working student Librarians in communities
Activity Choose a goal (or use your own) To increase staff awareness and use of health literacy interventions To improve community access to reliable online health information To improve health information literacy skills among patients and their families Complete the logic model worksheet - identify outcomes, activities, outputs, and resources
Attribute 4: A Health Literate Organization involves the population served in evaluation of health information and services.
Case Study Community Assessment Key informant interviews Program Tailored efforts based on findings Community-based partnerships Evaluation Outcome assessment: pre-post training evaluations, follow-up assessment Process assessment: story-based evaluation, community partner exit interviews Western Maine (rural) Providence, RI (Latino)
Evaluation Plan Process To determine what worked and what didnt For program improvement Outcome To demonstrate effectiveness, value, and influence For comparison purposes, to determine which programs are more effective
Attribute 5: A Health Literate Organization meets the needs of the populations and avoids stigmatization.
Process Assessment Methods Activity logs Teacher feedback forms Focused group discussion Interviews
Attribute 6: In a Health Literate Organization providers and staff use health literacy strategies in verbal communication and confirm understanding.
Case Study Outcomes (what you want)Indicators (how you will know) Increased knowledge of NLM resources Increased use of NLM resources Increased community experts to help others Will report use two weeks later of follow up Will name resources on post- surveys Will document helping others on story-based evaluation forms Outcomes & Indicators
Measurable Objectives Four components of a measurable objective 1. Condition Following the training After watching the video 2. Who the learner the patient the caregiver
Measurable Objectives 3. Performance (knowledge, attitudes, behavior) will name will agree will use 4. Criterion (quality/quantity) 5 out of 6 at least 2 the correct way
Characteristics of a Measurable Objective Is audience focused, not intervention focused (the participant will) Includes action verbs Targets one expectation Includes observable criteria for evaluation Matches goals to activities/strategies
Attribute 7: In a Health Literate Organization consumers have easy access to health information and services and navigation assistance.
Verbs to Avoid Know Comprehend Understand Appreciate Study Gain knowledge of Learn
Attribute 8: A Health Literate Organization develops plain language materials that are easy to understand and act on.
Case Study ConditionWhoPerformanceCriterion Immediately after training participantswill name Two weeks after training participantswill report using Three months training participants NLM resources on post-survey NLM resources on follow up interview will write a story Describing how they helped others find information on story-based evaluation form Writing Measurable Objectives
Activity Choose an outcome (or use your own) Increase confidence in using plain language/teach-back Increased awareness of available online health information resources Increased interaction with diabetes educators Complete the writing measurable objectives worksheet - identify condition, audience, performance, and criterion
Step 4: Implement and Evaluate Expect the Unexpected Be flexible Adjust to the environment Address learners needs Tweak your notion of success
Attribute 9: A Health Literate Organization addresses health literacy in high-risk situations, including transitions and about medicines.
Case Example Rhode Island: Pre/Post: Confidence in Ability to Find Health Information on the Internet Pre N= 52 0 Post N=503 Maine: Pre/Post: Ever Used NLM Resources Pre N=72 Post N=55
Case Example Successes Training participants shared resources with friends and neighbors Partners will continue to promote and distribute resources Providers used resources to find accurate information to use in their work Challenges Pre- and post-evaluations were hard to administer
Step 5: Apply and Share Results Apply Lessons Learned Make changes based on what you learned Replicate with similar or different audiences Share Your Findings Partners Colleagues
Attribute 10: At a Health Literate Organization they clearly communicates about health plans and health care costs.
Evaluation Resources Planning and Evaluating Health Information Outreach Projects Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach http://nnlm.gov/evaluation/guide/index.html
Questions & Comments Sabrina Kurtz-Rossi, M.Ed. Principal | Health Literacy Consultant 781-835-6488 firstname.lastname@example.org Kurtz-Rossi & Associates Health*Literacy*Education*Evaluation www.kurtz-rossi.com Member, Clear Language Group www.clearlanguagegroup.com Health Literacy Leadership Institute, June 10-14, 2013 http://healthliteracyleadreship.com