Contents PART I · PART II · PART III · PART IV · PART V ACKNOWLEDGMENTS INTRODUCTION Importance of Program Evaluation Using the Financial Education Evaluation Toolkit How to Use the Manual PART I. FINANCIAL EDUCATION OVERVIEW - THE IMPORTANCE TO FINANCIAL EDUCATORS Major Topics in Financial Education Key Components of Every Financial Education Program Target Audiences Methods of Financial Education Delivery Participants' Rights and IRB Requirements PART II. UNDERSTANDING FINANCIAL EDUCATION PROGRAM EVALUATION Learner Achievements Financial Education Impact Impact Indicators Predicting Program Impact Using the Logic Model Impact Hierarchy The Significance of Impact Hierarchy PART III. LAYING THE FOUNDATION—THE EVALUATION PLANNING PROCESS Where to Begin The Importance of Evaluation Data Writing Program Objectives Identifying Impact Indicators Step 1. Determine the Levels of Potential Impact Based on the Type of Program Short programs Long programs Multi-session programs Step 2. Use Program Objectives to Identify Potential Impact Indicators Short programs Long programs Multi-session programs Determining the Type of Evaluation for Each Program Evaluation of One-Time, Short Programs Advantage Disadvantage Evaluation of Long Programs Advantage Disadvantage Evaluation of Multi-Session Program Advantage Disadvantage Evaluation of Train-the-Trainer Programs Important Factors in Designing Evaluation Instruments Collecting Necessary Data and Information Determining Required Data and Information Limiting the Number of Questions Collecting Quantitative Data and Qualitative Information Accuracy and Reliability of Data Clarity of Questions Reading Level of Target Audience Sensitive Data and Information Designing Evaluation Instruments Step 1. Collect Data and Information for Impact Indicators Testing Participants' Knowledge Testing Participants' Confidence Recording Participants' Attitudes Testing Participants' Aspirations for Changes and Taking Charge Testing Participants' Financial Behavior Changes Recording Economic Impact Step 2. Collect Program Improvement Data and Demographic Information Step 3. Facilitate Data and Information Collection Tips for Easy Evaluation Tool Design PART IV. PUTTING THE EVALUATION DATABASE TO GOOD USE - APPLICATION OF EVALUATION OPTIONS Introduction to the Database Functionality Overview of the Database Decide the Type of Evaluation Post Evaluation Only Pre and Post Evaluation Stages to Change Evaluation Train-the-Trainer Evaluation Decide Impact Indicators Testing Knowledge Indicators Building Skills/Confidence Taking Charge Indicators Financial Behavior Indicators Financial Position Indicators Select, Add, or Edit Knowledge Questions and Behavior Statements Select, Add, or Edit Open-Ended Questions for Qualitative Information Select, Add, or Edit Demographic Questions Evaluation Instrument Blueprint Choose the Evaluation Option Step 1: Program Information and Follow-up Selection Step 2: Select Knowledge Questions Suggestions for Minimum Knowledge Question Selection Customizing Knowledge Questions Changing the Order of Questions Step 3: Choose Confidence and Behavior Indicators Step 4: Select Statements for Behavior and Skill Change Step 5: Select Program Questions Post Evaluation Only and Pre and Post Evaluation Stages to Change Evaluation Train-the-Trainer Evaluation Step 6: Select Program Participant Information Step 7: Follow-Up: Financial Progress Indicators Step 8: Follow-Up: Personal Achievements Step 9: Follow-Up: Additional Participant Information Questions Option 1: "Yes, using the same demographics questions I selected earlier" Option 2: "Yes, selecting different demographics questions" Option 3: "No" Step 10: Create Evaluation Edit and Customize the Instrument Example: Evaluation Tool Created by the Database and Printed After Editing PART V. BUILDING PROGRAM SUCCESS—REPORTING PROGRAM IMPACT Collecting Evaluation Data Building Evaluation Capacity Planning Evaluation in Advance and Allocating Necessary Resources Providing Instructions to Complete the Survey Motivating Participants to Complete the Evaluation Challenges to Conducting Follow-up Evaluations Need More Resources Difficult to Locate Participants Due to Relocations Low Rate of Response Keep the evaluation tool short, clear, and easy to respond to. Limit open-ended questions to a minimum. Conduct follow-up before participants forget about the program Contact non-respondents for reminders Personalize the request letter Analyzing and Summarizing Quantitative Data Summarizing Post Evaluation Only Survey Data Participants' rating of the instructor, materials, and the overall program Participants' knowledge about the content Participants' level of confidence to follow financial practices Participants' intended financial behaviors Demographic data Summarizing Pre and Post Evaluation Survey Data Analyzing and summarizing participants' change of knowledge and confidence Summarizing Stages to Change Evaluation Survey Data Summarizing Follow-up Evaluation Survey Data Analyzing and Summarizing Qualitative Data Identifying Key Findings Presenting Key Findings Utilizing Evaluation Findings Accountability Program Improvement Funding Partnerships APPENDIX: SAMPLE EVALUATION INSTRUMENTS SAMPLE A: POST EVALUATION ONLY SAMPLE B: PRE AND POST EVALUATION SAMPLE C: STAGES TO CHANGE EVALUATION SAMPLE D: TRAIN-THE-TRAINER EVALUATION