|
Table of Contents
LAYING THE FOUNDATION-THE EVALUATION PLANNING PROCESS Evaluation is an integral part of the educational programming process. To make evaluation useful in program improvement, the evaluation plan should be drawn at the beginning of the program instead of waiting until the end. Program planning and evaluation are interconnected activities of the program cycle as illustrated in Diagram 3.
Diagram 3. Educational Programming Cycle The educational programming process starts with the needs assessment of the target participants. Prioritizing needs then leads to defining program objectives. Once program objectives are defined, program development takes place. Educational materials and activities are planned to achieve objectives and meet participant needs, while parallel to this process, evaluation is planned as a guiding tool to determine whether the program achieves its objectives. Next, when the program is ready to be delivered, the evaluation plan should be in place to allow easy collection of impact and program/process evaluation data and information, as well as organic adjustments to the program that facilitate learning. Finally, by planning evaluation at the beginning of the program, financial educators will be able to use the collected data to assist with ongoing monitoring of the program and make program changes to ensure it achieves objectives. Monitoring programs is helpful to maximize the impact and achieve the cost effectiveness of programs. The Importance of Evaluation Data Evaluation data is used to make programmatic decisions such as expansion, continuation, reduction, or termination of the education program. If an educator does not plan for evaluation at the beginning of the program, the opportunity is lost to record program outcomes and understand the contributing factors to those outcomes. As a result, the educator is unable to make accurate decisions about the program. If there is no evaluation, program monitoring will not be possible because there is no base for making programmatic decisions. Therefore, further improvement of the program will not be possible. Impact evaluation allows educators, administrators, and funding agencies to make decisions about the program based on objective data. Conducting a meaningful evaluation is not possible without having clear program objectives because objectives are used as the evaluation criteria. The program outcome is compared with objectives to determine the worth or merit of the program. Therefore, it is important to develop program objectives very clearly. Program objectives should be created in a SMART format to guide program planning, delivery, evaluation, and monitoring. The acronym "SMART" refers to the following characteristics of objectives. The objective should be:
Each objective should be specific in terms of targeting participants and the planned change. This planned change should be measurable and reasonably achievable with the available resources within a specified time period. The more specific the objective in terms of the participant and intended change, the processes of program development, delivery, evaluation, and monitoring become clearer. If the participant group is specific, the financial educator can develop educational materials to meet their specific learning needs and learning abilities. If the change is specific, the educator can focus educational activities and evaluation to facilitate and document that change. Writing objectives is an art that requires practice. Clearly written objectives indicate what learning materials are needed and what changes can be anticipated. These anticipated changes in the participants indicate the potential impact of the program. Below are two sample programs, followed by several objectives relevant to the program.
As mentioned earlier in this manual, an impact indicator is a reasonable and meaningful measure of intended learner outcome. Examples of impact indicators are a change of participants' financial knowledge and practices. Program impact indicators vary with the type of program and its objectives. Longer programs have greater potential to create higher numbers of impact indicators than do short programs because participants have more learning opportunities. The following steps are helpful to identify appropriate impact indicators for different financial education programs. Step 1. Determine the Levels of Potential Impact Based on the Type of Program Financial education programs can be classified into the following three broad categories for the purpose of planning program evaluations. These classifications are intended as guidelines to help program educators conduct the most efficient and effective evaluation of their program. Short programs (for example, one-time, short presentations). In such short programs, there is no time to conduct pre and post-evaluations. Generally, this type of financial education program lasts less than two hours and has the potential to generate short-term and intermediate-term impacts. These impacts include participants' change of knowledge, confidence, and aspirations for changing behaviors. If the program is effective, there is potential to create intermediate-term impacts such as participants' behavior changes, and sometimes goal achievements. Long programs (for example, a day-long workshop). These programs are long enough to conduct pre and post-evaluations. Normally, this type of educational program lasts two hours or longer and has the potential to create short-term, intermediate-term, and long-term impacts. These impacts include participants' change of knowledge, skills, aspirations, behaviors, and socio-economic conditions. Multi-session programs (for example, a training program with a series of workshops). These programs are long enough to observe the behavior changes of participants. This type of educational program has the potential to generate intermediate and long-term impacts such as change of participants' financial practices and socio-economic improvements. Table 2 summarizes the potential impact of different financial education programs. Long-term impact is most likely to be achieved with high quality programming that maintains contact with the individual over a period of time. While short, one-time, programs may also result in long-term impact, it is less likely. Table 2 describes what type of impact and observable outcomes are expected based on the type of program. Table 2. The potential impact of different financial education programs
Step 2. Use Program Objectives to Identify Potential Impact Indicators Short programs. If the program is a short training workshop on saving, and the objective is to encourage participants to be aware of savings and motivate them to save, the impact indicators can be the participants' knowledge about saving, their confidence about saving money, and their readiness to begin saving habits. These are immediate impacts. Long programs. If the educational workshop is a relatively long program on saving and the objective is to change participants' saving habits, the impact indicators can be of the participants' knowledge and confidence about saving money, their readiness to save money, and their actual saving habit changes. Multi-session programs. If the financial education program has a multi-session training approach focused on learning to save to buy a home, and the objective is to help first-time homebuyers save for a down payment, the impact indicators can be the participants' actual saving habit changes, the amount of money saved, and the number of participants who reached their goal. Note: In multi-session programs, the educator meets the participant more than once; therefore, the educator has a greater opportunity to build trust with participants. Trust is necessary for collecting sensitive evaluation data such as the amount of money saved. Even if financial educators ensure the confidentiality of data, participants may not be comfortable reveling sensitive data until educators have built a relationship of trust with the participants. If the higher-level impact of the program can be documented, there is less need to document the lower-level impact because higher-level impact doesn't take place without reaching the lower-level impact. Determining the Type of Evaluation for Each Program The type of evaluation is mainly determined by the type of financial education program and the resources available to conduct evaluation.
Evaluation of One-Time, Short Programs Advantage. The post-only evaluation is easy to conduct. Disadvantage. Since there is no pre assessment, documenting changes as a result of the program in knowledge, attitudes, or aspirations is more difficult. If the program is comprehensive enough to generate higher-level impact and has adequate resources, a follow-up evaluation is an option to document the intermediate and long-term impacts of the program. The follow-up evaluation is conducted after some time has elapsed since the program concluded. This time period is normally three or six months after the program and is intended to document lasting change.
Evaluation of Long Programs Advantage. Since there is a pre evaluation, it is possible to compare pre and post evaluations to document changes. Disadvantage. It takes more time to administer pre and post evaluations. Participants must be identified for matching pre and post surveys. However, assigned identification numbers can be used instead of participants' names to match pre and post surveys.
Evaluation of Multi-Session Programs Advantage. Stages to Change Evaluation is helpful to document intermediate impacts. Disadvantage. This type of evaluation is possible only with multi-session programs. If resources are available, follow-up evaluation is a good option to document long-term impact of the program. For multi-session programs, educators could also use evaluation methods similar to those used for long programs. However, the Stages to Change Evaluation method has greater potential for richer summative and formative evaluation information.
Evaluation of Train-the-Trainer Programs Important Factors in Designing Evaluation Instruments The quality of the evaluation instrument significantly contributes to the quality of evaluation data gathered. The following factors are important guidelines in designing quality evaluation instruments.
Collecting Necessary Data and Information Determining Required Data and Information. The purpose of evaluation is to improve programs and document the impacts. The educator must, therefore, limit the amount of data and information collected to fulfill this evaluation task. The required evaluation data and information can be categorized into the following two groups:
Limiting the Number of Questions. Questions should be limited to those necessary for assessing program impact and the educational program delivery process. It is also important to keep evaluation questions short and simple to facilitate data collection. Adding any unnecessary questions may confuse the participant, impair the reliability of the instrument, and reduce the response rate. Collecting Quantitative Data and Qualitative Information. Quantitative data refers to numbers and qualitative information refers to non-numeric data such as stories, short written responses, and expressions made by the participants. There are strengths and weaknesses associated with both quantitative data and qualitative information. For example, quantitative data generally provides strong evidence for program accountability purpose. However, quantitative data does not reveal reasons for not achieving or exceeding program objectives. Qualitative information is needed to find reasons for those. Due to this, the quality of evaluation can be improved by combining both quantitative data and qualitative information to complement each other. Use of impact data coupled with success stories is often considered one of the best methods for documenting educational program impact.
Accuracy and Reliability of Data Clarity of Questions. Most of the time, evaluation tools are self-responding surveys. Therefore, evaluation questions should be written clearly and concisely to avoid ambiguity and help the participant answer accurately. Instructions should be clearly stated to help participants complete the survey easily. Generally, use close-ended and open-ended questions in evaluation instruments. If the possible response choices are provided as a list with the question, this is a closed-ended question. If the question is asked so that the respondent must use his or her own words to answer it, this is an open-ended question. When close-ended questions are used, the educator should be sure that the answer key contains all the possible responses to prevent response errors. Since open-ended questions are exploratory, they are appropriate for the process evaluation. It is important to ask easy questions at the beginning and harder questions at the end to facilitate the responding process. Reading Level of Target Audience. The reading level of questions is an important determinant of the accuracy and reliability of evaluation data. The reading level of the written language used to design the evaluation tool should not exceed the reading level of the target participant group to avoid potential errors in data collection. Sensitive Data and Information. Collecting sensitive information such as age, income, and so on can be somewhat challenging because generally participants do not like to reveal this data. Therefore, it is important to get this data in a way that the participant is comfortable in providing the information. One approach is to present question-and-answer choices that have ranges instead of exact values of sensitive data. For example, instead of asking "what is your annual household income," the question might be "in what range is your annual household income" and list the possible income categories the participant can choose from. Designing Evaluation Instruments The following three steps are helpful in designing evaluation tools.
Step 1. Collect Data and Information for Impact Indicators Testing Participants' Knowledge. Knowledge change is the most common impact indicator in any financial education program, and it can be recorded by asking questions related to the content of the program. The educator may use either multiple choice questions or true/false questions. The true/false question format is suitable for low literacy audiences. Use of a true/false format is helpful to keep the evaluation tool brief and save time needed to respond. Use of knowledge testing questions with true/false format is appropriate to evaluate programs presented to youths because knowledge acquisition is a reliable impact indicator for youth programs. The higher the number of questions used, the greater the reliability of measurement; however, as the number of questions increases, the response rate lowers. Due to this reason, the educator must come to a middle ground between reliability and practicality. Normally, 10 to 25 questions provide a reasonable level of accuracy. Below is an example of questions used to test participants' knowledge. See page 42 in Part IV for more information on this evaluation impact indicator. Example: Testing Knowledge
Testing Participants' Confidence. The confidence to carry out a financial management task is a reflection of one's financial management skills. The following question format can be used to assess participants' confidence to apply financial practices. See page 43 in Part IV for more information on this evaluation impact indicator. Example: Testing Participants' Confidence
Please circle the number that best describes your confidence to do the following:
Recording Participants' Attitudes. A scale to assess participants' attitudes can be developed by using value statements related to the financial practices that the program is planning to teach and then recording participants' responses to those statements on a 5-point Likert-type scale as illustrated in the following example. Compared to designing evaluation instruments for other impact indicators, designing a reliable scale for attitudes is a time-consuming, professional task. Due to this reason, it is advisable to use other indicators instead of using attitude as an impact indicator. Example: Recording Participants' Attitudes
Please circle the number that best describes your levels of agreement with each of the following statements.
Testing Participants' Aspirations for Changes and Taking Charge. This is a good impact indicator to document the participants' preparedness to apply learned financial management practices at the end of the educational program. These are the financial behaviors that the educator wants participants to practice and adopt. If the program is effective, participants will show their readiness to adopt these financial practices by planning to implement them. Generally, the objectives of financial education programs are to guide participants to adopt desired financial management behaviors. A common term used among financial educators for this type of planned action is "taking charge." That means taking charge of their financial circumstances by planning to adopt proactive financial practices. These practices represent the objectives of financial education programs. The following format is appropriate to record participants' aspirations to adopt desired financial practices. See page 43 in Part IV for more information on this evaluation impact indicator. Example: Testing Participant's Aspirations for Change and Taking Charge
Please circle the number that best describes your answer.
Testing Participants' Financial Behavior Changes. Recording participants' actual behavior change is possible only if it is a multi-session financial education program. For all the other financial education programs, follow-up evaluation is needed to assess participants' actual behavior changes. In a multi-session financial education program, the educator meets the same group of participants more than once. As a result, the educator has an opportunity to record the participants' financial behavior related to the content of the program before and at the end of the program series to document changes in behavior. The evaluation tool designed to record participants' financial behavior is based on the Transtheoretical Model. The Stages to Change evaluation tool uses the previously described five stages of change to assess the financial behavior of program participants. The following format is used to record the participants' stages of the change related to emphasizing financial behavior. See page 44 in Part IV for more information on this evaluation impact indicator. Example: Stages to Change Evaluation
For each financial practice, please circle the number that best describes your current behavior.
The Transtheoretical Model of change is helpful to understand and facilitate the behavior changing process. If the educator records the stage of the participants' behavior at the beginning of the educational program, he or she will be able to understand the composition of the group in terms of their behavior changing stages. If most of the participants' responses fall into the first column ("I am not considering this"), it indicates that the instructor has a reasonably challenging task to change their behavior. However, if the majority of responses fall into the middle category of response, it is relatively easy to facilitate the desired change. By recording participants' stages of change on this scale at the beginning, in the middle, and at the end of a multi-session program, the educator can compare the behavior changes at different stages of the program and document the impact of the program. By comparison, the following question format can be used for follow-up evaluations.
Example: Sample Follow-up Question Format
Since completing the program, how often do you practice the following financial practices?
Please list other changes you have made in your financial practices.
The Transtheoretical Model of Behavior Change This model of human behavior change was developed by Dr. James Prochaska and Dr. Carlo C. DiClemente at the University of Rhode Island. This model has been used extensively to promote health behavioral changes in areas such as smoking, diet, substance abuse, eating disorders, etc. According to the transtheoretical model, people go through a series of changes before they adopt the desired behavior. Prochaska and DiClemente (1984) describe five stages to change in their Transtheoretical Model and termed those stages as "precontemplation, contemplation, preparation, action, and maintenance." Precontemplation stage. An individual in this stage does not have any interest to change his or her behavior related to the desired direction in the near future. Those who are in this stage are not aware of the need for a change toward the desired direction. For example, someone who doesn't keep track of his or her spending will not show any interest in doing so if that person is at this stage. The main characteristic of the individuals in this stage is their ignorance about the need for a change to follow the desired behavior, and they might say, "I'm not considering this." Contemplation stage. Those who are in this stage are aware of the problem and are considering whether there is a need to correct the problem. At this stage, individuals compare the benefits of change versus maintaining the current behavior. For example, a person in this stage, who is not keeping track of spending, will tend to assess the pros and cons of keeping track of spending. A person in this stage might say, "I'm considering this." If someone is convinced that the change is more beneficial than maintaining the current behavior, he or she will move to the next stage of change. Preparation stage. Individuals in this stage have been convinced about the need for change and look for information and ways to fix the problem. This is the teachable stage to facilitate the change of desired behavior. Those who are in this stage try different ways to solve the problem. For example, a person in this stage, who is not keeping track of spending, will try some methods to keep track of spending and will say "I'm doing this sometimes." Action stage. Those in this stage take sustained actions to change their behavior and therefore fix the problem. They actively engage in changing their behavior toward the desired direction. For example, a person in this stage, who was not keeping track of spending, will implement different methods to keep track of spending and will say, "I'm doing this most of the time." Maintenance stage. If someone is in this stage, he or she continues the change as a desirable behavior. When a person moves into this stage, the change has been fully integrated into his or her life. For example, a person who previously was not keeping track of spending will say, "I'm doing this all of the time." Recording Economic Impact. The following format is useful to help participants record their own economic progress. If the educator has ability to do a follow-up, this recording sheet can be used to evaluate the economic impact of financial education programs. See page 45 in Part IV for more information on this evaluation impact indicator. Example: Recording Economic Impact
Please indicate how your overall financial position has changed since completing the program.
Example: Recording End-Results
Please indicate your financial position based on where you currently are in the program.
Step 2. Collect Program Improvement Data and Demographic Information This step includes demographic questions and program delivery process evaluation questions. Demographic questions should be selected based on the program need and would be in addition to the number of evaluation questions previously suggested. For example, if the program must reach a targeted ethnic group or needs to justify reaching every ethnic group, the ethnicity of participants needs to be collected. If the program is to be designed to select the best delivery method or educational materials for participants with different academic levels, data must be collected related to participants' levels of education. In order to assess initial motivation of participants, it may be necessary to collect data on whether participants are attending the program voluntarily. The program delivery process evaluation section is used to gather participants' ratings of the instructors, educational materials, and the overall program. By using the following question format, participant satisfaction can be assessed. Participants' ratings of the overall program can be considered as an impact indicator, also. Closely associated with program improvement information is participant demographic information. It is essential to gather demographic information regarding program participants to ensure that the program is reaching its intended target audience. Because funding and program development are often associated with serving some target population, some data needs to be gathered to show that the program has reached the intended audiences. Participant demographic questions regarding age, income, education, family status, race and ethnicity, and others will help address this information need and will illustrate who is being reached through the program, or whether the program marketing needs to be redirected toward an intended audience. Example: Program Improvement Data and Information Please rate the instructor(s), materials, and the overall program by circling the appropriate number.
The following types of open-ended of questions can be used to collect process evaluation data and information needed to improve the program.
What did you like the most about this program?
What did you like the least about this program?
How could this program be improved?
Would you recommend this program to others?
Comments or suggestions about the program:
Step 3. Facilitate Data and Information Collection
Tips for Easy Evaluation Tool Design Designing quality evaluation tools is a professional task and requires experience. The NEFE evaluation database makes it easy for financial educators to design professional caliber questions and tools based on their local program needs. Parts IV and V of this manual provide a step-by-step exploration of the online database and tips on reporting program impact. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
