|
Table of Contents
PUTTING THE EVALUATION DATABASE TO GOOD USE - APPLICATION OF EVALUATION OPTIONS Introduction to the Database Functionality The evaluation database is a self-guided, Web-based resource that allows educators to design evaluation instruments for financial education programs. It allows educators to choose pre-written measurement questions from the database as well as add their own questions to create a fully customized evaluation instrument that meets the needs of their program and audience. While the evaluation database has been designed to be easy to use, financial educators will find it helpful to become familiar with the database structure and functions before starting to use this tool. This section of the NEFE Financial Education Evaluation manual introduces educators to the database and takes them through a step-by-step example of how to design an evaluation instrument using the database. An overview of the database functions is followed by instruction on using the database to create program-specific evaluations. Sample evaluations are included in the Appendix at the end of this manual. Before educators will be able to use the online toolkit, they must agree to the terms of use set forth in the license agreement. In addition, educators must also be sure that they are in compliance with their organization’s human subjects requirements. The evaluation database can be found at: www.nefe.org/eval.
Overview of the Database
Below are explanations of these five main options available to educators designing an evaluation instrument to fit the program they're teaching.
Decide the Type of Evaluation
Choosing the option most appropriate for the financial education program being taught by the educator is the first step in designing the evaluation instrument. Each type of evaluation is described to help the educator decide which one is the best for his or her program evaluation. Note that each evaluation option gives educators the choice to also conduct a follow-up evaluation to document actual behavior change following participants' completion of the program. Educators may also choose to include an instructor information sheet, which is designed to help them keep track of useful information to evaluate programs. Descriptions of the four evaluation options available to educators are as follows: The Post Evaluation Only option is recommended for short programs lasting two hours or less. This evaluation focuses on documenting participants' levels of knowledge, skills, and planned behaviors at the end of the program. (See SAMPLE A in the Appendix for an example.) The Pre and Post Evaluation option is recommended for programs lasting two hours or more. A pre-test is administered to participants at the beginning of the program and a post-test is administered at the end. This evaluation can be used to document changes in knowledge, skills, and planned behaviors. (See SAMPLE B in the Appendix for an example.) The Stages to Change Evaluation option is based on Prochaska's and DiClemente’s Transtheoretical Model of Change, which documents the process of actual behavior change. Specifically, this option identifies the stage at which individuals are ready and able to change their behaviors. This information can then be used to apply appropriate educational programs that are tailored to meet individuals' specific needs at that particular stage. This option is recommended only for multi-session programs (programs that meet with the same group of participants more than once over a period of time). Participants' behaviors can be recorded at the beginning, middle, and end of the program so that changes in actual behaviors can be observed. This option may include additional progress reporting and long-term follow-up. (See SAMPLE C in the Appendix for an example.) The Train-the-Trainer Evaluation option is recommended for instructor-only training programs. This evaluation is administered at both the beginning and end of a training program and is similar in format to the Pre and Post Evaluation. However, this evaluation focuses on capturing changes in knowledge, skills, and planned behaviors of the instructors rather than the participants. This option can be used to document changes in both the instructors' teaching skills and personal financial behaviors. The follow-up option documents how the curriculum materials are being used and what additional programming needs exist. (See SAMPLE D in the Appendix for an example.)
Decide Impact Indicators
The "Testing Knowledge" indicators are included in the database as a series of true/ false knowledge questions. They have been categorized under different headings and subheadings to help educators easily choose testing knowledge questions that fit best with each particular education program. Questions are sorted based on approximate reading difficulty (from simple to difficult) within each subheading. When a major heading is selected, questions will not be grouped by subheading and some duplicate questions may appear that were pulled from different subheadings. Example: Testing Knowledge Please circle your answer to each of the following statements.
Some educators may choose to include three answer choices: "true," "false," and "don't know," or a similar third option that allows the participant to answer the question without being forced to guess. For data collection and reporting purposes, responses indicating the third option should be considered incorrect. The third answer choice can be added by the educator using word-processing software after the evaluation tool is created. The "Building Skills/Confidence" indicators capture changes in financial skills and confidence levels of the program participants, and they are phrased as action statements related to financial skills. They are categorized under headings and subheadings to help educators easily choose the building skills/confidence indicators that best mesh with the program being evaluated. Please note that the Building Skills/Confidence indicators selected by the educator are also automatically selected for the Taking Charge and Financial Behavior indicators. In other words, all three measures will use the same educator-selected indicators. Example: Building Skills Please circle the number that best describes how your confidence to do the following has changed:
The "Taking Charge" indicators capture the participants' intended changes in their current and future financial behaviors, and they are phrased as statements related to financial behavior. They are categorized under headings and subheadings to help educators easily choose the best taking charge indicators for the specific program. Example: Taking Charge Please circle the number that best describes how your confidence to do the following has changed:
The "Financial Behavior" indicators capture actual changes in financial behaviors using the stages-to-change approach. They are categorized under headings and subheadings to help educators easily choose the best financial behavior indicators for the program they are teaching. If the Financial Practices for Actual Behavior Change option is selected on the Post Evaluation Only or Pre and Post Evaluation, then this section will appear on the follow-up instrument. The indicators will be the same as those used in the Taking Charge and Building Skills/Confidence sections. Example: Stages to Change Evaluation For each financial practice, please circle the number that best describes your current behavior.
The "Financial Position" indicators are only used in the follow-up evaluation. They capture specific changes in dollar amounts by asking participants to assign a monetary figure to certain financial areas. Example: Determining the impact on financial status. Please indicate how your overall financial position has changed since completing the program.
Select, Add, or Edit Knowledge Questions and Behavior Statements The indicators span a wide range of financial topics including consumer decision-making, cash-flow management, savings and investments, credit and debt management, homeownership, and retirement and estate planning. The topics are organized into the following abbreviated main categories and related subcategories:
Educators can further customize the evaluation tool by editing any of the outcome indicators or by adding customized knowledge testing questions or behavior statements.
Select, Add, or Edit Open-Ended Questions for Qualitative Information
Select, Add, or Edit Demographic Questions Evaluation Instrument Blueprint Once educators have a general understanding of the database (e.g., evaluation options, outcomes and indicators, and financial topics), they can put this knowledge to use creating an evaluation tool that best matches the needs of their financial education program. This section provides a step-by-step illustration using the database to design a customized evaluation tool. For this example, the Pre and Post Evaluation option has been selected to show educators the steps involved with creating an evaluation tool. However, all evaluation options require similar steps and procedures. The notable exception to this is for the Stages to Change Evaluation. This option allows educators to collect information from program participants in a series of stages, rather than at the beginning or beginning and end of the program. Educators can design evaluation instruments for an Initial Observation, Mid-Term Observation, and an End-of-Program Observation. In addition, educators also can create a Progress Report to track participants' economic progress at various stages of the program. The initial observation and the end of program observation have been set as the default option for Stages to Change Evaluation. Appendix C: Progress Report After the educator completes each step in the process, he or she clicks the word "Next" on the screen to progress to the subsequent step. To return to prior screens, click the word "Previous."
Choose the Evaluation Option
Evaluation Tool Selection Page
After educators choose an evaluation type, they will be led through the subsequent steps to create the evaluation tool. The number of steps educators will complete before the tool is finished varies depending on the evaluation option chosen. Table 3 summarizes the number of steps involved in designing evaluation tools with different options. Table 3. Steps for designing evaluation tools with different options
Step 1: Program Information and Follow-up Selection The Follow-Up Evaluation collects information on participants' changes in actual financial behaviors, achievement of financial goals, and overall financial position. Educators should be aware that if the follow-up option is selected, an optional statement is automatically added to the bottom of the post evaluation to collect the name, address, and phone number of the program participant. Educators will also need to input their name, title, telephone, email, fax, and due date of the follow-up survey. Along with this information, the return mailing address for the follow-up survey must also be included. The Instructor Information Sheet is a standardized form that collects basic program data from trainers and instructors on such information as date and location of program, number and profile of participants, curricula used, topics covered, and delivery methods. (For specific details, see the sample evaluations in the Appendix.) Educators are encouraged to use all available evaluation forms included with each main evaluation type. For this reason, the database default setting selects all options. If educators choose not to include certain forms, they can deselect the appropriate box to exclude forms from the final evaluation tool. In this illustration, it is assumed that the educator selects the Follow-Up Evaluation and the Instructor Information Sheet for the Pre and Post Evaluation option. The Web page of the database will be displayed as follows. Example for Step 1: With Follow-up Evaluation and Instructor Information Sheet
Step 2: Select Knowledge Questions Suggestions for Minimum Knowledge Question Selection To adequately measure changes in knowledge, it is recommended that educators select the following minimum questions for each evaluation tool option:
Example for Step 2: Web Page Before Selecting Questions
Customizing Knowledge Questions Educators can edit existing questions or add their own true/false questions. To edit existing questions, educators select the questions they want from the database and then click at the bottom of the page to "Add Custom/Edit Questions." Educators are taken to a screen where they can edit and add questions. Example for Step 2: Web Page After Selecting Add/Edit Question Option
Changing the Order of Questions As the knowledge questions are selected, they will appear in a column on the left-hand side of the screen. The order in which the questions appear is how they will be listed in the final evaluation. To change the order, the educator clicks the up-and-down arrows in front of each question, or omits the question by using the "Remove" button which appears as a minus sign. Example for Step 2: Web Page After Selecting Questions
At the end of Step 2, the educator will have a set of customized questions for testing the financial knowledge of the program's participants. As educators choose questions to create their evaluation tool, they can preview the tool by clicking on the words "preview evaluation" located at the top of the left-hand column under the header "Added Statements." This link appears when the educator begins adding knowledge questions to their evaluation.
Step 3: Choose Confidence and Behavior Indicators For the Post Only Evaluation and the Pre and Post Evaluation, educators are asked if they want to add the following impact indicators into the evaluation tool:
It is recommended that educators include all of these sections. Note, however, that the option to add the Financial Practices for Actual Behavior Change will only appear if the Follow-Up Evaluation option was selected originally in Step 1. If the educator decided not to conduct a follow-up evaluation, this section will not appear. Once educators make the choice of what they would like to measure, they proceed to the next step. Example for Step 3: Web Page with Confidence and Behavior Options
The database will prompt educators to select five financial action statements for the Building Skills for Confidence Levels and Taking Charge for Potential Behavior Change sections of the evaluation instrument. Educators should note that statements in the Taking Charge for Potential Behavior Change section in the post evaluation tool and the Financial Practices for Actual Behavior Change section in the follow-up evaluation tool are similar or identical in format. Therefore, when educators select a statement for the Taking Charge for Potential Behavior Change section, the respective statement for the Financial Practices for Actual Behavior Change section will automatically be added to the follow-up evaluation tool. Note that the options for the Stages to Change Evaluation and the Train-the-Trainer Evaluation are slightly different than for the Post Only Evaluation and the Pre and Post Evaluation. For the Stages to Change Evaluation, educators are only asked to add Financial Practice Indicators to track changes in actual financial behaviors. For the Train-the-Trainer Evaluation, educators are asked if they want to add the following sections:
The Building Teaching Skills and Taking Action for Teaching sections allow educators to select indicators that capture confidence levels and behaviors related to an instructor's ability to teach and use the program materials. The Shaping Personal Skills and Taking Action for Personal Financial Success sections allow educators to select indicators that capture an instructor's confidence and ability to manage their own finances.
Step 4: Select Statements for Behavior and Skill Change It is recommended that educators select five action statements for the Post Evaluation Only option and five to 10 action statements for all other evaluations. Action statements should be selected based on the program's objectives. Example for Step 4: Web Page with the Topics of Action Statements
As the educator selects action statements, these will be listed automatically in the left-hand column under the headings Financial Practice Indicator(s) and Taking Charge Indicator(s), as shown in the following example of Step 4. Example for Step 4: After Selecting Financial Action Statements for Confidence and Behavior Change
As with the knowledge statements, educators can edit and add statements by clicking on the "Add Custom/Edit Statements" link at the bottom of the Web page. If educators want to add their own statements, they should carefully format their statements to match the structure and wording of the other statements they have selected. Example for Step 4: Adding Custom Statements
Again, educators can change the order of the financial action statements by using the up-and-down arrows in front of each question. Statements can be deleted using the "Remove" option button that appears as a minus sign. After financial action statements are selected for the behavior and skill change indicators, the educator is ready to move to the next step.
Step 5: Select Program Questions Example for Step 5: Select Program Questions
The qualitative questions included in the database vary with the type of evaluation and are listed below. Post Evaluation Only and Pre and Post Evaluation
Step 6: Select Program Participant Information The categories include:
Note that demographic questions for lower-literacy target audiences are listed first for each category. The demographic questions in the Instructors/Educators category should be used only for the "Train-the-Trainer Evaluation" option. Example for Step 6: Selecting Program Participant Information Questions
In addition to selecting from demographic questions in the database, educators can add or edit demographic questions by clicking on "Add Custom/Edit Statements" at the bottom of the Web page. If educators choose this option, they will see the following page displayed. Example for Step 6: Select Program Participant Information
As with the knowledge and behavior indicators pages, demographic questions appear in the left-hand column as they are selected. The order of the questions can be changed using the up-and-down arrows, and questions can be deleted using the "Remove" option button, shaped like a minus sign.
Step 7: Follow-Up: Financial Progress Indicators These economic indicators are helpful to document the changes in financial position for program participants. If educators need additional economic impact indicators for the Follow-Up Evaluation, they can enter them into the text box for "Add Custom Indicator" and click the "Add Question" Button. The new indicators will automatically be added to the table. Educators should note that if they chose the "Follow-Up Evaluation" option, the action statements that were selected to capture changes in confidence levels, skills, and behaviors are carried over to the follow-up evaluation tool. Example for Step 7: Follow-Up: Financial Progress Indicators
Step 8: Follow-Up: Personal Achievements Example for Step 8: Follow-Up: Personal Achievements
It is recommended that educators include both of these questions on their Follow-Up Evaluation tool to document the program impact on the participants and others. Educators also may add their own questions by entering them into the "Add Custom Question" text box at the bottom of the page and clicking "Add Question." Please note that if the question has more than one line, the <br> tag must be inserted to put lines of text on separate lines rather than hitting "Enter" on the keyboard. Please see the example below.
Step 9: Follow-Up: Additional Participant Information Questions
Option 1: "Yes, using the same demographics questions I selected earlier"
Option 2: "Yes, selecting different demographics questions"
Option 3: "No" Example for Step 9: Follow-Up: Additional Participant Questions
The follow-up evaluation tools for the Stages to Change Evaluation and the Train-the-Trainer Evaluation are somewhat different from the follow-up evaluation tool used in the Post Evaluation Only and Pre and Post Evaluation options. It is important for the educator to note that the Stages to Change Evaluation includes an End-of-Program Observation as well as the option for a final follow-up. The format for the End-of-Program Observation is consistent with the format for the Initial and Mid-Term Observations. Specifically, the Financial Practice Indicators that were selected for the prior observations are carried over so that the educator does not have to reselect the indicators. The follow-up evaluation is also similar in format to the follow-up tools for the other evaluation instruments. The main difference is that this follow-up tool includes the Financial Practice Indicators using the Stages to Change Evaluation format. The Stages to Change Evaluation also has the option to create a Progress Reporting Sheet. Finally, educators should note that the follow-up for the Train-the-Trainer Evaluation is different in format from all of the other evaluation instruments. For the Train-the-Trainer Evaluation, educators select from a series of questions about how the materials and resources from the training program have been utilized since the training (i.e., components of curriculum/curricula being used, target audiences reached, challenges in administering the program, delivery methods, and future plans for programming). At a minimum, the follow-up for the Train-the-Trainer Evaluation includes two questions that capture how the program has impacted target audience(s) and whether the program and training were effective in general. From here, educators can add other questions. (See SAMPLE D in the Appendix for an example of a train-the-trainer follow-up evaluation.)
Step 10: Create Evaluation Example for Step 10: Create Evaluation
The evaluation tool is created in a separate browser window by clicking the "Create Evaluation" button at the top of the page. Educators are also provided with final instructions on how to create the evaluation, including how to print, save, and edit the final evaluation. After educators have created their evaluation, they may see a Web site URL listed at the bottom of the survey pages. To remove the URL from the printed evaluation, educators go to File > Page Setup in the Internet browser navigation bar at the very top of the page. There, they can remove text from the footer. If educators want the URL visible in other printed Web pages, they can save the footer text so that it can be restored. A unique feature of the evaluation database is that educators can copy the URL address in the internet browser's address line and paste the address into emails. When the recipient goes to the web address, the evaluation tool created by the initial educator will be displayed. Once the evaluation has been created, to print the evaluation, educators click on File > Print at the top of their Internet browser navigation bar.
Edit and Customize the Instrument
Example: Evaluation Tool Created by the Database and Printed After Editing
UGA Cooperative Extension 
Post-Evaluation Evaluation ID Number: __________ Date: __________ Please rate the instructor(s), materials, and the overall program by circling the appropriate number.
Testing Knowledge Please circle your answer to each of the following statements.
Building Skills Please circle the number that best describes how your confidence to do the following has changed:
Taking Charge Please circle the number that best describes your answer.
What did you like the most about this program? What did you like the least about this program? How could this program be improved? Would you recommend this program to others? ____Yes ____No What is your age?______ What is the highest level of education you have completed? ____ Some high school ____ High school graduate (or GED) ____ Some college ____ Associate's degree ____ Bachelor's degree ____ Post graduate degree What is your gender? ____ Male ____ Female Did you receive an earned income tax credit (EITC) last year? ____ Yes ____ No Comments or suggestions about the program: (OPTIONAL) Share your name/address/phone number, if you are willing to allow us to contact you for follow-up comments. Name:_________________________________________Phone Number:_____________________________ Address:_________________________________________________________________________________
Required Notice
You must include in all copies of the Toolkit, portions of the Toolkit, or derivative works based on the toolkit the following notice and disclaimer: The National Endowment for Financial Education (NEFE) owns all right, title, interest to the NEFE Financial Education Evaluation Toolkit. This work is based on the Toolkit. The use of this work is subject to the following terms. This work may be used only as expressly permitted by the following terms and may not be used in any way expressly prohibited by the following terms:
NEFE neither endorses nor is responsible for the accuracy or reliability of the Toolkit or this work, and shall have no liability for investments or other decisions based on the Toolkit or this work. This work is provided "AS IS" without any representations or warranties of any kind from NEFE. NEFE disclaims all warranties, whether express, implied, or statutory, regarding the Toolkit or this work. DisclaimerNEFE neither endorses nor is responsible for the accuracy or reliability of the toolkit or any work derived from the toolkit, and NEFE shall have no liability for investments or other decisions based on the toolkit or any work derived from the toolkit. THE TOOLKIT IS PROVIDED "AS IS" WITHOUT ANY REPRESENTATIONS OR WARRANTIES OF ANY KIND AND NEFE DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS, IMPLIED, OR STATUTORY, REGARDING THE TOOLKIT, INCLUDING ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT. In particular, NEFE shall not be liable for any: (1) errors, inaccuracies, omissions, or other defects in, or lack of timeliness or authenticity of, the toolkit or any work derived from the toolkit, or for any claims or losses arising there from or occasioned thereby; (2) third-party claims, losses or liabilities of any nature in connection with the toolkit or any work derived from the toolkit, including, but not limited to any direct, indirect, special, consequential, punitive or other damages, or any lost profits or revenue. It is your responsibility to evaluate the accuracy, completeness or usefulness of any information, opinion, advice or other content available through the toolkit or any work derived from the toolkit. Please seek the advice of professionals, as appropriate, regarding the evaluation of any such specific information, opinion, advice or other content. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
