|
|
|
Program Evaluation -- (Cont'd)
Devise the evaluation design By this stage we know exactly what to evaluate (we have a good understanding of the program), we know why it is being evaluated, we know who will use the information and how it will be used, we know what issues the evaluation will address, we know what indicators to use to address each issue, and we know what methodologies to use to address each issue. Now we have to decide the best way to design the study so we end up with the most credible and useful possible evaluation results. There is no single correct evaluation design. The idea is to come up with the best design possible under the circumstances. The design must be flexible enough to handle unanticipated changes or problems, which come up frequently in any evaluation. Write the evaluation framework -- The first five steps comprise the "pre-evaluation assessment," also called evaluability assessment, the front-end analysis that helps determine how best to evaluate the program. The assessment generates the terms of reference for the evaluation (or the "evaluation framework"). A thorough framework includes a complete program description, evaluation objectives and requirements, key issues to explore, data sources, potential evaluation indicators, desired methodologies, deadlines, and budget. In short, it designs the evaluation. To conduct a good evaluation, there must, at the very least, be a proper description of the program so that the type of intervention supposedly being implemented is known in advance. A pre-evaluation assessment generates a thorough description of the program's structure ¬ its objectives, logic (cause and effect relationships), activities, and indicators of successful performance. In clarifying program structure and intent, it can determine the plausibility of the program achieving its intended goals, identify opportunities to improve program performance, and serve to ensure credible and useful evaluations. It considers such factors as the program's characteristics, available research methodology, cost, and constraints on the use of the desired research methods in determining the best evaluation design. If done properly the assessment will prevent evaluators from measuring something that does not exist (because the program hasn't been implemented as planned), and from measuring something that is of no interest to management and policy makers. Knowledge that a program has or has not been implemented as planned is essential when trying to discern reasons why the program performed as it did. Knowledge of what kinds of information decision makers need is essential to producing useful results. With this information, potential pitfalls and constraints for the evaluation can be identified and controlled, and the appropriate methodology for the evaluation can be developed. Program staff are asked to identify realistic definitions of performance by linking activities and resources to outcomes, specifying plausible outcomes, and identifying indicators of performance for key activities and outcomes. Information and conceptual gaps may be revealed through this process, which managers can address to improve program performance. It is also a necessary step in determining what is important to evaluate, as opposed to what could be evaluated. |
|
André Paradis Tel: (905) 335-0307 Fax: (905) 335-0080 andre@measureresults.com Webmaster |