|
|
|
Program Evaluation -- (Cont'd)
Collect evaluation data -- The evaluation framework includes a work plan that specifies the who, what, when, where, why and how of data collection. It should be followed closely unless unforeseen contingencies dictate a change. An outline for a typical work plan might be as follows:
Analyze evaluation data -- The analysis phase of an evaluation is basically a data reduction process, where copious amounts of information are considered, distilled and analyzed, then synthesized to answer the evaluation questions; and in the case of a summative study, to arrive at an overall judgment of value. Although data analysis usually occurs during the late stages of an evaluation, planning for the analysis should begin at the earliest stages. Once the evaluation questions are settled upon, the evaluator and client should agree on what information is needed to properly address each question and how it will be analyzed. The kinds of analysis appropriate for a particular evaluation will be dictated by the evaluation questions to be answered and the type of data gathered to answer them. Qualitative data usually come from documents, interviews, focus groups, or case studies. Seldom is there much to "analyze" from documents for evaluations; information from documents is usually factual and related to program goals, rationale, operations, formal structure and so on. Such information is important but is generally simply reported rather than analyzed. Analysis of data from interviews and focus groups is very similar. The first step is to transcribe the proceedings from each interview/meeting. Transcriptions provide a record of the discussion and form the basis for further analysis. The most common analytic procedure is called the "cut-and-paste technique." Nowadays done by computer, it involves carefully reading the transcript and identifying sections that are relevant to each research question. All passages relevant to an issue are placed together; then they are interwoven and interpreted to identify the main themes (recurring thoughts, opinions, feelings) expressed by respondents. For analysis of case studies, data are organized and reported by case. First, we assemble the raw data (i.e., all the data gathered on the case ¬ observational, interview, documentary). Then we condense, edit, and sort the data by topic or chronologically. Finally, we write up the case study as a descriptive, analytic and evaluative narrative of the person or project, organized thematically and perhaps chronologically. Before quantitative data analysis can begin, data from various sources generally has to be merged into one master file. Sources of quantitative data include management information systems, baseline and follow-up surveys, and performance tests. If a good administrative data set (MIS) is available, the master file should be built on it. If there is no management information system, the master file should be built on survey data (in fact, it may include only survey data). Once the master data set has been created, it must be subjected to statistical and manual checks to verify accuracy and check for logical errors. The analysis begins at a descriptive level. Detailed profiles are drawn of participants, non-participants and the program using simple descriptive statistics such as central tendency (mean, median, mode), variability (e.g., standard deviation, range), and frequency. Process evaluations normally would go no further with quantitative analysis. But the descriptive analysis merely sets the stage for the central findings in a summative evaluation -- central findings concerning the impact of the program. Here statistical tests are required to determine whether the difference between treatment and comparison groups was large enough to indicate a real impact on the participants (perhaps due to the program), or whether the difference was so small that it cannot be distinguished from chance influences. A finding of statistical significance must be followed up with a more rigorous evaluation to rule out threats to validity if the change is to be accurately attributed to the program. If the difference between groups is significant and threats to internal validity are ruled out, then one may infer that the intervention had an effect on the outcome measures. The response to each evaluation issue interweaves quantitative and qualitative information to reach valid, reliable and interesting conclusions about the impact of the program. Finally, cost-benefit or cost-effectiveness analyses is carried out to answer the question, "Is the program worth what it costs?" Report the findings -- All reports should be preceded by a meeting with the client to reach agreement as to content and format. We always submit a detailed outline of the proposed report about a week before the meeting. This serves as a basis of discussion. All major findings are -- positive and negative -- are substantiated and reported objectively. The report is written clearly and concisely. An oral presentation of the findings often follows the written report. |
|
André Paradis Tel: (905) 335-0307 Fax: (905) 335-0080 andre@measureresults.com Webmaster |