Harrel, A. (n.d.). Evaluation Strategies for Human Services Programs.
Washington DC: The Urban Institute.
View entire document
Developing a Logic Model
It is impossible to interpret evaluation findings without a clear understanding of program goals, implementation sequences, and the expected links between them and expected program benefits. Expectations about these linkages are made explicit by developing a logic model. Such a model is developed by discussing with service providers and funders the goals of and rationales behind program organization and content, examining planning documents and program reports, and reviewing research findings on similar programs or problems. The literature review may be particularly helpful in identifying plausible causal links and any factors other than the program which should be considered in the evaluation.
The logic model provides a simplified description of the program, the intended outputs, and the intended outcomes. Program characteristics include the population to be reached, the resources to be used, and identification of the types and levels of service elements. Outputs are immediate program products resulting from the internal operations of the program, such as the delivery of planned services. Examples of output indicators in the area of programs for vulnerable children and youth might include the numbers of children immunized, home visits by case managers, or youth completing a job training program. These program outputs are, in turn, the vehicle for producing the desired program outcomes, for example, decreases in childhood illnesses, decreases in abuse and neglect cases, or increases in youth employment. Careful attention must be paid to when the anticipated outcome should be expected to occur. For this reason it is often useful to divide outcomes into intermediate versus longer term. For example, improved school attendance in early grades might be an intermediate outcome associated with the longerterm outcome of dropout prevention. Care must be given to focusing on outcomes which will occur within the study period.
A classic failure in selecting an outcome that is expected to occur within the time frame of the study occurred in evaluations of the DARE drug prevention program, an educational program for fifth and sixth graders designed to prevent drug use. Evaluation results showed no significant prevention of drug use at the end of the program. This result should have been anticipated, since drug use does not typically begin among youth in this country until the mid-teen years (14 to 17). An age-appropriate intermediate outcome should have been selected as the primary outcome measure, such as improved peer resistance skills and changes in beliefs about the risks of drug use.
The logic model should also include explicit mapping of the conditions present in the program environment or characteristics of the target group or community that may affect the program's ability to achieve its goals. Non-program characteristics of the program organization, community or target population that are likely to influence the outputs and outcomes and/or use of program services are called antecedent variables. Conditions or events in the program, target population, or community that may limit or expand the extent to which program outputs actually produce the desired outcomes are called mediating variables. For example, a drug abuse prevention program may be less effective if the program staff are inexperienced, or if the local community offers fewer recreational alternatives to substance abuse and/or more active open drug markets (antecedent variables). Offering other support services in combination with the program may enhance its impact (a mediating variable).
In impact evaluations the logic model is used to spell out how, and for whom, certain services are expected to create specific changes/benefits. For example, if the program includes parenting classes, the logic model will identify this activity as a key program component and show the types of changes in parenting that will be used to measure program outcomes (e.g., by improving parental assistance with homework or helping parents communicate more effectively with adolescents).
In performance monitoring, the logic model is used to focus on which kinds of output and outcome indicators are appropriate for specific target populations, communities, or time periods. For example, among indicators of child improvement in school, one might expect attendance to improve in the first semester of a program, but academic test score improvement only after a significant period of program participation-with the timing possibly varying by the age and developmental stage of the children.
In process evaluation, the logic model is used to identify expectations about how the program should work-an "ideal type"-which can then be used to assess the deviations in practice, why these deviations have occurred, and how the deviations may affect program outputs. This assists program managers (and evaluators) to identify differences (including positive and negative unintended consequences), consider possible mechanisms for fine-tuning program operations to align the actual program with the planned approach, or re-visit program strategies to consider alternatives.(2)
Logic models are constructed to show temporal sequences, building left to right, and they typically diagram relationships with arrows. An example of a logic model is shown in Exhibit A. It was developed by the Urban Institute during the planning of the evaluation of the Children At Risk program (CAR). CAR is an intensive intervention program designed to prevent involvement in drugs and crime, and to foster healthy development among adolescents ages 13 to 15 who exhibit serious risk indicators and live in severely distressed inner-city neighborhoods.
The intervention consists of eight required program components:
Case Managers employed by the program make a service plan for all members of the household of participating youth and provide intensive follow-up on referrals to needed services, handling a. caseload of 15;
Family Services include parenting skills training for all parents, and referral to other services as needed (intensive family counseling, stress management/coping skills training, identification and treatment of substance abuse, health care, job training and employment programs, housing, and income support services);
Education Services include tutoring or homework assistance for all youth, and referral to other services as needed (educational testing, special education classes);
After-School and Summer Activities for all CAR youth include recreational programs and life-skill/leadership development activities, combined with training or education;
Mentoring is provided by local organizations for youth in need of a caring relationship with an adult. The role of the mentor is to: (a) inform youth about alternative available choices (e.g., activities and goals); (b) familiarize them with strategies available for pursuing those choices; (c) provide training, opportunities for practice, and feedback in the development of skills for implementing particular strategies; and (d) provide relationships through which youth are affirmed, inspired, and encouraged to make healthy choices;
Incentives such as gifts and special events are used to build morale and attachment to the pro-social goals of the program (e.g., gift certificates, trips, and vouchers for pizza, sports shops, movies, and stipends for community service during summer programs);
Community Policing/Enhanced Enforcement is used in all target neighborhoods to create safer environments with less drug activity. Law enforcement activities include out-stationing police in schools and neighborhood locations to maintain order and enhance relationships with community groups;
Criminal/juvenile Justice Intervention involves collaboration between case managers and juvenile court personnel to provide community service opportunities and enhanced supervision of youth in the justice system.
Antecedent variables include the levels and types of neighborhood, family, peer group, and personal risk factors for participants as well as their demographic characteristics. These are influences that are present before the program intervention.
Mediating variables include exposure to other social or educational services, perceptions of opportunities, and social norms. These are influences that
Logic Model Used in Evaluation of the Children At Risk Program
operate at the same time as the program is operating. The program components are designed to achieve the intermediate outcomes-reductions in risk factors and enhancement of protective factors at the end of program participation. These intermediate outcomes, measured at the end of program participation, are hypothesized to be requisite steps towards the desired longer-term outcomes-prevention of drug use, drug selling, delinquency, school failure and dropout, and teen parenthood.
Program outputs, not shown in this diagram, include indicators of performance such as the number of tutoring sessions provided, number of home visits by case managers, and number of times parents participated in program activities.