Project Evaluation: Steps, the CIPP Model, Logic Models, OECD/DAC Criteria and a Worked Example

Many organizations report on a project with activity counts: training rounds held, people reached, budget spent. Those numbers show that a project ran, not that it worked. This guide explains how to evaluate whether it worked: the steps, the frameworks evaluators use, how to choose indicators and how to judge value for money.

Worked example (hypothetical): a one-year, 4-million-baht project in one Thai province that trains unemployed adults as air-conditioning technicians. The target is 200 trainees, with at least 60 percent of graduates employed or doing paid technician work within six months. All figures are invented for illustration.

What is project evaluation?

Project evaluation is the systematic collection and analysis of evidence to judge the value of a project: whether it met its objectives, what it changed and for whom, and whether it was worth the resources used. UNDP's handbook (2009) defines evaluation as a rigorous and independent assessment of completed or ongoing activities, to find out how far they are achieving their objectives and to support decisions. The same methods apply to programmes, plans and policies, which the OECD calls interventions.

The OECD (2019) gives evaluation two jobs: accountability to funders and the public, and learning for the people running the work. A useful evaluation can tell a funder what the money bought and tell the team what to change.

What should a project evaluation cover?

• —Need and design: does the project address a real need, and does its design fit the problem?
• —Implementation: were activities delivered on plan, on time and to the intended people?
• —Results: were the objectives met, and how much of the change is due to the project?
• —Use of resources: were budget, time and people well used compared with alternatives?
• —Impact and sustainability: did wider change follow, and will benefits last after funding ends?

Monitoring vs evaluation

In Thai public-sector work the two are paired as M&E (การติดตามและประเมินผล), but they do different jobs. UNDP (2009) defines monitoring as the ongoing process of getting regular feedback on progress towards goals. Gertler and colleagues (2016), writing for the World Bank, note that monitoring usually tracks inputs, activities and outputs, while evaluations are periodic and answer specific questions about design, implementation and results.

Formative vs summative, and when to evaluate

A formative evaluation happens during implementation, to improve performance. A summative evaluation happens at the end of a project or phase, to establish what was achieved and what it was worth. UNDP (2009) describes four timings; Thai terms of reference call the middle two mid-plan (ระยะครึ่งแผน) and end-of-plan (ระยะสิ้นสุดแผน) evaluations.

Logic models and the results chain

A logic model is a picture of the links between a programme's resources, its activities and the changes it hopes to achieve, as the W.K. Kellogg Foundation's Logic Model Development Guide (2004) defines it. A results chain is the same sequence read as cause and effect: the World Bank describes inputs, activities and outputs expected to improve outcomes. The terms are used almost interchangeably.

What are the components of a logic model?

Kellogg sets out five components. The first two are your planned work; the last three your intended results.

• —Inputs: human, financial, organizational and community resources.
• —Activities: what the programme does with them, such as training or services.
• —Outputs: direct products of activities, usually counted, such as classes held.
• —Outcomes: changes in participants' knowledge, skills, behaviour or status.
• —Impact: fundamental change, intended or unintended, in organizations, communities or systems.

Kellogg expects short-term outcomes within one to three years, longer-term outcomes within four to six and impact within about seven to ten. Projects funded year by year can usually measure only short-term outcomes, and should say that impact is expected rather than measured.

A worked logic model: the technician training project

Read it left to right as if...then statements. If there are funds, trainers and placements, then the course can run. If trainees complete it, then there are certified graduates. If they are matched to employers, then they find work. If the work lasts, then household income becomes steadier.

Every arrow hides an assumption, such as local employers really needing more technicians. If one fails, a project can deliver every output and no outcome, so a good evaluation tests the assumptions as well as the results.

Theory of change vs logic model

A theory of change explains how and why a project is expected to produce its results. Gertler and colleagues (2016) describe it as the causal logic of a programme: the sequence of events leading to outcomes, and the conditions and assumptions that must hold.

Many evaluations use both: a theory of change to think through how change happens, condensed into a logic model for indicators and monitoring. Some funders ask for a logical framework (logframe), the same chain set out as a matrix with indicators, means of verification and assumptions.

The CIPP model: Stufflebeam's four components

The CIPP model was developed by Daniel L. Stufflebeam. His 2007 checklist, published through the Evaluation Center at Western Michigan University, traces it from papers in 1966 and 1967 to a 1972 paper showing it serves summative as well as formative evaluation.

CIPP stands for Context, Input, Process and Product, which ask: what needs to be done, how should it be done, is it being done, and did it succeed? Stufflebeam sums up its main theme as evaluation that aims "not to prove, but to improve."

Product in CIPP covers all results, including outcomes and impact, not only outputs in the logic-model sense; align the terms in the evaluation plan if you use both frameworks.

How many stages does the CIPP model have?

The core has four. The 2007 checklist splits Product into impact (were the right beneficiaries reached?), effectiveness (quality and significance of outcomes), sustainability (do contributions continue?) and transportability (can it work elsewhere?), giving seven components; with the contract, metaevaluation and final report, ten in all. Components can be used selectively and in any order. Note that impact here means reach, narrower than the OECD/DAC meaning.

The six OECD/DAC evaluation criteria (2019 revision)

The OECD's Development Assistance Committee (DAC) set out five criteria in 1991 and revised them in Better Criteria for Better Evaluation, adopted on 10 December 2019, which added a sixth: coherence. Each criterion is a lens on the intervention.

The OECD's principles for use: apply the criteria thoughtfully, letting the evaluation questions shape how each is read, and not mechanistically. Not every evaluation needs all six.

Effectiveness covers the more closely attributable results; impact covers the higher-level effects the project contributes to. Graduates finding work is effectiveness; steadier household income is impact.

CIPP or OECD/DAC?

CIPP follows a project's life from context to results, so it suits evaluations that start with the project. OECD/DAC divides by lenses of value, so it suits mid-term and final evaluations. Roughly, Context matches relevance, Input and Process match design and efficiency, and Product spans effectiveness, impact and sustainability. Use whatever the terms of reference require; either way, build the indicators on the results chain.

Efficiency vs effectiveness

Effectiveness asks whether a project achieved its objectives. Efficiency asks how economically and promptly it turned resources into results, compared with feasible alternatives. In Thai the pair is ประสิทธิผล and ประสิทธิภาพ, and they are often mixed up. Compare the example (Project A) with a second hypothetical design (Project B) aiming at the same 60 percent target.

Per graduate, Project B looks more than twice as efficient. Per person who found work, it costs more, and it missed the target. Measure efficiency against outcomes, not just outputs, as the OECD's definition allows. And a project that meets its target can still be inefficient if another approach would have done it for less, so efficiency always needs a comparison.

Choosing indicators

SMART indicators

UNDP (2009) defines SMART as Specific, Measurable, Achievable, Relevant and Time-bound; Kellogg uses Action-oriented, Realistic and Timed for the last three. The test is that two people measuring the same indicator get the same number. "Trainees have better livelihoods" fails it; "share of graduates in a regular job or at least 10 days a month of paid technician work within six months; target 60 percent" passes.

Baselines, targets and data sources

UNDP describes the baseline as the foundation for measuring change, ideally gathered while the programme is designed; without it, change is very hard to measure. Only then can a realistic target be set. An indicator is only measurable if the data can be obtained at a reasonable cost. For each indicator, record:

• —Definition: does employed include daily contract work?
• —Baseline and target: the value before the project, the value sought and by when.
• —Source and method: records, survey, interviews or another agency's data.
• —Frequency and owner: how often, and who collects, checks and uses it.

When a result cannot be measured directly, UNDP suggests a proxy indicator, with its limits stated.

Before and after, comparison groups and mixed methods

The hard question is how much of a change the project caused. That depends on the counterfactual, what would have happened without the project, which must be estimated from a comparison group. Gertler and colleagues (2016) warn that a before-and-after comparison of one group is a risky estimate, because it assumes nothing would have changed anyway.

In the example, graduates had no job before the course and 60 percent were employed six months later. Before-and-after credits the project with 60 points. But eligible applicants waiting for the next intake reached 25 percent over the same period, so the project's effect is about 35 points: roughly 63 of 180 graduates, not 108.

Usable comparison groups include waiting-list applicants, a neighbouring area, or similar people in agency records. With before-and-after data for both groups, difference-in-differences compares the two changes. When applicants outnumber places, a lottery is both fair and a credible comparison.

Numbers show how much changed, not why. Mixed methods combine both: survey every graduate, then interview dropouts and employers. See focus groups vs surveys and how many survey responses are enough. Where outcomes include satisfaction or awareness, see the satisfaction evaluation guide, the awareness evaluation guide and our service-user satisfaction surveys.

Value for money: cost per outcome, CBA and SROI

The World Bank defines cost-effectiveness analysis as comparing the cost of two or more options in reaching a common outcome, and cost-benefit analysis (CBA) as comparing total benefits with total costs, all in money.

In the example, cost per person employed is 4,000,000 divided by 108, or 37,037 baht; counting only the 63 people employed because of the project, it is 63,492 baht. Use the second figure to compare alternatives.

For a simple CBA, suppose graduates' average monthly income rose from 5,000 to 9,000 baht and the comparison group's from 5,000 to 6,000. The gain due to the project is about 3,000 baht a month. Over twelve months that is 180 times 3,000 times 12, or 6,480,000 baht: about 1.62 times the cost, before asking how long the gain lasts and discounting.

SROI also counts outcomes stakeholders consider material, such as less stress over debt or employers able to fill vacancies, and deducts deadweight, attribution and drop-off. See what SROI is and how to calculate it, try the step-by-step SROI calculator, or read about SROI evaluation.

Evaluation types commissioned by Thai government agencies

Thai public-sector evaluation has a legal basis. The Royal Decree on Criteria and Procedures for Good Governance, B.E. 2546 (2003), section 22, has the National Economic and Social Development Council and the Bureau of the Budget arrange value-for-money assessments of state missions, and counts social benefits and harms that cannot be priced. Section 45 requires agencies to have independent assessors evaluate mission results, service quality, public satisfaction and value for money. Laws are reviewed under a separate 2019 act on legislative drafting and evaluating the results of laws.

Whatever framework the terms of reference set, draw the results chain first. For performance indicators, see KPI cascading and the strategic management guide.

How to evaluate a project in six steps

• —1. Set the questions. Who will use the findings, for which decision, by when? Write a few answerable questions. The OECD notes that an evaluability assessment can check whether the data can answer them.
• —2. Map the results chain. Draw the logic model and, for complex work, a theory of change with its assumptions.
• —3. Choose criteria and indicators. Pick a framework that fits, such as CIPP or OECD/DAC, and set SMART indicators with baselines and targets.
• —4. Design the measurement. Choose the comparison, size the sample, prepare quantitative and qualitative tools and obtain consent.
• —5. Analyse and judge. Assess effectiveness, efficiency and value for money, cross-check sources, and test whether conclusions hold when assumptions change.
• —6. Report and use. Recommend what to scale, change or stop, who owns it and how to measure again. Stufflebeam suggests sending draft reports to users about ten days before a feedback workshop.

A short evaluation plan template

The core of an evaluation plan is a matrix linking questions to criteria, indicators and data. Here it is for the example; replace the cells to use it as a template.

Around the matrix, the plan states the purpose and users, the logic model, the design and sample, the schedule, team and budget, and how findings will be reported.

Common mistakes

• —Reporting outputs as results. People trained is an output, not an outcome.
• —No baseline. Data collected only at the end relies on recall. Plan the evaluation with the project.
• —Before-and-after alone. Change that would have happened anyway is credited to the project, as 108 became about 63 in the example.
• —Every criterion for every question. Long reports that answer nothing; the OECD warns against mechanistic use.
• —Efficiency per output. A low cost per trainee can mean a high cost per job.
• —Satisfaction as a stand-in for outcomes. Satisfied trainees are not employed graduates.
• —Findings never used. Vague recommendations, no owner, or a report that arrives after the budget decision.

When to bring in an external evaluator

Many organizations can run their own monitoring. Evaluations that must be independent, need a comparison group or must value results often call for an outside evaluator. NXT Consulting Group is registered with Thailand's Ministry of Finance Consultant Database Center as a consultant in research and evaluation (RE). NXT engagements include a performance and economic impact evaluation for ETDA and an SROI evaluation for a unit of the Faculty of Engineering, Chulalongkorn University. See our project evaluation consulting page.

Talk to us about a project evaluation

References

• —Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact Evaluation in Practice (2nd ed.). World Bank and Inter-American Development Bank.
• —OECD DAC Network on Development Evaluation (2019). Better Criteria for Better Evaluation: Revised Evaluation Criteria Definitions and Principles for Use. OECD.
• —Stufflebeam, D. L. (2007). CIPP Evaluation Model Checklist (2nd ed.). The Evaluation Center, Western Michigan University.
• —UNDP (2009). Handbook on Planning, Monitoring and Evaluating for Development Results. United Nations Development Programme.
• —W.K. Kellogg Foundation (2004). Logic Model Development Guide. W.K. Kellogg Foundation.
• —Royal Decree on Criteria and Procedures for Good Governance, B.E. 2546 (2003), Royal Gazette vol. 120, part 100 Kor, sections 22 and 45 (Thai text).