Strong project design is the foundation of successful M&E. Developing a programme theory, specifically a theory of change and results framework, can help reintegration programme managers best understand its objectives, intended outcomes, logical thinking and assumptions. This facilitates the monitoring and evaluating of the interventions. The programme theory should be developed as early as possible in the programme design phase so it can guide programme development and implementation.
The programme development stage lays the foundation for M&E by:
Clearly articulating the desired results an intervention aims to achieve;
Outlining how it aims to achieve them;
Stipulating how progress towards these results will be measured.
When planning a new reintegration intervention, it is important to think through and explain how the intervention is expected to contribute to a chain of results. This is called a programme theory and is an important tool for designing an intervention. The programme theory represents all the building blocks that are required to bring about a higher-level change or result.
Programme theory can provide a conceptual framework for monitoring as well as evaluation. There are various different types of programme theory, including the logic model, intervention logic, the causal model, results chain and theory of change. This Handbook will describe two complementary approaches that can help to articulate how a reintegration intervention is expected to achieve results. The two approaches are the “theory of change” and the “logical framework”.
This chapter presents an overview of, and considerations to make, for effective international cooperation.
5.2.1 Theory of change
5.2.2 Results Framework
5.2.3 Types of monitoring
5.2.4 Results’ Monitoring Framework
5.2.1 Theory of change
The theory of change is a type of logical thinking exercise that occurs primarily during the development of an intervention but is also helpful during its implementation.
A theory of change describes and explains how and why a result or desired change is expected to happen in a particular context. It focuses on mapping out what a programme or change initiative does (its activities) and how these lead to results (outputs, outcomes, objectives). In this way the theory of change articulates a hypothesis about how change happens by explaining the connection between an intervention and its effect.
It does so by surfacing the logic and rationale for an intervention and articulating the assumptions inherent in the approach.46
The theory of change is particularly suited for interventions seeking social or community-based change or those related to empowerment initiatives. It can also be used to measure the complexity of transformation and change, because it acknowledges that social change is not linear but dynamic and complex. Given the fact that reintegration interventions (at individual, community and structural levels) are complex and aim to cover multiple dimensions at economic, social and psychosocial levels, a theory of change can be a useful tool for defining the rationale behind the expected process of change brought about by reintegration interventions.
It is recommended to develop the theory of change using a participatory approach that includes all actors involved in reintegration. It is a collaborative process that can encourage discussion around questions such as:
- Why do we think this change will happen?
- What evidence is there to support this?
- Is this logical?
- What assumptions are we making?
This will also help all involved clearly understand the link between M&E activities and desired results.
The theory of change helps reveal assumptions to be ‘tested’ through an intervention’s actions. Assumptions therefore play a central role in developing a theory of change. Generally, a theory of change can be articulated using the “If X, then Y, because of Z” formula. That is, “If X action occurs, then Y result will occur, because of Z assumption(s).” The process of surfacing underlying assumptions helps both identify where logical jumps are being made and identify missing key steps in the change process.
Understanding how a theory of change works helps better monitor and evaluate an intervention. A common challenge when designing an intervention are logical leaps and gaps. Often there is a disconnect between strong problem analysis and seemingly unrelated activities meant to address the problem. This is reflected in a causal pathway with weak links between objectives, outcomes, outputs and activities. Through surfacing underlying assumptions, the theory of change is a bridge between analysis and programming.
There are multiple pathways that can lead to a specific objective or the highest level of change. While there may be many other reasons for a specific change to occur, not all of these can be addressed through one single intervention. A theory of change identifies the multiple pathways to change and the most realistically achievable pathway.
A fully developed theory of change clearly spells out the sequence in which outcomes are likely to happen, and how early and intermediate outputs relate to outcomes. Sometimes outcomes are closely related, but they can also occur independently. These changes and connections are often represented visually, for example through a chart or a set of tables (see Table 5.2).
Once results are framed in a theory of change, indicators for each of these can be formulated. As explained, monitoring a theory of change focuses on assessing whether or not the assumptions hold true. Therefore, when developing indicators for monitoring, it is important to take the assumptions of the theory of change into account. (See the “Results’ Monitoring Framework” section for more on indicators and how to formulate them.)
Theory of change diagrams are generally flexible in format and may be simple or complex. They can be vertical, horizontal or circular. The chart below is just one of many ways of illustrating a theory of change. It illustrates an example of what a theory of change for an integrated approach to reintegration could include. It articulates an overall holistic vision of the intended impact of each reintegration intervention, while also spelling out conditions that should be in place for this impact to occur.
Table 5.2: Illustration of theory of change: Integrated approach to reintegration
|What needs to be done to produce outputs?
|What are components and services to be provided to returnee and community or at structural level?
|What do we want to change through reintegration?
|What are we trying to achieve with reintegration intervention?
|Assessment of the returnee’s situation upon return through reintegration.
|Returnees are provided with tailored reintegration assistance.
|Returnees have sufficient levels of economic selfsufficiency, social stability, and psychosocial well-being in their community of return.
|Returnees are able to overcome individual challenges impacting their reintegration.
|Provide tailored training sessions to enhance returnees’ skills.
|Returnees have adequate skills and knowledge to increase employability and livelihood opportunities.
|Provide referrals to services (such as health, psychosocial support, business plan development, and others as needed).
|Returnees access the services they need to facilitate their reintegration.
|Conduct assessments of the main communities to which migrants return.
|Community-based reintegration activities are designed to respond to communities’ needs and priorities.
|Communities are involved in the design and implementation of community-based reintegration.
|Communities have the capacity to provide an enabling environment for reintegration.
|Establish community-level advisory groups to support socioeconomic needs and provide linkage with key financial stakeholders.
|Returnees and their communities are able to access support to facilitate socioeconomic reintegration.
|Hold community-based dialogues and events between returnees and their communities.
|Communities are accepting of returnees.
|Sensitize local and national stakeholders on the various aspects of reintegration.
|Increased knowledge and skills among local and national stakeholders to address reintegration needs.
|Local and national stakeholders (governmental and non-governmental) have enhanced capacities for the provision of essential and reintegrationrelated services.
|Adequate policies and public services are in place to address the specific needs of returnees and communities alike.
|Establish consultative process to develop Standard Operating Procedures (SOPs).
|Developed SOPs that are in line with migration, development and other relevant policies.
|Conduct a stakeholder mapping at local and national level for reintegration programming
|Well-established referral mechanism to support returnees and their communities with their reintegration needs.
5.2.2 Results framework
A results framework or logical framework (“logframe”) clearly formulates intended results, outlines targets and specifies how to plan for success and achieve results.
A logframe helps identify an intervention’s operational design and is therefore the foundation of M&E for that intervention. It is a summary of an intervention’s intended approach to attain results and is based on the situation and problem analysis undertaken during the conceptualization stage. It summarizes the logical sequence in which an intervention aims to achieve desired results and identifies the inputs and activities required to achieve these results. It also provides indicators and sources of verification to measure progress towards achieving results.
A logframe is mostly used in the form of a matrix, which encourages linear thinking about change. It is often viewed as a management instrument for planning, monitoring and evaluation.
The table below is a sample template results’ matrix. The columns are further described in section 5.2.4.
Table 5.3: A template results’ matrix
|Verification source and data collection method
5.2.3 Types of monitoring
Different M&E approaches can be considered for assessing results at each level of intervention (individual, community, structural). The appropriate monitoring approach depends on the overall programme theory of change, main stakeholders, the indicators developed in the results framework and the programme timeline (short or long term).
While there are many more types of monitoring, for the purpose of this Handbook, the following most relevant types are mentioned:
- Programme monitoring tracks progress and performance throughout the entire reintegration programme (covering project activities, results, budget and expenditure, and risk).
- Beneficiary monitoring tracks individuals’, communities’, governments’ and other relevant stakeholders’ perceptions of an ongoing or completed intervention. Beneficiary monitoring is a way to include beneficiaries in monitoring. It assesses beneficiary satisfaction or dissatisfaction, the level of participation and inclusion, access to resources, how they were treated and their overall experience of change. This type of monitoring is recommended (and particularly useful) for generating qualitative data (narratives of reintegration) from beneficiaries or even any stakeholder. This gives realistic feedback for reintegration interventions and can be used as a tool for programme visibility.
- Reintegration governance assessment assesses at national and regional levels the reintegration ecosystem. This includes the level of engagement of various stakeholders (including migrants, diaspora groups, local authorities and relevant organizations), potential livelihoods’ possibilities and mechanisms for durable solutions. At this level, collaboration of multiple stakeholders is required to assess whether implemented reintegration interventions have made any impact. This should happen over a longer term, at least 16–18 months after the reintegration intervention begins.
As with all programming, it is important to set up clear financial monitoring procedures, as well as risk monitoring.
When designing a reintegration initiative, resources should be allocated specifically for M&E. An overall range for M&E as recommended by the evaluation community is 5–10 per cent of the total budget, with 2–4 per cent for evaluation and 3–6 per cent for monitoring. However, this is purely indicative. Similarly, M&E activities should be reflected in the initiative’s workplans to support consistent and effective monitoring practices.
Develop a thorough workplan with a clear indication of team’s role and responsibility (that is, who is responsible to deliver what), including the timeline of deliverables. It allows clarity and increases ownership among team members. The team can agree on milestones and check-in intervals to review whether they are on track. This can be done at the inception phase through a mini workshop, where roles and responsibilities of the entire team and stakeholders are presented and agree with clear timeline for deliverables.
5.2.4 Results-monitoring framework
The logical framework can be used as a basis for setting up a results-monitoring framework. This framework enables both all members of the implementing team and all stakeholders, to track progress being made towards achieving intended results.
As a monitoring tool, the results-monitoring framework can be used alongside a detailed work plan, financial reporting tools and a risk management plan to create a more holistic monitoring approach.
What follows is a sample results-monitoring framework based on the theory of change or result matrix outcomes. It outlines the questions that the framework’s components aim to respond to. This should be developed for all outputs and outcomes and for the objective(s). Further explanation on indicators, baseline and target, means of verification, the data collection method and the timeline is provided in the following sections.
Table 5.4: Results-monitoring framework
|Data source and collection method
|Baseline and target
|First positive result or observed change immediately after the intervention.
|Where and how will information be gathered to measure the indicator?
|How will the data be analysed?
|At what stage will the data be collected to measure the indicator?
|Who is responsible for organizing data collection, verification and storage?
|Baseline: What is the value of the indicator at the beginning of the intervention? Target: What is the expected value of the indicator upon completion of the intervention?
|Returnees have sufficient level of economic self-sufficiency, social stability and psychosocial well-being in their community of return.
|For example, the number of returnees who reach an overall (composite) reintegration score of 0.5 and above, disaggregated by sex, age and vulnerability.
|For example, a survey among beneficiaries who have received reintegration assistance.
|Quantitative and qualitative.
|4–6 months after provision of reintegration assistance.
|Name to be included. This could be an M&E officer.
|Dependent on country’s caseload.
|Communities benefit from the design and implementation of community-based reintegration.
|For example, the percentage of community members reporting satisfaction of communitybased reintegration activities.
|For example, community participatory monitoring (focus group discussions, community interviews). Direct observation.
|Quantitative and qualitative.
|4–6 months after start of communitybased activities.
|Name to be included. This could be an M&E officer
|For example: Baseline: could be 0 if no previous activities have taken place. Target: 50%
|Local and national stakeholders (governmental and nongovernmental) have enhanced capacities for the provision of essential and reintegrationrelated services.
|For example, the percentage of stakeholders declaring that they are more engaged in the field of reintegration assistance (disaggregated by type of support).
For example, preand post-training survey.
Semi-structured interviews with local and national stakeholders.
|Qualitative and quantitative.
|3–6 months after capacitybuilding activities and periodically during partners meetings.
|Name to be included. This could be an M&E officer
|For example: Baseline: according to initial stakeholder mapping. Target: 70%
Indicators are measurable pieces of information that help assess how work or activities lead to results. They show progress towards targets and whether a result is achieved. During monitoring, indicators are meant to measure outputs and outcomes, and for evaluation they can be used at the impact level.
When selecting and defining indicators:
- Define key concepts, such as what does ‘sustainability’ mean for reintegration interventions. Agree on common definitions of key concepts.
- If a reintegration initiative has a regional nature, harmonize indicators across countries, so data can be compared and analysed.
Data source and collection method
Based on the indicators selected, data sources identify where and how information is gathered for the purpose of measuring the specific indicators. The data collection method identifies the method(s) to be used to collect the data. Commonly used methods include:
- Document or desk review
- Surveys (mini and formal)
- Interviews (including key informant and exit interviews, see section 2.7)
- Focus group discussions
- Testing or direct measures
- Mapping (for example, community maps)
See Annex 4.A for more detail on data collection methods.
Data collection sources can include questionnaires, checklists, topic guides,47 or project administrative documents such as handover certificates, case file documents, and so on.
When creating a data collection tool, remember to:
- Include fields that record the name of the data collector and the date and location of data collection, biodata and contact information of the respondent.
- Include free and informed-consent and confidentiality clause in the personal data collection instrument if the tool is not anonymous (see section 5.1.1).
- Address data-management requirements for the specific data collection tool. This can include budgeting for resources or staff time to develop and use the tool, as well as databases or systems that may need to be set up and maintained.
Language in data collection tools should be neutral and objective. Consider the data collection skills and technology available in the country. Different tools require different skills and failure to match capacity with the tool creates data bias and error. It is recommended to pre-test the data collection tool.
When it comes to generating feedback through monitoring beneficiaries,48 sampling as a method can be specified at the planning stage of monitoring or evaluation.49 This method is particularly useful, as often it is unrealistic to meet every beneficiary or visit every project site. Instead, use of a smaller group of beneficiaries, their geographical coverage, allocated resources and security context are all key aspects to be considered. Hence sampling is useful to:
- Minimize data bias and improving data quality;
- Reduce the time and money spent on data collection.
Sampling involves a variety of techniques. The choice of technique depends on the context, type of population, information available, data collection method and type of data collected by the project. All techniques provide different answers on:
- Representation: the degree to which the sample “represents” the larger group;
- Sample selection: how the people or places are chosen;
- Sample size: how many people, services and so on to include in the sample.
- If sampling is planned, programme M&E officers with skills in this area should be recruited or trained.
How the data will be analysed will depend on the data collection method. Different tools are needed based on the type of analysis required. Some data collection methods can be analysed for both qualitative and quantitative information. For example, if the indicator is “presence of legislation that reflects international best practice”, the data source would be where the information (data) comes from (copy of the legislation), while the data collection method would be a document review (review of the legislation). Data analysis can be qualitative in nature, for example an expert undertaking an assessment of the degree to which the legislation is in line with international best practices.
The timing and frequency of data collection should be clearly defined from the outset of planning. Reintegration programme implementation often takes place in varied geographical places and with various partners, something crucial to consider when deciding the frequency of data collection, because this has budget implications. For example, if the indicator being measured is “referral to psychosocial support”, then it would make sense to monitor the number of persons being referred on a regular basis, such as monthly or quarterly.
Normally the results-monitoring framework is transferred to a clear workplan, where monitoring steps and their frequency are outlined.
There should be clear roles and responsibilities for data collection, verification and storage (see sections 5.3.2 and 5.3.3), especially when multiple stakeholders are involved. There should also be a data controller for personal data who ensures that data protection principles are being followed.
Baseline and target
A baseline provides a foundation against which to measure change over time. The baseline is the first measurement of an indicator; it assesses conditions pre-implementation and sets the conditions against which future change will be measured. A baseline study can have budget implications but can also be based on a previous evaluation or a desk review. When budget is limited, or when security constraints or other factors do not allow for a baseline study, the monitoring visit in which a specific indicator is measured for the first time can be considered the baseline.
The target is what the intervention hopes to achieve and is usually defined in relation to the baseline.
IOM’s Reintegration Sustainability Survey
IOM developed a standardized Reintegration Sustainability Survey to evaluate the sustainable reintegration of returnees in the economic, social and psychosocial dimensions. This survey helps answer the question: To what extent have migrants achieved a level of sustainable reintegration in communities to which they returned?
This survey, along with the scoring system, can be used as a case management tool, for beneficiary monitoring and for programme evaluation. It is primarily designed to be administered to returnees 12–18 months after their return. However, the survey can be completed multiple times throughout a returnee’s reintegration process. For example, depending on available resources, a first (baseline) reintegration score could be generated during the first counselling session that is used to assess needs (month 0–1) and compared to intermediary score 6–9 months after return to assess progress. A final score (month 12–18) then measures reintegration sustainability.
Intermediary monitoring scores collected during the reintegration assistance period can serve to readjust assistance based on reintegration scores for the three different dimensions.
Scoring after the conclusion of reintegration assistance is perhaps the most valuable – because it reflects the sustainability of the returnee’s situation. These scores can also feed into final programme evaluation. They can be analysed to indicate the effectiveness of different types of reintegration assistance for different categories of returnees, in different contexts. Data generated through the scoring system also provides necessary evidence of the influence of community and structural-level factors on the reintegration of individuals (for example, poor access to health care is systematically reported in a set area) and can therefore feed the development of targeted community and structural-level interventions.
Trends in reintegration scores can be easily analysed in relation to basic profile information. Reintegration scores can be compared across sex, gender and age. They can compare patterns for returnees assisted through voluntary return and those returning through other means. The recommended variables for an analysis of reintegration sustainability are listed below:
- Sex and gender
- Date of return
- Age at time of return
- Host country prior to return
- Country of origin
- Length of absence from country of origin
- Mode of return
- Community of return same as community of origin?
- Possible situations of vulnerability (determinants/triggers)
- Type of occupation
See Annex 4 for more information on the Reintegration Sustainability Survey.
46 IOM definition of theory of change adapted from the Center of Theory of Change, What is Theory of Change? (2017).
47 A topic guide is an outline of key issues and areas of questioning used to guide a qualitative interview or group discussion.
48 Beneficiaries include returnees, community members and local stakeholders.
49 A sample is a part of the population, used to describe the whole group. Sampling is the process of selecting units from a population, to describe or make inferences about that population; that is, to estimate what the population is like based on the sample results.