Consultant: Final Evaluation
Final Evaluation: Yemen Joint Response 3
Under the Dutch Relief Alliance the Yemen Joint Response continued into a third phase in 2017 (Yemen Joint Response 3, or YJR3), running from 1st March – 31st December 2017 (10 months.
The consortium of six organisations (CARE (lead), Oxfam, Save the Children, War Child, ZOA and Stichting Vluchteling/IRC) delivered a multi-sectoral humanitarian response with health, WASH, and food security activities, incorporating gender and protection as cross cutting issues.
The geographical focus of the total YJR3 intervention was across 5 governorates (Abyan, Aden, Hajjah, Lahj and Sana’a) with members spread across different districts, apart from one district where an integrated approach was carried out by two members. Two additional districts were added as a result of the top up funding.
In addition to direct activities the JR planned to build on learnings from previous phases and seeking out opportunities to meet Grand Bargain commitments, i.e. using cash as a modality for multi-sector relief, supporting localisation, and sustainability, wherever the context allowed.
Purpose and Objectives:
The Specific purpose of this evaluation is twofold:
- Accountability: generate evidence of key achievements and challenges to inform the donor (Dutch MoFA)
- Learning: captures lessons learned and good practices focusing upon key areas of the YJR3 implementation to be shared with DRA members, policy and decision makers.
The evaluation of the YJR3 should focus primarily on evaluating the programme level performance achieved through delivery of activities, as assessed against the programme logframe. The main question to answer is whether the programme has made a positive difference for beneficiaries.
The evaluation in Yemen has two objectives:
Objective 1: Evaluate to what extent the collaborating organizations working in YJR3 have delivered effective, efficient, relevant and timely humanitarian aid to beneficiaries as set out in the log frame, according to OECD DAC criteria.
Objective 2: Evaluate and assess whether the collaboration between YJR3 members has added value to the interventions with a positive effect on beneficiaries (“collaborative impact”) and other stakeholders (local partners, local NGO sector, local Government, other international agencies and coordination mechanisms in Yemen) (“collaborative value”). What has contributed to this added value and what has not? Have there been any adverse effects?
Scope of the evaluation
Geographic scope and target groups for the YJR3 evaluation will depend on factors outlined below under methodology including programmatic changes during the course of implementation, access constraints and detailed methodology to be proposed by the lead evaluator.
A range of sectors and activities should be examined within the scope of the evaluation.
Thematic areas were also integrated within YJR3 and may be seen to have an impact on programme effectiveness. These areas should be considered within the evaluation in terms of consortium learning, so that next steps can be proposed and taken into account in the current phase of implementation.
Thematic areas are:
- Harmonised WASH and cash programming
- Conflict sensitive programming
- Staff capacity building on integrating protection, with a focus on gender
- Localisation and capacity building of local actors (NGOs and other local organisations)
Secondly, the outputs from joint activities delivered in country/thematic areas (see below) should also be examined to evaluate whether added value of the consortium has been realised in this third phase, resulting in increased impact.
Approach and methodology
The consultant will be responsible for the design and implementation of the evaluation with a team of local enumerators to be managed as needed.
The consultant should adopt an evaluation methodology coherent with the overall framework and scope of the evaluation. The evaluator will collect primary data from direct beneficiaries of the project and also use secondary data. The evaluation will use both quantitative and qualitative methods, as well as document review and, when available and appropriate, analysis of programme monitoring data to be made available by JR member organisations. The evaluation requires a participatory methodology whereby the work engages all key stakeholders including beneficiaries, community members, YJR partner staff and other stakeholders in Yemen where identified and available.
- Develop overall evaluation methodology including data collection tools and reporting templates
- Liaise with the Consortium Coordinator (CARE Yemen) and local contacts of member organisations regarding planning and setting up field visits and beneficiary visits for data gathering
- Conduct Key Informant Interviews (KIIs) with JR partners in Yemen (to prioritize areas of focus and to gather initial inputs and content
- Recruit, train, supervise and manage a team of local enumerators to carry out quantitative data collection in the assigned target locations
- Conduct KIIs, case studies and/or Focus Group Discussions (FGDs) with beneficiaries and key stakeholders in target locations in agreement with the lead evaluator; highlight and propose stories of change, where appropriate.
- Ensure that data collection meets research requirements and quality standards, including data checking and cleaning before analysis
- Responsible for data analysis and report writing (Inception Report, Draft and Final Report) according to the agreed work plan
- Attend meeting to present draft report and incorporate feedback from YJRIII partners in Yemen into the final report.
- Provide final evaluation report (see outputs, below) including timely submission of collected (raw, translated) quantitative and qualitative data
- Organise and oversee any translation of raw material that may be needed, into English, and ensure quality of translation (to be checked/approved by the Consortium Coordinator, to ensure information being used mirrors original information provided and the analysis is based on correct information)
- Keep the Consortium Coordinator updated on progress over the course of the data collection, with regular check-ins and requests for support as needed, respond to requests for information from the Consortium Coordinator regarding progress or challenges
Combined qualitative and qualitative research methods are expected to be used. To gain better understanding of the program’s achievements the evaluator can use a number of tools to collect data including:
- Primary data collection: quantitative survey of beneficiaries from a selected sample of locations across the JR implementation areas, (approach to be confirmed in inception report depending on timescales and constraints of the operating environment). Next to the data collection method, an appropriate and strategic sampling method should be selected e.g. criterion sampling, snowball sampling, critical case sampling, purposeful random sampling or mixed purposeful sampling methods.
- Desk Review: all related documents such as proposal, log frame, previous evaluations, mid- term reports, field reports minutes, and any other available documents
- Focus group discussions: at field level with groups of beneficiaries, carried out by national evaluator (if relevant the field coordinator with the support of relevant staff from the implementing partners could assist in facilitating focus group discussions)
- Key Informant Interviews (KIIs): with main stakeholders involved, such as consortium members in Yemen, MEAL and gender coordinators in Yemen, beneficiaries (and non-beneficiaries in the local community if relevant), local leaders
The evaluator is asked to propose the combination of methods to be used in carrying out the evaluation and present it in the inception report.
The inception report, findings and recommendations, and the final report will be reviewed and analysed by an international consultant.
Objective 1: Evaluate to what extent the collaborating organizations working in YJR3 have delivered effective, efficient, relevant and timely humanitarian aid to beneficiaries as set out in the log frame
Questions to include in the evaluation are listed below based on the OECD-DAC criteria:
Effectiveness (including reach)
- To what extent have the planned objectives in the log frame of YJR3 been reached?
- What were the major factors influencing the achievement of the objectives of the joint response?
- Have there been any unexpected positive or negative side-effects on beneficiaries as a result of the aid that was provided under the YJR and how was this dealt with by implementing partners?
- Have proper accountability and risk management framework(s) been in place to minimize risks on program implementation?
Efficiency (including cost-effectiveness)
- How efficient was the delivery of aid within YJR3 per organization and in partnership under the JR, not only in terms of expenditure, but also in terms of implementation of activities?
- What would have been additional opportunities within YJR3 for efficiency gains without compromising quality?
- To what extent was the programme able to adapt and provide appropriate response to changing local needs and the priorities of the people? (taking into account top up funding that was allocated after 3 months of implementation, with a focus on emergency needs)
- How relevant were the objectives and activities in addressing humanitarian needs in Yemen?
- How have priority needs, target groups and locations for YJR3 implementation been identified by partners?
- How do beneficiaries perceive the relevance of YJR3 humanitarian aid and how has the provided aid supported their lives? Are there any stories of change?
- Did the partners ensure timely implementation of programme activities in comparison to project planning?
- Could funds be re-allocated in time during implementation to respond to new developments, e.g. considering specifically the top-up throughout YJR3?
Objective 2: Evaluate and assess whether the collaboration between YJR3 members has added value to the interventions with a positive effect on beneficiaries (“collaborative impact”) and other stakeholders (“collaborative value”)
We would also like to establish if there has been any significant progress in YJR3 towards collaborative impact and collaborative value.
Questions to be included:
- To what extent has collaboration between YJR3 members contributed to effective, efficient, relevant and timely delivery of aid and enhanced impact for the beneficiaries?
- How has the collaboration between YJR3 partners contributed to appropriate response of specific needs and priorities of the most vulnerable groups, including women, children and disabled persons? Has the gender aspect been sufficiently taken into account?
- Are there additional benefits to working in consortium identified that have not previously been mentioned (to beneficiaries, members or other stakeholders)?
- How did the consortium support local partners and the localisation agenda? Could it have performed better in this area?
Deliverables & Timeframe:
The lead evaluator should submit the following deliverables within the agreed timeframe (see below) and budget (to be specified)
An inception report/work plan and evaluation matrix, which will serve as an agreement on shared understanding between CARE NL & YE and the lead consultant on how the evaluation will be conducted. The inception report will address the following:
- Understanding of the issues and questions raised in the ToR
- Data sources; how to assess the questions in the ToR (Evaluation Matrix);
- Research methodology, including sampling procedure and suggested sample size;
- Timeline, indicating all activities involved and allocation of time per activity31
- Budget for the lead evaluator
- Data collection tools i.e. survey questionnaire
A (maximum 25 page) report in English (draft and final in MS Word and PDF), excluding annexes. The report needs to be analytic in nature with relevant use of graphs, tables and a dashboard with results. The draft report should be made available to members for feedback and comments before the feedback workshop, with at least 3 working days for review.
The consultant needs to produce three hard copies of all deliverables.
The report includes the following:
- Executive Summary (1 page)
- Methodology, including sampling and limitations
- Analysis and findings of the evaluation
- 4 cases of success and learnings stories and quotes from respondents
Recommendations should look ahead to the Yemen JR4 (1st Jan – 31st Dec 2018) and consider what future opportunities may exist for:
- further collaboration to increase effectiveness
- increased accountability to beneficiaries
- improved localization of aid through the Joint Response
- Annexes/supporting data:
- Photographs of the evaluation areas (separate file) – to be advised by field contacts whether this is possible
- Bibliography of consulted secondary sources
- Finalized data collection tools
- List of key informants
- Raw data of the research in MS Excel format
- PowerPoint presentation of findings and recommendations, validation and feedback sessions with key stakeholders at YE and NL level
- Feedback meeting - to present the draft evaluation findings to all partners and gather feedback
Inception report, including methodology, work plan and draft data collection tools. To be reviewed/ cleared by international consultant.
Enumerators Training and finalization tested data collection tools
Fieldwork and draft report, including all annexes (see below)
Presentation of preliminary findings and recommendations to CARE, partners and/or other key stakeholders for verification
Final report, incorporating feedback from CARE, Partners and International consutant including all annexes (see below), and all raw data
- Table: locations of YJR3 activities
- YJR3 narrative proposal (February 2017)
- YJR3 top up narrative proposal (March 2017)
- YJR3 final consolidated budget (including top up activities)
- YJR3 final consolidated logframe (including top up activities)
The consultancy is expected to take place in the month of March 2018 with a total of 31 working days, including preparation, evaluation design, field work, report writing, presentation of findings and recommendations to CARE Yemen, partners and external stakeholders, and producing final report incorporating feedback and answering relevant questions.
The evaluator should be an experienced and independent consultant with the following expertise:
- Advanced university degree in International Development, Social Sciences or any other related field with a minimum of 5 years of professional in international development and project evaluation.
- Demonstrated experience in assessments and/or evaluations of interventions.
- Previous professional experience in Yemen;
- Excellent understanding of humanitarian and gender issues.
- Have an understanding of operating conditions in an insecure environment
- Advanced analytical and report writing skills.
- Be willing to travel extensively in the working areas of the project.
- Fluent in English and Arabic
Management of the consultancy and logistical support
The principal contacts for this consultancy will be CARE’s YJR3 Consortium Coordinator. CARE Yemen will provide all reasonably available secondary information as the Consultant may require to perform his/her obligations under this Agreement. Logistical arrangements for fieldwork will be provided by CARE Yemen as agreed with the consultant. All data gathered in the field is property of CARE. The consultant shall ensure at all times the confidentiality of data, respect the privacy of all individuals concerned.
The Consultant will operate within CARE’s established security systems.
Terms of payment:
The following are the terms of payment:
- 20% upon receipt of inception report (incorporating methodology and tools) and approved by CARE and an invoice for payment.
- 30% upon receipt of the draft report accepted by CARE and an invoice for payment.
- 50% upon the final presentation and final report accepted/approved by CARE and an invoice for payment.
The deadline for submission of applications is February 21st, 2018. All applications should include the following:
- Cover letter (maximum 1 page) stating the candidate’s availability during consultancy period and updated CV’s of all study team members, including three references with contact details
- Technical proposal: Which should include (i) brief explanation about the Consultant with particular emphasis on previous experience in this kind of work; (ii) profile of the Consultant to be involved in undertaking the evaluation, (iii) Understanding of the TOR and the task to be accomplished, (iv) draft plan and methodological approach
- Financial Proposal: Which should include all the costs (including consultancy fees, accommodation and living costs; transport cost; stationeries, and supplies needed for data collection; enumerators’ fees; and costs related to other person/research assistants that will take part).
Interested consultants or firms should submit their applications to: firstname.lastname@example.org and email@example.com Please indicate clearly “Final Evaluation: Yemen Joint Response 3” as the subject heading.
Applications will be evaluated based on the following criteria:
- Technical experience and expertise
- Quality of proposal
- Cost-effectiveness of proposal