Consultancy: Project Evaluation-REFERENCE: IC/PNG/010-2022-International and National

Location: Bougainville (with travel to other locations in the Autonomous Region of Bougainville), Papua New Guinea

Type of Contract: Individual Contract (IC)

Project: Sustaining Peace in Bougainville/ Post-referendum Process Support Projects

Starting Date: 1st April 2022 (tbc)[1]

Duration: 30 working days - April- May 2022

[1] Depending on update and donor’s approval of project no-cost extension, start date of the Evaluation is likely to be on 1st April 2022.

Background and Context

In 2001, the Bougainville Peace Agreement (BPA) was signed between the National Government of Papua New Guinea (GoPNG) and leaders representing the people of Bougainville. The BPA marked the end of a decade-long civil conflict in which up to 20,000 people died and many more were left without family, access to basic services and infrastructure, traumatized and scarred for life. With an estimated population of over 300,000 people, speaking 28 languages, in 33 constituencies stretching from the atolls and islands to the mountains that dominate the Centre of the mainland, Bougainville is an incredibly diverse region.  Bougainvilleans voted overwhelmingly for independence in the 2019 referendum and the two governments are now consulting on the ratification of the referendum outcomes and the next steps in Bougainville’s peace process.

The UN Peacebuilding Fund support in Papua New Guinea

The UN Peacebuilding Fund (UN PBF) Programme in Papua New Guinea started in 2014, following declaration of eligibility of PNG for PBF support in 2013. The Peacebuilding Priority Plan, covering period of 2015-2018, supported implementation of the Bougainville Peace Agreement (BPA), reconciliation and weapons disposal, as well as preparations for Bougainville Referendum.   The 11th of December 2019 marked a historical moment for PNG - Bougainville relations, and implementation of the Bougainville Peace Agreement (BPA), with the announcement of the Bougainville Referendum result of 97.7 per cent of voters choosing Independence from PNG. Following the referendum vote, the PBF support covers aspects of political dialogue in post-referendum process, engaging outlier factions in peace process and awareness-raising activities on BPA and post-referendum processes.

The Sustaining Peace in Bougainville Project

The UN PBF funded Sustaining Peace in Bougainville project commenced in June 2018, and focused on supporting the peaceful and inclusive completion of the 2019 Bougainville Referendum. The is a joint project implemented by UNDP, UNFPA and UN Women.

Following the successful completion of the referendum, and beginning of post referendum process in an environment where these complex risks will likely contribute to key challenges to sustaining peace in Bougainville, the Project has received cost-extension until January 2022.  The second phase is designed to assist the two governments and the people of Bougainville and Papua New Guinea to continue to sustain peace and address potential tensions points during the post-referendum period. The cost extension closely complements and collaborate with the UNDP’s ‘Post Referendum Support Project’.

Where the Post Referendum Support Project targets its support to the joint consultations and the ratification process through the operationalisation of the joint secretariat and technical support to the working groups, the PBF-funded project complements by ensuring marginal groups are empowered to participate in the consultations and in the decisions, and that those remaining groups outside of the peace agreement are supported to join, in an environment where Human Rights are upheld. Given the complementarities of the two projects, UNDP has a joint Annual Workplan for two projects, specifying the activities funded through each project.

The Project has the following outcomes:

Outcome 1:  Enhanced political dialogue between the two Governments and the two Parliaments, ensuring decisions around BPA implementation, the referendum and post referendum are progressed jointly

The Project extended support to Post Referendum Joint Ministerial Consultations Preparation Team, as defined by the JSB meeting of March 2020, that is in charge of the Post-Referendum Inter-governmental. Logistical and administrative support was provided to ensure timely conduct of political dialogue consultation meetings and implementation of joint agreement on post- referendum processes, and resolutions by the two governments. This outcome is implemented in close partnership with the range of institutions, departments and other key actors working to advance the peace process in Bougainville. This include, among others, the Offices of both Chief Secretaries, NCOBA, DoIMI (Department of Independence Mission Implementation) as well as both parliaments.

Outcome 2:  Increased dialogue and awareness on the BPA, referendum, and post referendum issues, ensuring that both the population in and outside of Bougainville is informed and feels included in the process

The Outcome is focused specifically on supporting granular awareness activities on the post referendum process and progress, and Peace Agreement through key groups: Women, Youth, Churches, Veterans and Communities to empower them to participate in the process. To achieve this, community facilitated dialogues were conducted based on agreed joint key messages through out Bougainville. The project also supports a national media-based awareness campaign to prepare national adverts to keep all Papua New Guineans updated on progress.

Outcome 3: Strengthened unification of outlier communities into the peace architecture and post-referendum dialogue

The Outcome is focused on designing initiatives and implementing activities that support the two remaining outlier groups to engage with the peace architecture. The Outcome aims to increase community participation across Bougainville with specific focus on outlier factions, reporting on and prevention of Human Rights abuses, and identification of economic opportunities, as well as promoting  community buy-in for peace.

Evaluation Objective

Sustaining Peace in Bougainville Project commenced in June 2018, and findings of Bougainville Perception Survey conducted in 2017 were used as a basis for baseline indicators. The Project received cost-extension in July 2020 with new project end date as January 2022. This is a joint Project, implemented by UNDP, UNFPA and UN Women.

The Post-Referendum Process Support Project commenced in July 2020 and lasts until June 2022. The Project frames the support that the UN through UNDP can provide for the immediate post-referendum processes, including provision of technical and logistical support to the Secretariat, capacity building, support on the ratification process, continued awareness to the people of Bougainville and PNG on the progress.

The evaluation will assess the achievement of the Sustaining Peace in Bougainville and Post-referendum Process Support Project results against what was expected to be achieved as outlined on the projects results framework. The lessons for this evaluation will improve sustainability of benefits from this project that can be further enhanced in other UN Programming in the Autonomous Region of Bougainville.

This evaluation is being conducted to document good practices, challenges and capacities that are at hand and that are missing that can inform UN Programming in Papua New Guinea going forward.

Evaluation Approach and Methodology

The evaluation team, comprised of Lead Consultant (International recruitment) and Local Consultant (local recruitment), is expected to follow a participatory consultative approach. This will be achieved through close engagement with the UN Head of Office in Bougainville (Political Liaison Officer), project team, programme team, UNDP, UNFPA and UN Women Country Officers’, the Autonomous Bougainville Government (ABG) and the National Coordination of Bougainville Affairs (NCOBA) including other partners, stakeholders and beneficiaries as and where relevant.

Desk Review

The evaluation team will review all sources of relevant information including, but not limited to, the following:

• Papua New Guinea Peacebuilding Priority Plan (2015-2017)
• Report on Evaluation of Peacebuilding Priority Plan in PNG in 2018
• Project documents: Sustaining Peace in Bougainville Project Document for Phase 1 (2018-2020); Sustaining Peace in Bougainville Project Document for Phase 2 (2020-2022), jointly implemented by UNDP, UNFPA and UN Women;
• Annual Project Workplans on implementation of SPB Project by UNDP, UNFPA and UN Women;
• Project document for Post-referendum Process Support Project (2020-2022), implemented by UNDP and Annual Workplans;
• M&E Plan and Results Framework
• Perception Survey Report 2017
• Conflict Analysis for Bougainville and the Lessons Learned Report of March 2020
• Mid-year and Annual Project Progress reports (2018 - 2021)
• Risks Logs

Evaluation design

The evaluation will utilise a non-experimental design, which examines the efficacy of the project and makes forward looking recommendations for UN implementation in ARoB. Data collection will include a review of project documents, key informant interviews and, where possible and deemed relevant, focus group discussions (FGDs) with partners, stakeholders as well as both project direct and indirect beneficiaries .

Clear ethical guidelines should be followed and implemented, obtaining consent from any individuals interviewed or included in FGDs. The evaluation should be gender and youth responsive, as outlined in the evaluation objectives, using a participatory methodology where possible/relevant. Any limitations in the design should be clearly stated and data should be triangulated to enhance the robustness of the data.

Evaluation Scope

The evaluation will be conducted at the end of the project and will cover the entire cycle of the project. It will take into consideration activities conducted across the entire region (Autonomous Region of Bougainville), evaluating the reach of the project in the process.

The Autonomous Region of Bougainville has (3), regions, (13) districts 33 constituencies and over 450 Wards. Decision on the specific locations for the evaluation will be proposed by the project RUNOs and agreed with the PBF Secretariat based on the methodology and proposed sample size. The Project envisages representation of the three regions for this exercise, as well as a specific focus on Buka, the Autonomous Bougainville Government (ABG) and respective ABG departments.

In accordance with the evaluation objectives, and guided by the OECD DAC Evaluation Criteria in Settings of Conflict and Fragility[1] and United Nations Evaluations Group norms and principles, the following key areas will be examined as indicated below:

Relevance To what extent Project objectives and design respond to partners/key stakeholders needs, policies, and priorities?  Does the Project monitor Bougainville post referendum context to adjust the interventions  circumstances change? Assess the appropriateness and relevance of the project design to the project outcome and deliverables. This will include an examination of the theory of change, and testing the hypotheses that informed it

• To what extent was the project in line with the Government of PNG’s national development priorities, ABG regional priorities, UNDP, UNFPA and UN Women current country programme outputs and outcomes, UNDAF and the SDGs?
• To what extent did the project contribute to the theory of change for the relevant country programme outcome?
• To what extent were the lessons learnt from other projects including perspectives of men, women, and young people, who could affect the outcomes, and those who could contribute information or other resources to the attainment of stated results, taken into account during the project design process?
• To what extent did the project contribute to gender equality, the empowerment of women and a human rights-based approach?
•  To what extent were the project risks  and opportunities managed over the life cycle of the project to see its relevance to the peace building priorities two Governments leading to expected results?.

Coherence How compatible was the intervention with other interventions in the Bougainville context? For example, with other existing ABG strategies, policies or programmes.

• Assess whether the objectives remained valid over the course of the project, and whether adjustments were made.

Effectiveness: Is the intervention achieving its objectives? The extent to which the intervention delivers, or is likely to deliver, results in an economic and timely manner. Furthermore, assess the major achievements of the project in relation to its stated objectives and intended results. As far as possible, this should be a systematic assessment of progress based on monitoring data, including:

• Key achievements supported evidence where possible, i.e. opinions on the project’s effectiveness based on interviews with key partners and beneficiaries.
• Internal and External Communication including working relationships within the team, partners, stakeholders, donors and beneficiaries.
• Learning processes such as self-evaluation , coordination and exchange with other PBF and related projects that contributed to key successes and failures
• Assess other programme management factors important for delivery, such as capacity gaps (these could be in the project team, other internal functions such as HR or Finance, or external organisations as appropriate).
• To what extent did the project contribute to the UNDP, UNFPA, UN Women  country programme outcomes and outputs, the SDGs,  the United Nations Sustainable Development Cooperation Framework (UNSDCF) and national development priorities?
• To what extent was  UNDP, UNFPA and UN Women programme outputs, outcomes, partnership strategies been appropriate,  effective and achieved? If not, why?
• Were the project objectives and outputs clear, practical and feasible within the dedicated timeframe? Did they clearly address women, men, youth and other vulnerable groups?

Impact:

• What difference did the intervention make? The extent to which the intervention generated or was expected to generate significant positive or negative, intended or unintended, higher-level effects.
• Describe any unforeseen impacts or unintended consequences (whether positive or negative).
• Identify any exceptional experiences that should be highlighted e.g. case-studies, stories, best practice.

Efficiency: How well were resources utilised? Efficiency of planning and implementation. Were plans used, implemented and adapted as necessary? For example:

• Were all activities in the project work plan   fully implemented?
• Assess whether the project can be considered as delivering value for money for its present scope/scale of impact?
• To what extent where resources used economically to deliver the project? Where there any opportunities for scale up?
• To what extent was the project management structure and M&E systems as outlined in the project document efficient in generating the expected results?
• To what extent were financial and human resources used effectively? Were resources (funds, male and female staff, time, expertise, etc.) allocated strategically to achieve outcomes?

Sustainability  The extent to which the net benefits of the intervention continue, or are likely to continue. Potential for sustainability, replication and expansion? Will the benefits last?

Assess the key factors affecting sustainability of the project, such as:

• To what extent will targeted men, women, youth and other vulnerable people benefit from the project interventions in the long-term?
• To what extent will financial and economic resources be available to sustain the benefits achieved by the project? Are there any financial risks that may jeopardize the sustainability of project outputs affecting women, men, youth and other vulnerable groups?
• Are there any institutional, social or political risks that may jeopardize sustainability of project outputs and the project contributions to country programme outputs  outcomes and benefits?
• To what extent do mechanisms, procedures and policies exist to allow primary stakeholders to carry forward the results attained on gender equality, empowerment of women, human rights and human development?
• Other factors to consider as well are as follow:
• Has/will the project contribute to the lasting benefit? Which organizations could/will ensure continuity of project activities in the project area?
• Is there evidence of Government, civil society organizations/partners/communities that have copied, up scaled or replicated project activities beyond the immediate project area? Is such replication or magnification likely?
• Assess and make recommendations on UNDP, UNFPA and UN Women interventions and  key strategic options for the future of the project i.e. exit strategy, replication, scale-up, continuation include gender dimensions?
• Comment on any existing plans
• Make additional recommendations, including broader recommendations on each UN agencies presence and work in the region.

Cross – Cutting

Human Rights

• To what extent have poor, indigenous and physically challenged, women, men, boy, girls and other disadvantaged and marginalized groups benefited from the work of UNDP, UNFPA and UN Women in the country?

Gender Equality

• To what extent have gender equality and the empowerment of women been addressed in the design, implementation and monitoring of the project?
• Is the gender marker assigned to this project representative of reality?
• To what extent has the project promoted positive changes in gender equality and the empowerment of women? Did any unintended effects emerge for women, men, youth or other vulnerable groups?

Disability

• Were persons with disabilities consulted and meaningfully involved in programme planning and implementation?
• What proportion of the beneficiaries of a programme were persons with disabilities?
• What barriers did persons with disabilities face?

Stakeholder Engagement:

• To what extent were different stakeholders involved in project implementation?
• To what extent were project management and implementation participatory, and was this participation of men, women, youth and other vulnerable groups contributing towards achievement of the project objectives?
• To what extent was the project appropriately responsive to the needs of the national constituents (men, women, youth and other groups) and changing partner priorities?

Evaluation Deliverables

The individuals to be contracted will work with the  Reference Group made up of UNDP, UNFPA, UNDP and the Bougainville Programme Manager to execute the following deliverables:

1. Preparation for the Evaluation 

• Conduct a desk review of past reports, Results Framework, other documents including the baseline and the Mid-term Evaluation reports
• Develop a methodology for the collection of data
• Develop questionnaire against the key evaluation questions
• Develop a field evaluation schedule

1. Presentation of Findings

• Conduct Field Missions
• Data collection

1. Draft Report

• Analyse data gathered
• Develop a report of the methodology and findings against the key questions
• Circulation of draft report for comments

1. Final Report

• Incorporate comments and final report submitted

The whole exercise is estimated to take 6 weeks but there is always flexibility to adjust as and when necessary.

Evaluation Management

Supervision

The End-of-Project Evaluation will be executed under the direct supervision of the UNDP Focal Point in collaboration with the Reference Group with on the ground support from available RUNO (UNDP, UNFPA and UN Women ), PBF focal point. Further support will be given by the Port Moresby Technical team where necessary support though the duration of the evaluation. The AROB focal point Officer will provide further information on project activities and stakeholders, and will facilitate liaison with implementing partners and beneficiaries.

Reporting

The evaluator hired will report directly to the UNDP focal point. They will report progress on a weekly basis and will work in close liaison with the Reference Group throughout the evaluation.

 Geographical Area

The Autonomous Region of Bougainville has (3), regions, (13) districts 33 constituencies and over 450 Wards. Decision on the specific locations for the evaluation will be proposed by the Evaluation Management Team and the Consultants on the basis of the methodology and the proposed sample size. Representation of the three regions for this exercise, as well as specific focus on Buka, the National Government in Port Moresby and the ABG is expected.

Institutional Arrangement

(i)          Team Composition

A team of two independent evaluators will conduct the evaluation. One is an International who will play a team leader role with experience and exposure to evaluating PBF funded projects across Asia and the Pacific Region. The other is the team expert, who is a national. The Team Leader will be responsible for the overall assessment of the project starting with the preparation for the evaluation, presentation of the findings, development of the draft report and the final report. The Team expert will report to the team leader and support the Team Leader in the entire assignment.

The individuals must also be specialized in research, statistical work and surveys (design, tools, data collection & analysis), preferably in the field  of  peacebuilding  and  governance,  with  operating experience in volatile areas, with good knowledge of and networks in Bougainville, and with possibility to assemble a field team quickly in Bougainville (with the support of the UN which will provide a list of possible local team members). The structure should have a good understanding of the sensitivities and risk management strategies concerning surveys which deal with political and peacebuilding issues.

(ii). Education, Experience & Competencies

Education:

• Hold   a   university   degree (Masters),   preferably   in   social   sciences,  statistics, demography, political science, law, international relations, public administration or economics and a thorough knowledge of qualitative methodologies.

Experience:

• Should have at least a minimum of 7 years of experience in monitoring and evaluation and research, including survey design and conduct, data analysis and report writing.

Competencies:

• Good understanding of issues on peacebuilding and governance and experience in Bougainville are highly desirable.
• High level planning, organizational and time management skills, including flexibility, attention to detail and the ability to work under pressure to meet challenging deadlines;
• Excellent interpersonal skills, including ability to establish strong cooperative relationships with senior government officials, civil society and donors;
• Ability to quickly adapt to change, and to remain calm under pressure; and
• Proven cross-cultural communication and the ability to function effectively in an international, multicultural environment.

(iii)       Reporting

The Team Leader and the Team Expert will report directly to the evaluation Management Team to be established by UNDP, UNFPA and UN Women.

Deliverables, Price Proposal & Payment Schedule

Duration of work

The contract days are 30 working days which is 6 weeks for the Team Leader and the team expert. The work schedule proposes the duration of work within this timeframe immediately after the contract is signed. The work is estimated to commence no later than 01 April 2022.

The contract price is a fixed price based on the results, with the length of service being approximate. The budget submission should include all costs to allow the implementation of activities provided for in Sections C and F, such as professional fees for team members, travel expenses, attachments, multiplication of survey questionnaires, etc.

The payment schedule will be directly related to the deliverable submissions as illustrated in the table below:

Evaluation

Cumulative analysis

The proposals will be evaluated using the cumulative analysis method with a split 70% technical and 30% financial scoring. The proposal with the highest cumulative scoring will be awarded the contract. Applications will be evaluated technically, and points are attributed based on how well the proposal meets the requirements of the Terms of Reference using the guidelines detailed in the table below:

When using this weighted scoring method, the award of the contract may be made to the individual consultant whose offer has been evaluated and determined as:

a) Responsive/compliant/acceptable, and

b) Having received the highest score out of a pre-determined set of weighted technical and financial criteria specific to the solicitation.

* Technical Criteria weighting; 70%

* Financial Criteria weighting; 30%

Only candidates obtaining a minimum of 49 points in the Technical Evaluation would be considered for the Financial Evaluation. Interviews may be conducted as part of technical assessment for shortlisted proposals.