Terminal Evaluation Terms of Reference (ToR) for UNDP-supported GEF-financed projects-International and National (2 Positons)

INTRODUCTION

In accordance with UNDP and GEF M&E policies and procedures, all full- and medium-sized UNDP-supported GEF-financed projects are required to undergo a Terminal Evaluation (TE) at the end of the project. This Terms of Reference (ToR) sets out the expectations for the TE of the full-sized project titled “Facilitating Renewable Energy and Energy Efficiency Applications for Greenhouse Gas Emission Reduction (PIMS#5569) implemented through the Climate Change Development Authority and Responsible Parties namely Papua New Guinea (PNG) Power Limited, Milne Bay, Eastern Highlands and East Sepik Provincial Governments. The project started on the 01 October 2017 and is in its 5th year of implementation. The TE process must follow the guidance outlined in the document ‘Guidance for Conducting Terminal Evaluations of UNDP-Supported, GEF-Financed Projects’ Guidance for Terminal Evaluations of UNDP-supported GEF-financed Projects’.

PROJECT BACKGROUND AND CONTEXT

Papua New Guinea’s greenhouse gas (GHG) reduction efforts to date have focused on the forestry sector. Yet, despite low per capita energy use at present, with only 15 percent of the population having access to electricity. The adoption of renewable energy (RE) and energy efficiency (EE) technologies in PNG has strong potential both to reduce current GHG emissions and avoid future, growing GHG emissions expected as rising per capita energy use and electrification accompany development of the nation. Analysis shows that community RE mini-grid and township diesel center-based EE programs present particularly compelling win-win propositions for PNG. Over time, both will present substantial cost savings over the business-as-usual case, in which diesel is used, while at the same time lowering present or avoiding future GHG emissions. FREAGER will demonstrate the relevant technologies for and aim to achieve widespread replication of micro/mini-hydro mini-grids, solar PV mini-grids, and township EE programs. It will aim to do so by removing barriers to these technologies in the areas of policy and planning, technical and commercial viability, availability of financing, and information and awareness. In the policy and planning area, the project will promote these technologies via: capacity building programs for officials, through which the officials develop actual pipeline RE mini-grid projects and township EE programs; design of policy incentives; standards work; national roadmaps for community RE mini-grids and township EE programs; and provincial level RE and EE plans.

Technical and commercial viability will be supported through capacity building for technical personnel on RE mini- grids and on building and industrial EE. It will also be supported through developing and disseminating information on best international sourcing channels for EE and RE and on the “honest,” best costing for community RE mini grids in PNG. Project demos will include two mini-hydro mini-grids and one PV mini-grid, with support also provided for productive use of RE at all three sites. In addition, demos of comprehensive EE programs (including building and industrial energy audits and retrofits, support for residential customers, support for future, large power customers, and street lighting retrofits) will be developed in two townships that are fully powered by diesel. The project will also support demonstration of PPAs and billing systems, as well as O&M training, for two mini-hydro projects that are already under development. Efforts related to financing EE and RE will include capacity building for the financial sector and the set-up of an ESCO fund to finance EE retrofits and a loan fund for community RE projects. Information and awareness work will include support to develop domestic manufacturing of RE mini-grid and EE products (and thus achieve lower cots) and briefings on the cost competitiveness of RE and EE as compared to diesel. It will further include a multi-channel media campaign for RE and EE and educational materials on RE and EE. It will also include RE mini-grid how-to handbooks tailored to PNG, pipeline RE mini-grid project listings, data from RE resource assessments, and curriculum from various project workshops, all available at a one-stop-shop website on RE and EE in PNG.

The project has the following Goal, Objective and Outcomes:

Goal: Reduction in Green House Gas emissions from the energy production and energy end use sectors in PNG

Objective: Enabling the application of feasible renewable energy and energy efficiency technologies for achieving greenhouse gas emission reduction in PNG.

Outcome 1: Rigorous implementation and enforcement of approved national and provincial energy policies, plans, and standards to promote the application of renewable energy and energy efficiency technologies.

Outcome 2A: Enhanced technical-commercial viability and capacity in the application of energy efficiency technologies and development of feasible RE-based energy systems in the country.

Outcome 2B: Increased installed capacity of RE based power systems and implementation of viable EE technology applications in PNG.

Outcome 3: Improved availability of, and access to, financing for renewable energy and energy efficiency initiatives in the energy generation and end-use sectors.

Outcome 4. Improved awareness of, attitude towards, and information about renewable energy and energy efficiency applications in the energy generation and end-use sectors.

TE PURPOSE

The TE report will assess the achievement of project results against what was expected to be achieved and draw lessons that can both improve the sustainability of benefits from this project, and aid in the overall enhancement of UNDP programming. The TE report promotes accountability and transparency and assesses the extent of project accomplishments.

The TE is also being conducted to document best practices, challenges and capacities that are at hand and that are missing that can inform UNDP CO Programing going forward. The UNDP CO Management and the Implementing Partner/Executing Agency will act on the TE Results. The TE is in line with the UNDP PNG’s current evaluation plan. 

The TE will assess project performance against expectations set out in the project’s Logical Framework/Results Framework (see ToR Annex A). The TE will assess results according to the criteria outlined in the Guidance for Terminal Evaluations of UNDP-supported GEF-financed Projects.

TE APPROACH & METHODOLOGY

The TE report must provide evidence-based information that is credible, reliable, and useful. All relevant evidentiary documents must be presented/provided to the TE evaluators to confirm the reported results of the project’s baseline/co-financed and incremental activities, delivery of agreed component outputs and levels of achievement of the end-of-project targets of the objectively verifiable indicators that are set out in the project results framework (log frame). It is important to also provide explanations/justifications of the attribution of any indirect results (e.g., energy savings, GHG emission reductions, etc.) of parallel/associated activities of the project. In this regard, the TE Team must state in the TE report if the team has checked, evaluated, verified and confirmed all the evidentiary documents during the terminal evaluation and provide comments regarding, and where necessary, pertinent recommendations to improve, the credibility, reliability and usefulness of such documents.

The Project Management Office/Unit and the commissioning UNDP country office must provide all relevant sources of information including documents prepared during the preparation phase (i.e. PIF, UNDP Initiation Plan, UNDP Social and Environmental Screening Procedure/SESP) the Project Document, project reports including annual PIRs, project budget revisions, lesson learned reports, national strategic and legal documents, and any other materials that the team considers useful for this evidence-based evaluation. The TE team shall review all of these sources of information, including the baseline and midterm GEF focal area Core Indicators/Tracking Tools submitted to the GEF at the CEO endorsement and midterm stages and the terminal Core Indicators/Tracking Tools that must be completed before the TE field mission begins. 

The TE team is expected to follow a participatory and consultative approach ensuring close engagement with the Project Team, government counterparts (the GEF Operational Focal Point), Implementing Partners, the UNDP Country Office(s), the Regional Technical Advisor, direct beneficiaries and other stakeholders.

Engagement of stakeholders is vital to a successful TE. Stakeholder involvement should include interviews with stakeholders who have project responsibilities, including but not limited to UNDP PNGCO Management, Climate Change Development Authority and PNG Power Limited; the executing agencies, senior officials and task team/component leaders, key experts and consultants in the subject area, Project Board, project beneficiaries, academia, local government and CSOs, etc. Additionally, the TE team is expected to conduct field missions to Alotau in the Milne Bay Province, Goroka in Eastern Highlands Province and Wewak in East Sepik Province , including the following project sites namely Samarai Island, Gotomi, Miruma and Maprik.  

The specific design and methodology for the TE should emerge from consultations between the TE team and the above-mentioned parties regarding what is appropriate and feasible for meeting the TE purpose and objectives and answering the evaluation questions, given limitations of budget, time and data. The TE team must use gender-responsive methodologies and tools and ensure that gender equality and women’s empowerment, as well as other cross-cutting issues and SDGs are incorporated into the TE report.

The final methodological approach including interview schedule, field visits and data to be used in the evaluation must be clearly outlined in the TE Inception Report and be fully discussed and agreed between UNDP, the Implementing Agencies, stakeholders and the TE team.

The final report must describe the full TE approach taken and the rationale for the approach making explicit the underlying assumptions, challenges, strengths and weaknesses about the methods and approach of the evaluation.

DETAILED SCOPE OF THE TE

The TE will assess project performance against expectations set out in the project’s Logical Framework/Results Framework (see ToR Annex A). The TE will assess results according to the criteria outlined in the Guidance for TEs of UNDP-supported GEF-financed Projects Guidance for Terminal Evaluations of UNDP-supported GEF-financed Projects.

The Findings section of the TE report will cover the topics listed below. A full outline of the TE report’s content is provided in ToR Annex C.

The asterisk “(*)” indicates criteria for which a rating is required.

Findings

Project Design/Formulation

• National priorities and country driven-ness
• Theory of Change
• Gender equality and women’s empowerment
• Social and Environmental Standards (Safeguards)
• Analysis of Results Framework: project logic and strategy, indicators
• Assumptions and Risks

• Lessons from other relevant projects (e.g. same focal area) incorporated into project design
• Planned stakeholder participation
• Linkages between project and other interventions within the sector
• Management arrangements

Evaluate whether the project design (e.g., approach, activities and outputs) was adequate/sufficient and appropriate to achieve the project objective and outcomes that were set out in the project results framework.  

Project Implementation

• Adaptive management (approved changes to the project design and project outputs during implementation, whether such changes were adequately and properly implemented, and impacts/results of the implemented changes)
• Actual stakeholder participation and partnership arrangements (in addition, also cite issues/challenges encountered, impacts of such issues/challenges on project implementation and results; and the resolution of these)
• Project Finance and Co-finance (evaluate actual project financing, actual realization of committed co-financing, and any leveraged financing – provide evidentiary documents to support the evaluation)
• Monitoring & Evaluation: design at entry (*), implementation (*), and overall assessment of M&E (*)
• Implementing Agency (UNDP) (*) and Executing Agency (*), overall project oversight/implementation and execution (*)
• Risk Management, including Social and Environmental Standards (Safeguards)

Evaluate whether the actual project implementation did or did not facilitate the provision of the necessary resource inputs for the implementation of project activities and the delivery all the required project outputs.

Project Results

• Assess the achievement of outcomes against indicators by reporting on the level of progress for each objective and outcome indicator at the time of the TE and noting final achievements.

Evaluate the following: (a) whether all the approved project outputs were delivered. These include outputs in the original project design and other approved outputs that were included based on adaptive management; (b) how these outputs contributed to the achievement of the end-of-project targets of the project; and (c) actual resource inputs that were utilized to deliver each output.

• Evaluate the results of the project activities (i.e., GEF-funded and baseline/co-financed activities that were carried out by project partners) that are contributing towards the end-of-project target of the objective indicator and each outcome indicator. This may also include monitored results from indirect activities that were facilitated, enabled or influenced by the FRAGER Project’s activities. The relevant evidentiary documents on these activities must be evaluated to verify and confirm potential attribution of the results to the FREAGER Project.  
• Relevance (*), Effectiveness (*), Efficiency (*) and overall project outcome (*) - For “effectiveness”, evaluate to what extent the barriers that the project is designed to remove were actually removed.
• Sustainability: financial (*)     , socio-political (*), institutional framework and governance (*), environmental (*), overall likelihood of sustainability (*) – For overall likelihood of sustainability, evaluate whether the removed barriers will recur or not, and suggest ways of ensuring that the removed barriers will not recur.
• Country ownership
• Gender equality and women’s empowerment
• Cross-cutting issues (poverty alleviation, improved governance, climate change mitigation and adaptation, disaster prevention and recovery, human rights, capacity development, South-South cooperation, knowledge management, volunteerism, etc., as relevant)
• GEF Additionality
• Catalytic Role / Replication Effect
• Progress to impact

One important issue that must be considered in the reported results that are contributing to the achievement of the project targets is their attribution to the FREAGER Project. Make sure that all declared results are attributable to the Project. Where necessary, explain the attribution or non-attribution.

Main Findings, Conclusions, Recommendations and Lessons Learned

• The TE team will include a summary of the main findings of the TE report. Findings should be presented as statements of fact that are based on analysis of the data, and evidentiary documents. One important issue that must be considered in the reported results that are contributing to the achievement of the project targets is their attribution to the FREAGER Project. Make sure that all declared results are attributable to the Project. Where necessary, explain the attribution or non-attribution.
• The section on conclusions will be written in light of the findings. Conclusions should be comprehensive and balanced statements that are well substantiated by evidence and logically connected to the TE findings. They should highlight the strengths, weaknesses and results of the project, respond to key evaluation questions and provide insights into the identification of and/or solutions to important problems or issues pertinent to project beneficiaries, UNDP and the GEF, including issues in relation to gender equality and women’s empowerment.
• Since the FREAGER Project strategy is barrier removal, one of the main conclusions of the TE must be on the extent of barrier removal that the Project has achieved. Explain in detail (based on the project results) for each project component of the barrier(s) is/are removed, and to what extent the barrier removal was achieved.
• Recommendations should provide concrete, practical, feasible and targeted recommendations directed to the intended users of the evaluation about what actions to take and decisions to make. The recommendations should be specifically supported by the evidence and linked to the findings and conclusions around key questions addressed by the evaluation.
• The TE report should also include lessons that can be taken from the evaluation, including best practices in addressing issues relating to relevance, performance and success that can provide knowledge gained from the particular circumstance (programmatic and evaluation methods used, partnerships, financial leveraging, etc.) that are applicable to other GEF and UNDP interventions. When possible, the TE team should include examples of good practices in project design and implementation.
• It is important for the conclusions, recommendations and lessons learned of the TE report to incorporate gender equality and empowerment of women.

The TE report will include an Evaluation Ratings Table, as shown below:

ToR Table 2: Evaluation Ratings Table for Facilitating Renewable Energy and Energy Efficiency Applications for Greenhous Gas Emission Reduction

TIMEFRAME

The total duration of the TE will be approximately 35 Working days over a time period of 8 weeks starting on 01 August 2022. The tentative TE timeframe is as follows:

Options for site visits should be provided in the TE Inception Report.

[1] Outcomes, Effectiveness, Efficiency, M&E, Implementation/Oversight & Execution, Relevance are rated on a 6-point scale: 6=Highly Satisfactory (HS), 5=Satisfactory (S), 4=Moderately Satisfactory (MS), 3=Moderately Unsatisfactory (MU), 2=Unsatisfactory (U), 1=Highly Unsatisfactory (HU). Sustainability is rated on a 4-point scale: 4=Likely (L), 3=Moderately Likely (ML), 2=Moderately Unlikely (MU), 1=Unlikely (U)