2023 International Joint Initiative for Research in Climate Change Adaptation and Mitigation: Review Process

This document describes the review process for the 2023 International Joint Initiative for Research in Climate Change Adaptation and Mitigation. It is intended to explain the review process and provide a guide for reviewers involved in the assessment of applications.

The 2023 International Joint Initiative for Research in Climate Change Adaptation and Mitigation represents a collaboration among a Consortium of research funders from Brazil, Canada, Germany, Norway, South Africa, Switzerland, the United Kingdom and the United States.

Members of the multidisciplinary/multisectoral review panel and external reviewers are asked to read the relevant sections of this document before beginning to review applications assigned to them. You may ask for clarification on any subject from NFRF program staff at any time.

Overview of the 2023 International Joint Initiative for Research in Climate Change Adaptation and Mitigation

The 2023 International Joint Initiative for Research in Climate Change Adaptation and Mitigation call will support interdisciplinary (integrating information, data, techniques, tools, perspectives, concepts or theories from two or more disciplines or bodies of specialized knowledge) and transsectoral (involving the academic, research, economic (businesses), societal―governmental and non-governmental organizations―and community sectors as appropriate) research on participatory, contextually and culturally appropriate mitigation and adaptation responses to at least two of the eight representative key risks of climate change.

Projects must focus on responding to the needs of those most impacted by the effects of climate change, such as communities in low- and middle-income countries, Indigenous territories or groups that are vulnerable due to their geographic, social and/or economic circumstances. All projects are required to partner with participating communities) in the co-creation, implementation and ownership of the research and outcomes, and to develop approaches related to policy implementation and knowledge mobilization. In this way, the call aims to strengthen the connections between research, governance and communities, to ensure that funded projects are both transformative and impactful. This approach ensures that the projects also develop strategies related to policy, communication and community partnerships, to encourage acceptance, support and the behavioural changes required for implementation. The integration of team members from vulnerable groups is required.

For further details on the call, including eligibility requirements, see the call document. The instructions for the notice of intent to apply (NOI) and full application provide more details on the information required at each stage of the competition.

Competition stages

To apply to this call, teams must submit a notice of intent to apply (NOI) by May 2, 2023. The research team must then submit a full application by September 12, 2023. The review process occurs in two stages, as depicted in Figure 1.

NOIs will be used for administrative purposes to:

• Assess the eligibility of team members to apply for funding from the selected Consortium Partners; and
• Identify external reviewers and compose the multidisciplinary/multisectoral review panel.

Full applications will be reviewed by external reviewers and assessed by members of the multidisciplinary/multisectoral review panel, considering the external reviewers’ input.

Principles of Merit Review

Full applications are assessed by external reviewers and members of the multidisciplinary/multisectoral review panel. External reviewers and review panel members are all considered to be “reviewers,” to whom the following principles apply.

Fairness

The success of the merit review system depends on the willingness and ability of all reviewers in the process to be fair and reasonable; to exercise rigorous judgment; and to understand—and consider in a balanced way—the context of each application.

Bias

All reviewers are asked to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought; fundamental versus applied research; certain subdisciplines; areas of research or approaches (including emerging ones); size or reputation of an institution; or the age, personal factors, sex or gender of the applicants. All reviewers are cautioned against judging an application based on these factors. Before review panel members are able to view any applications, they must first complete the required Unconscious Bias Training Module.

Conflict of interest and confidentiality

The Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations ensures the effective management of conflict of interest of any participant in the review process and, during the review process, ensures the confidentiality of personal and commercial information submitted to the program.

Conflict of interest

Members are responsible for evaluating the merits of applications assigned to them for review, except for those for which they have a conflict of interest.

A conflict of interest is a conflict between a person’s duties and responsibilities as a participant in the review process and that person’s private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the review panel member, external reviewer, referee or observer:

• would benefit professionally or personally if the application being reviewed is funded;
• has a professional or personal relationship with any of the applicants (this includes nominated principal investigators, co-principal investigators, co-applicants and collaborators) or applicants’ institutions; or
• has a direct or indirect financial interest in the application being reviewed.

A conflict of interest may be deemed or perceived to exist when a review panel member, external reviewer or observer:

• is a relative or close friend, or has a personal relationship with any of the applicants;
• could gain or lose financially/materially from the funding of the application;
• has had longstanding scientific or personal differences with any of the applicants;
• is currently affiliated with any of the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
• is professionally affiliated with any of the applicants as a result of having, in the last six years:
  • had frequent and regular interactions with any of the applicants while carrying out their duties at their department, institution, organization or company;
  • been a supervisor or a trainee of any of the applicants;
  • collaborated, published or shared funding with any of the applicants, or made plans to do so soon; or
  • been employed by any of the applicants’ institutions; and/or
• feels for any reason unable to impartially review the application.

All review panel members are subject to the same conflict of interest guidelines. If any of these situations apply, the member must declare a conflict of interest. NFRF staff are responsible for resolving areas of uncertainty. A conflict of interest can be declared at any time during the competition cycle. Contact NFRF program staff if your conflict of interest status changes.

Confidentiality

The information included by applicants in their applications is protected by Canada’s Privacy Act and is provided for the purposes of review only. Details of the application, scoring and recommendation for a specific application are confidential and must never be divulged. Under no circumstances should members disclose to anyone the recommendations from the scoring; this also applies after the competition is over and award recipients are announced.

Recommendations made by the review panel are subject to approval and may be changed for reasons of budget, administrative error or lack of full adherence to policies.

Review panel members are asked not to communicate any information about the review of a specific application or offer to applicants or anyone outside of the panel or share opinions on the applicants’ chances of success or failure.

In turn, applicants are not to contact review panel members regarding the status of their applications (ratings, rank, etc.). By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of the funding decision. This does not include any notes or feedback a review panel member has provided that were not shared in writing with other review panel members.

Roles and responsibilities

External Reviewers

External reviewers are individuals with expertise related to a specific application. External reviewers are asked to provide a written assessment of the application according to the criteria. Given the interdisciplinary nature of the projects, external reviewers are asked to comment only on the aspects of the proposal they are comfortable assessing, making note of the aspects they are unable to assess. Members of the multidisciplinary/multisectoral review panel will consider the external reviewer assessments in their evaluation of the application.

External reviewers typically review only one or two applications but may be invited to review up to five.

Multidisciplinary/Multisectoral Review Panel

A multidisciplinary/multisectoral review panel composed of international members with expertise across a broad range of research disciplines will evaluate the applications. To uphold the call’s commitment to equity, diversity and inclusion (EDI) and to ensure the highest quality of review, members of the panel will be diverse, taking into consideration factors such as area of expertise; sector; language; diversity; region; institution size; career stage; knowledge of EDI best practices; and experience in research by and with Indigenous Peoples and vulnerable groups. Members are selected to ensure the panel has the capacity to review applications in both of Canada’s official languages.

The review panel recommends which applications should be funded and prepares a ranked list of applications based on their relative merit.

Members

Review panel members evaluate applications and, with other members, make recommendations based on their assessments. Specific responsibilities of members include:

• ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the evaluations and/or meeting;
• ensuring the integrity and quality of the merit review process, and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• completing the online training module on unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions before conducting evaluations and attending the meeting;
• reviewing, in depth, the applications assigned to them;
• ensuring that all documents are considered in their assessments;
• providing ratings for applications assigned to them by the deadline;
• participating in the review of applications assigned to them by providing a verbal assessment to the panel during the meeting, where applicable;
• preparing feedback to applicants, where applicable; and
• ensuring that all review materials provided are handled safely and disposed of according to program policy.

Co-chairs

Co-chairs are responsible for ensuring that the multidisciplinary/multisectoral review panel deliberations function smoothly, effectively and objectively, according to the call’s policies. The co-chairs establish a positive, constructive and fair-minded environment in which the applications are evaluated. They fulfill an oversight role and participate in the review of applications.

The co-chairs’ responsibilities include:

• ensuring that the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations is followed before, during and after the meeting;
• ensuring the integrity and quality of the merit review process, and that each application receives a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• completing the online training module on unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions before the merit review meeting;
• including all assigned members in the discussion of applications;
• ensuring that all documents are considered in the panel’s assessments;
• guiding the panel to a consensus recommendation, where applicable;
• assisting with the preparation of feedback to applicants, where applicable;
• contributing to discussions on policy issues; and
• ensuring that all confidential review materials provided to them are handled safely and disposed of according to program policy.

2023 International Joint Initiative for Research in Climate Change Adaptation and Mitigation Consortium Partners’ Committee

The Committee consists of representatives of each Consortium Partner involved in the multilateral international initiative. The Committee receives recommendations from the multidisciplinary/multisectoral review panel and identifies projects to be funded, considering the ranked order of applications following merit review, the funding requested in each application, and the funding available from each Partner. The Committee members, on behalf of their organizations, also ensure the evaluation process is rigorous, objective and transparent, in keeping with the standards of excellence expected by the Consortium Partners and consistent with the call’s objectives.

NFRF staff

NFRF staff are responsible for delivering the competition according to the policies and processes of the call. The staff’s responsibilities include:

• ensuring that the panel follows the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the review meeting;
• ensuring the integrity and quality of the merit review process and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• assigning applications to review panel members;
• providing advice and guidance to the panel on the program’s policies;
• ensuring that all documents are considered in the panel’s assessments;
• keeping notes on procedural aspects of the panel’s functions;
• recording comments made by the panel for each application, where applicable;
• working with the co-chairs to manage conflicts of interest, where applicable;
• recording concerns raised by the panel on issues requiring subsequent attention by staff;
• ensuring that all review materials provided are handled safely and disposed of according to program policy; and
• facilitating the final approval of recommended applications.

Review process

The purpose of the application review process is to identify the most meritorious applications to be recommended for funding by the multidisciplinary/multisectoral review panel, while adhering to the principles of merit review.

Evaluation by external reviewers

All efforts are made to secure a minimum of two external reviews for each application. External reviewers do not provide ratings but are asked to comment on the fit to program, interdisciplinarity, high risk, high reward and feasibility criteria. External reviewers will have access to all parts of the application, except the EDI in the research practice section.

External reviewers must check for conflicts of interest with any application they are invited to review. If there is a conflict of interest, please notify NFRF staff and do not proceed with the review.

Assessment

External reviewers comment on the interdisciplinarity, high risk, high reward and feasibility components of the proposed project, as follows:

• Does the project fit the expectations of the call? (yes/no)
  • Comment on the fit-to-program
• Is the project interdisciplinary? (yes/no)
  • Comment on the interdisciplinary nature of the project
• Is the project high risk? (yes/no)
  • Describe  the strengths and weaknesses
• Is the project high reward? (yes/no)
  • Describe the strengths and weaknesses
• Is the project feasible? (yes/no)
  • Describe the strengths and weaknesses
• Any areas of concern or comments related to the proposal

Evaluation by the multidisciplinary/multisectoral review panel

Assignment

Each application is assigned to three review panel members. Language abilities and institutional conflicts of interest will be considered in assigning applications to members.

Check for conflicts of interest

Shortly after the deadline, review panel members will receive an email informing them that their list of assignments is ready. Members are asked to log in to the Convergence Portal, review the summary of each of their assigned applications in the “Ability to Review” tab, and indicate if there is a conflict of interest. Once a member has checked all assigned applications for conflicts, they immediately gain access to the applications for which they are not in conflict in the “Committee Assignments” tab. Some changes to assignments may occur as conflicts of interest are identified by members. If there are any changes, the panel member will be notified by NFRF program staff. Any problems should be brought to the attention of NFRF program staff as soon as possible, or at any point in the process.

Assessment

Multidisciplinary/multisectoral review panel members receive access to all content from the NOI and full application, as well as to the external reviews.

All assigned applications are to be reviewed and scored according to the merit indicator matrices for the selection criteria:

• Fit-to-program (pass/fail)
• interdisciplinarity (pass/fail)
• EDI (pass/fail)
• high risk: 20% of overall score
• high reward: 40% of overall score
• feasibility: 40% of overall score

Members can assign one of seven ratings for the high risk, high reward and feasibility criteria (see Merit indicators). In assessing feasibility, members must consider the approach to Indigenous research (if applicable) and EDI considerations in the research design.

Members are asked to provide comments for any criterion assessed as a “fail” and may also comment on any areas of concern related to the application.

Result

Members’ ratings for the high risk, high reward and feasibility criteria will determine an overall score for each application. The final rating by criterion will be determined by the mean of members’ individual ratings for the criteria. The overall score for an application is a weighted average of the final rating by criterion: 20% high risk, 40% high reward, 40% feasibility. Overall scores will be used to produce a ranked list with the strongest applications having the highest scores.

The overall scores will be used to identify the top-rated applications that will be discussed by the review panel. The cut-off band for those to be discussed will be determined by the distribution of scores and the number of applications received. Members can identify any application to be discussed at the meeting, regardless of its overall score. All other applications will not be discussed, will not be recommended for funding, and will not receive comments from the review panel.

Multidisciplinary/multisectoral review panel meetings

The review panel meets virtually to discuss applications and identify those to be recommended for funding. Applications to be discussed will be divided between multiple virtual rooms and members will discuss applications across their assigned rooms. Each discussion is presided by a co-chair. NFRF program staff are present to help the co-chairs and answer program- and policy-related questions.

Discussion of applications

Twenty minutes are allocated to discuss each application. The discussion is moderated by one of the co-chairs, who ensures that all assigned members participate in discussing the strengths and weaknesses of the application according to each criterion. Following discussion, members will be invited to adjust their ratings if the discussion changed their assessment of one or more criteria.

Final meeting

The final list ranking all applications according to their overall merit will be agreed to. The ranked list will be created based on the final overall scores of each application. The review panel may consider the following elements of diversity to indicate priority between equally rated applications:

• risks not tackled by other proposals ranked as high or higher in the call;
• an uncommon combination of risks;
• challenges faced by vulnerable groups in unique geographic regions not tackled by other proposals ranked as high or higher.

Feedback to applicants

Concise feedback is prepared for all applications discussed at the meetings. One of the reviewers is identified in advance to lead preparation of the feedback for each application.

Consortium Partners Committee

The Consortium Partners Committee will meet to review the ranked list resulting from the review process. The list will be used to determine which applications will be funded. Applications will be funded in their ranked order from highest score to lowest. An application in the list will only be skipped in cases where one Partner has no budget remaining to award to it.

Review timeline

Applications and review material

Incomplete or non-adherent applications

The onus is on the applicants to provide complete and sufficient information that adheres to the Convergence Portal instructions for attachments and the instructions for completing the NOI and full application. Problems related to application content should be brought to the attention of NFRF program staff, which can be done at any point during the competition. To maintain fairness in the competition, applicants must adhere to the guidelines in preparing application materials. If staff determine that information provided is incomplete or does not adhere to guidelines or instructions, the application may be removed from the competition.

Eligibility of applicants

Eligibility decisions are the responsibility of the Consortium Partners’ staff. Review panel members who have doubts about a researcher’s eligibility should review the application on the same basis as all others and alert NFRF staff to potential eligibility problems as soon as possible. The eligibility criteria for applicants can be found in the eligibility sections of the call material annexes.

Indigenous research

If a review panel member or external reviewer receives a proposal in which the applicants have answered “Yes” to the question “Does your proposal involve Indigenous research as defined by SSHRC?”, they must use SSHRC’s Indigenous Research Statement of Principles and Guidelines for the Merit Review of Indigenous Research as references in assessing the application related to or involving Indigenous research. The guidelines are provided to help reviewers build understanding of Indigenous research and research-related activities, and to assist them in interpreting the specific evaluation criteria in the context of Indigenous research.

Merit indicators

The merit indicators include pass/fail indicators for the fit-to-program, interdisciplinarity and EDI criteria, and an evaluation scale for the high risk, high reward and feasibility indicators. The evaluation scales include references to major points of consideration to guide review panel members toward arriving at a rating for a given criterion. Although only four ratings are provided for each evaluation scale, members may select a rating between each rating described (e.g., between exceptional and very good), for a seven-point scale.

All applications are evaluated using the same merit indicators. Review panel members are encouraged to use the full range of ratings, as appropriate, to achieve a distribution of ratings that reflects the quality of the applications being evaluated. Members are expected to discuss and justify (or adjust) their ratings during the review panel meeting.

Areas of concern

All reviewers can comment about any areas of concern in the application, including ones related to any of the selection criteria, including EDI in research practice (review panel members only); fit to program; interdisciplinarity; EDI in the research design; Indigenous research; budget; or others.

Note that reviewers must not include any identifying information about the team members (names, research groups, departments, institutions, etc.) when entering their comments in the Convergence Portal.

Budget

While the proposed budget is not assessed as part of the selection criteria, it will be reviewed and discussed, particularly where it may affect the feasibility of completing the research with the proposed resources.

Handling documents used in peer review

Merit review documents contain personal information as well as information that the unauthorized disclosure of could reasonably be expected to cause serious injury (such as prejudicial treatment, loss of reputation or competitive advantage) to an individual, organization or government. Therefore, these documents are subject to regulation under Canada’s Privacy Act, the Access to Information Act and the Policy on Government Security. Protocols must be followed to ensure that information contained in applications, internal and external reviews, and panel discussions remains strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Refer to the Guide on Handling Documents Used in Peer Review for further details.

How to access applications for review

All review panel members and external reviewers will access the applications for review through the Convergence Portal, and will only have access to applications they have been assigned.

Multidisciplinary/multisectoral review panel members

Potential review panel members will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profile. Shortly after the application deadline, review panel members will receive an email asking them to accept the terms and conditions (T&Cs) in the Convergence Portal. Members must accept the T&Cs before they can continue with the review process. After acceptance, two new tabs will appear on the home screen when the member logs into the Convergence Portal: “Ability to Review” and “Committee Assignments.” If the T&Cs are accepted before applications are assigned, both tabs will remain blank until applications are assigned.

Following this step, members will receive an email informing them that the applications assigned to them are ready under the “Ability to Review” tab. Any problem with the assignment of applications should be brought to the attention of NFRF program staff as soon as possible, or at any point in the process.

External reviewers

External reviewers will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profiles. Following this, they will receive an email asking them to accept the T&Cs of the Convergence Portal. External reviewers must accept the T&Cs before they can continue with the review process.

Once the reviewer has accepted these, a new tab will appear on their home screen when they are logged into the Portal: “Reviewer Assignments.” If the T&Cs are accepted before applications are assigned, this tab will remain blank until applications are assigned.

Shortly after the application deadline, external reviewers will receive an email informing them that the applications assigned to them are ready under the “Ability to Review” tab. Any problem with the assignment of applications (e.g., conflict of interest) should be brought to the attention of NFRF program staff as soon as possible, or at any point in the process. Once the external reviewer indicates that there is no conflict, they will be able to access the application in the “Reviewer Assignments” tab.

Legal and ethical information

Responsible conduct of research

Canada’s federal research funding agencies—CIHR, NSERC and SSHRC—are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions and the agencies that, together, help support and promote a positive research environment.

Confidentiality

Members appointed to the review panel must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers describing expectations and requirements.

Canada’s Privacy Act

Personal information refers to any information about an identifiable individual. Based on Canada’s Privacy Act, personal information provided by applicants must be used only for assessing applications, making funding decisions, and related uses at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal. It is important for panel members to adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Canadian Human Rights Act

The activities of CIHR, NSERC and SSHRC are subject to the Canadian Human Rights Act. The purpose of the Act is to extend the laws in Canada to give effect to the principle that all individuals should have opportunity equal with other individuals to make for themselves the lives that they are able and wish to have. They should also have their needs accommodated, consistent with their duties and obligations as members of society, without being hindered in or prevented from doing so by discriminatory practices based on race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, genetic characteristics, disability or conviction for an offence for which a pardon has been granted or in respect of which a record suspension has been ordered.

Canada’s Official Languages Act

All review panel members and NFRF program staff must be aware of their obligations and rights as legislated in Canada’s Official Languages Act.

A word of thanks

On behalf of the Canada Research Coordinating Committee (CRCC) and the Consortium Partners, we would like to thank external reviewers, members of the multidisciplinary/multisectoral review panel, and co-chairs for agreeing to participate in the review process for the International Joint Initiative for Research in Climate Change Adaptation and Mitigation. The success of the review process is made possible by dedicated people like you who generously give your time and expertise. The CRCC, the Consortium Partners and the global research community greatly appreciate your efforts.