2022 Special Call–Research for Postpandemic Recovery: Reviewer Manual


On this page


Purpose of the manual

This manual is a guide for reviewers for the New Frontiers in Research Fund (NFRF) 2022 Special Call—Research for Postpandemic Recovery. It describes activities to be undertaken by external reviewers, by Multidisciplinary/Multisectoral Review Panel members and by panel co-chairs, and outlines the policies, guidelines and deliverables relevant to these activities. The content is intended to guide reviewers and outline principles rather than provide applicants with a set of rules.

A word of thanks

On behalf of the Canada Research Coordinating Committee (CRCC), we would like to thank you for agreeing to participate in the review process for the NFRF 2022 Special Call—Research for Postpandemic Recovery. The success of the review process is made possible by dedicated people like you who generously give your time and expertise. The CRCC and the research community greatly appreciate your efforts.

Review panel members and external reviewers are asked to read the relevant sections of this document in their entirety before beginning to review applications assigned to them. You may ask for clarification on any subject from NFRF program staff at any time.

Overview of the 2022 Special Call—Research for Postpandemic Recovery

The goal of this Special Call is to mobilize Canadian-led research efforts in support of a more equitable, sustainable and resilient postpandemic reality. It is expected this competition will support a diverse portfolio of projects that directly address one or more of the research priorities outlined in the UN Research Roadmap for the COVID-19 Recovery, comprising any of the UN’s priorities and subpriorities, including, but not limited to, the “quick-win”, “best-buy” and “game-changer” priorities.

These Special Call grants support innovative projects that leverage ingenuity and research from the full range of disciplines to inform solutions to the socio-economic consequences of the COVID-19 crisis, and to guide the design and implementation of recovery efforts that can accelerate progress towards the UN’s Sustainable Development Goals.

Diversity of perspectives is important, and the fund encourages research proposals that involve disciplines, thematic areas, approaches or subject areas eligible for funding across the three research funding agencies. Given the socio-economic framework and goals of the UN Roadmap, proposals are expected to include at least one expert in social, economic, or socio-economic research and/or implementation science among the principal investigators (nominated PI or co-PI).

Eligible proposals include short- and medium-term projects, as well as discrete phases of longer-range research programs. All projects are expected to have demonstrable results/impact by the end of the grant.

For further details on the 2022 competition, including eligibility requirements, see the competition overview. The instructions for the notice of intent to apply (NOI) and full application provide more details on the information required at each stage of the competition.

Principles of Merit Review

Fairness

Success of the NFRF merit review system depends on the willingness and ability of all reviewers in the process to be fair and reasonable; to exercise rigorous judgment; and to understand—and consider in a balanced way—the particular context of each application.

Bias

All reviewers are asked to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought; fundamental versus applied research; certain subdisciplines; areas of research or approaches (including emerging ones); size or reputation of an institution; or the age, personal factors, sex or gender of the applicants. All reviewers are cautioned against judging an application based on these factors. Before review panel members are able to view any applications, they must first complete the required Unconscious Bias Training Module.

Conflict of interest and confidentiality

The Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations ensures the effective management of conflict of interest of any participant in the review process and, during the review process, ensures the confidentiality of personal and commercial information submitted to the program.

Conflict of interest

Members are responsible for evaluating the merits of applications assigned to them for review, with the exception of those for which they have a conflict of interest.

A conflict of interest is a conflict between a person’s duties and responsibilities as a participant in the review process, and that person’s private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the review panel member, external reviewer, referee or observer:

  • would benefit professionally or personally as a result of the application being reviewed;
  • has a professional or personal relationship with any of the applicants (this includes nominated principal investigators, co-principal investigators, co-applicants and collaborators) or applicants’ institutions; or
  • has a direct or indirect financial interest in the application being reviewed.

A conflict of interest may be deemed or perceived to exist when a review panel member, external reviewer or observer:

  • is a relative or close friend, or has a personal relationship with any of the applicants;
  • is in a position to gain or lose financially/materially from the funding of the application;
  • has had longstanding scientific or personal differences with any of the applicants;
  • is currently affiliated with any of the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
  • is professionally affiliated with any of the applicants as a result of having in the last six years:
    • had frequent and regular interactions with any of the applicants in the course of their duties at their department, institution, organization or company;
    • been a supervisor or a trainee of any of the applicants;
    • collaborated, published or shared funding with any of the applicants, or made plans to do so in the immediate future; or
    • been employed by any of the applicants’ institutions; and/or
  • feels for any reason unable to provide an impartial review of the application.

All review panel members are subject to the same conflict of interest guidelines. If any of these situations apply, the member must declare a conflict of interest. NFRF staff are responsible for resolving areas of uncertainty. A conflict of interest can be declared at any time during the competition cycle. Please contact NFRF program staff if you have any changes.

Confidentiality

The information included by applicants in their applications is protected by the Privacy Act and is provided for the purposes of review only. Details of the application, scoring and recommendation for a specific application are confidential and must never be divulged. Only NFRF staff can release information. Under no circumstances should members disclose to anyone the recommendations from the scoring; this also applies to when the competition is over and the award recipients are announced.

Funding recommendations made by the review panel are subject to approval by the NFRF steering committee and may be changed for reasons of budget, administrative error, or lack of full adherence to policies.

Review panel members are asked not to communicate any information relating to the review of a specific application, or offer to applicants or anyone outside of the panel opinions on the applicants’ chances of success or failure.

In turn, applicants are not to contact panel members regarding the status of their applications (ratings, rank, etc.). By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of the funding decision for a competition. This does not include any notes or feedback a review panel member has provided that were not shared in written form with the other panel members or NFRF staff.

Roles and responsibilities

Strategic and program steering committees

The CRCC provides strategic direction and oversight for the NFRF and serves as the high-level steering committee. As a tri-agency program, program oversight is delegated to the NFRF steering committee, which includes the presidents of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC); and the deputy ministers of Innovation, Science and Economic Development Canada, and Health Canada. The NFRF Steering Committee makes decisions on which applications to fund based on the review panel’s recommendations. It also ensures the evaluation process is rigorous, objective and transparent, in keeping with the standards of excellence expected by the funding agencies and consistent with the program’s objectives.

Multidisciplinary/Multisectoral Review Panel

Membership

The Multidisciplinary/Multisectoral Review Panel is composed of experts who represent the various research disciplines of the three federal research funding agencies (CIHR, NSERC and SSHRC). To achieve a balance among panel members, factors such as area of expertise; language; inclusion in the four designated groups (women, Indigenous Peoples, racialized minorities and persons with disabilities); region; institution size; career stage; knowledge of best practices for equity, diversity and inclusion (EDI); and experience in research by and with Indigenous Peoples are taken into account, as is the inclusion of non-academics. Members are selected to ensure the panel has the capacity to review proposals in both of Canada’s official languages.

The Multidisciplinary/Multisectoral Review Panel makes recommendations to the NFRF Steering Committee about which applications should be funded.

Co-chairs

Co-chairs are responsible for ensuring that the Multidisciplinary/Multisectoral Review Panel deliberations function smoothly, effectively and objectively, according to the program’s policies. The co-chairs establish a positive, constructive and fair-minded environment in which the applications are evaluated. They fulfill an oversight role and participate in the review of applications.

The co-chairs’ responsibilities include:

  • ensuring that the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations is followed before, during and after the meeting;
  • ensuring the integrity and quality of the merit review process, and that each application receives a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
  • ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
  • completing the online training module on unconscious bias in peer review;
  • participating in preparatory meetings/discussions and information sessions before the merit review meeting;
  • including all members in the review of applications;
  • ensuring that all documents are considered in the panel’s assessments;
  • guiding the panel to a consensus recommendation, where applicable;
  • assisting with the preparation of feedback to applicants, where applicable;
  • contributing to discussions on NFRF policy issues;
  • participating in discussions on Multidisciplinary/Multisectoral Review Panel membership for future competitions; and
  • ensuring that all confidential review materials provided to them are handled safely and disposed of according to program policy.

Multidisciplinary/Multisectoral Review Panel members

Review panel members evaluate applications and make recommendations to the NFRF Steering Committee based on their assessments. Specific responsibilities of members include:

  • ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the evaluations and/or meeting;
  • ensuring the integrity and quality of the merit review process, and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
  • ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
  • completing the online training module on unconscious bias in peer review;
  • participating in preparatory meetings/discussions and information sessions before evaluations and/or the merit review, where applicable;
  • reviewing in depth the applications assigned to them;
  • ensuring that all documents are considered in their assessments;
  • providing ratings for applications assigned to them by the deadline or before the merit review meeting, where applicable;
  • participating in the review of applications assigned to them by providing a verbal assessment to the panel during the meeting, where applicable;
  • participating in the discussion and review of all relevant applications, according to their role, where applicable;
  • preparing feedback to applicants, where applicable; and
  • ensuring that all review materials provided are handled safely and disposed of according to program policy.

NFRF staff

NFRF staff are typically represented at the merit review meetings by the director, deputy director and/or manager, and the program officers. The staff’s responsibilities include:

  • ensuring that the panel follows the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the review meeting;
  • ensuring the integrity and quality of the merit review process and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
  • assigning applications to review panel members;
  • providing advice and guidance to the panel on the program’s policies;
  • ensuring that all documents are considered in the panel’s assessments;
  • keeping notes on procedural aspects of the panel’s functions;
  • recording comments made by the panel for each application, where applicable;
  • working with the co-chairs to manage conflicts of interest, where applicable;
  • recording concerns raised by the panel on issues requiring subsequent attention by staff;
  • ensuring that all review materials provided are handled safely and disposed of according to program policy; and
  • facilitating the final approval of recommended applications by the NFRF Steering Committee.

Applications and review material

Incomplete or non-adherent applications

The onus is on the applicants to provide complete and sufficient information that adheres to the Convergence Portal instructions for attachments and the instructions for completing the NOI and full application. Problems related to application content should be brought to the attention of NFRF program staff, which can be done at any point during the competition. To maintain fairness in the competition, applicants must adhere to the guidelines in preparing application materials. If staff determines that information provided is incomplete or does not adhere to guidelines or instructions, the application may be disqualified.

Eligibility of applicants

Eligibility decisions are the responsibility of the NFRF program staff. Review panel members who have doubts about a researcher’s eligibility should review the application on the same basis as all others and alert the NFRF staff to potential eligibility problems as soon as possible. The eligibility criteria for applicants can be found in the eligibility section of the competition overview.

Indigenous research

If a review panel member or external reviewer receives a proposal in which the applicants have answered “Yes” to the question “Does your proposal involve Indigenous research as defined by SSHRC?” they must use SSHRC’s Indigenous Research Statement of Principles and Guidelines for the Merit Review of Indigenous Research as references in assessing the application related to or involving Indigenous research. The guidelines are provided to help reviewers build understanding of Indigenous research and research-related activities, and to assist them in interpreting the specific evaluation criteria in the context of Indigenous research.

Merit indicators

The merit indicators include pass/fail indicators for the interdisciplinarity/fit to program and EDI criteria, and an evaluation scale for the high risk, high reward and feasibility indicators. The evaluation scales include references to major points of consideration to guide review panel members toward arriving at a rating for a given criterion. Although only four ratings are provided for each evaluation scale, members may select a rating between each rating described, for a seven-point scale.

All applications are evaluated using the same merit indicators. Review panel members are encouraged to use the full range of ratings, as appropriate, to achieve a distribution of ratings that reflects the quality of the applications being evaluated. Members are expected to discuss and justify (or adjust) their ratings during the full application stage.

Areas of concern

All reviewers can comment about any areas of concern in the application, including ones related to any of the selection criteria, including EDI (review panel members only) and interdisciplinarity; gender-based analysis plus (GBA+); Indigenous research; budget; or others.

Please note that reviewers must not include any identifying information about the team members (names, research groups, departments, institutions, etc.) when entering their comments.

Budget

The proposed budget is not one of the selection criteria. It is not typically discussed by reviewers, except where it may affect the assessment of feasibility in accomplishing the research with the proposed resources. However, if a reviewer notices an expense that is not eligible under the Tri-Agency Guide on Financial Administration, or one they perceive has been greatly overestimated, they should bring it to the attention of NFRF program staff.

How to access applications for review

All review panel members and external reviewers will access the applications for review through the Convergence Portal, and will only have access to applications they have been assigned.

Multidisciplinary/Multisectoral review panel members

Potential review panel members will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profile. Shortly after the application deadline, Multidisciplinary/Multisectoral Review Panel members will receive an email asking them to accept the terms and conditions (T&Cs) in the Convergence Portal. Members must accept the T&Cs before they can continue with the review process. After acceptance, two new tabs will appear on the home screen when the member logs into the Convergence Portal: “Conflicts of Interest” and “Committee Assignments.” If the T&Cs are accepted before applications are assigned, both tabs will remain blank until applications are assigned.

Following this step, members will receive an email informing them that the applications assigned to them are ready under the “Conflicts of Interest” tab. Any problem with the assignment of applications should be brought to the NFRF program staff’s attention as soon as possible, or at any point in the process.

External reviewers

External reviewers will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profiles. Following this, they will receive an email asking them to accept the T&Cs of the Convergence Portal. External reviewers must accept the T&Cs before they can continue with the review process.

Once the reviewer has accepted these, a new tab will appear on their home screen when they are logged into the Portal: “Reviewer Assignments.” If the T&Cs are accepted before applications are assigned, this tab will remain blank until applications are assigned.

Shortly after the application deadline, external reviewers will receive an email informing them that the applications assigned to them are ready under the “Reviewer Assignments” tab. Any problem with the assignment of applications should be brought to the attention of NFRF program staff as soon as possible, or at any point in the process.

Review process

Full application stage

The purpose of the application review process is to identify the most meritorious applications to be recommended for funding by the Multidisciplinary/Multisectoral Review Panel.

Review timeline

Date Activity
August 9, 2022 Application deadline
August 15, 2022 External reviewers are given access to applications
August 30, 2022 Review panel members receive assignments
August 30, 2022 Orientation and calibration sessions for members
September 16, 2022 Deadline for review panel members to indicate conflicts of interest
September 30, 2022 Deadline for external reviewer assessments
November 4, 2022 Deadline for review panel members to record scores
November 14-16, 2022 Scores are analyzed and applications are identified for discussion by members through virtual meetings
November 28 - December 2, 2022 Multidisciplinary/Multisectoral Review Panel virtual meetings
December 5-9, 2022 Review panel members finalize comments for the applications that were discussed
January 2023 Application results are released to applicants
February 2023 Feedback survey is sent to review panel members

Evaluation by external reviewers

External reviews focus on the proposal itself. External reviewers are asked to comment on the high risk and high reward criteria, as well as the feasibility criterion as it relates to the research plan only. External reviewers will have access to the summary from the NOI, plus the GBA+ section, research proposal, budget justification, literature references, and applicants’ biographical information attachments from the full application. External reviewers will not have access to an application’s EDI section.

Assignment

Each application will be assigned to two external reviewers with expertise in the fields of the application. External reviewers will typically review only one or two applications but may be invited to review up to five. Since the projects are interdisciplinary, external reviewers are asked to comment only on the aspects of the proposal they are comfortable assessing and to note those they are not.

External review process

External reviewers comment on the high risk, high reward and feasibility components of the proposed project, as follows:

  • Is the project high risk? (yes/no)
    • Description of the strengths and weaknesses
  • Is the project high reward? (yes/no)
    • Description of the strengths and weaknesses
  • Is the project feasible? (yes/no)
    • Description of the strengths and weaknesses
  • Any areas of concern or comments related to the proposal

Evaluation by the Multidisciplinary/Multisectoral Review Panel

Assignment

Each application is assigned to three Multidisciplinary/Multisectoral Review Panel members for review. Language abilities and institutional conflicts of interest will be considered in assigning applications to members.

Check for conflicts of interest

Shortly after the deadline, review panel members will receive an email informing them that their list of assignments is ready. Members are asked to log in to the Convergence Portal, review the summary of each of their assigned applications in the “Conflicts of Interest” tab, and indicate if there is a conflict of interest. Once a member has checked all assigned applications for conflicts, they immediately gain access to the applications for which they are not in conflict in the “Committee Assignments” tab. Some changes to assignments may occur, as conflicts of interest are identified by members. If there are any changes, the panel member will be notified by NFRF program staff. Any problems should be brought to the attention of the NFRF program staff as soon as possible, or at any point in the process.

Assessment

Multidisciplinary/Multisectoral Review Panel members receive access to all content from the NOI and full application stages, as well as to the external reviews.

All eligible applications are to be reviewed and scored according to the merit indicator matrices for the selection criteria:

  • high risk: 30% of overall score
  • high reward: 40% of overall score
  • feasibility: 30% of overall score
  • interdisciplinarity / fit to program (pass/fail)
  • EDI (pass/fail)

Members can assign one of seven ratings for the high risk, high reward and feasibility criteria (see Merit indicators) . In assessing feasibility, members must consider the approach to Indigenous research (if applicable) and GBA+ considerations in the research design.

Members are asked to comment on:

  • Interdisciplinarity / fit to program (if the score is a fail)
  • EDI (if the score is a fail)

Members can also comment on any areas of concern related to the application.

Result

Based on the Multidisciplinary/Multisectoral Review Panel ratings, applications will be placed in “bands” of equally rated applications.

Top-rated applications will be discussed by the Multidisciplinary/Multisectoral Review Panel. The cut-off band for those to be discussed will be determined based on the distribution of scores and the number of applications received. Members can identify any application to be discussed at the meeting, regardless of its overall score.

Remaining applications will not be discussed and will not be recommended for funding.

The CRCC has committed to ensuring that a proportion of the awards will be reserved for applications led by early career researchers (ECRs), equal to the proportion of ECR-led applications received. The program monitors the proportion of ECR-led applications recommended for funding. It is possible that lower-rated ECR-led projects may be recommended, in order to meet the program’s commitment to supporting ECRs.

Multidisciplinary/Multisectoral Review Panel meeting

The Multidisciplinary/Multisectoral Review Panel meets virtually to discuss applications and identify those to be recommended for funding. Applications to be discussed will be divided across virtual rooms and members will discuss applications across their assigned virtual rooms. Each room is presided over by three co-chairs, with each co-chair representing one of the three agencies: CIHR, NSERC and SSHRC. NFRF program staff are present to help the co-chairs and answer program- and policy-related questions.

Discussion of applications

Twenty minutes are allocated to discuss each application. The discussion is moderated by one of the co-chairs (as defined above) and proceeds according to the following steps:

  1. The NFRF program officer introduces the application, the three assigned panel members and the co-chair.
  2. The co-chair facilitates the discussion of the application, beginning with the interdisciplinarity and EDI criteria.
    1. The first reviewer provides their rating (pass/fail) and a brief rationale. The other assigned reviewers are then invited, in turn, to provide any additional information or differing viewpoints.
    2. While one reviewer presents their opinion during this process, other reviewers can join the discussion to add additional information and perspectives.
    3. If a reviewer revises their ratings during the discussion, they must update these ratings in the Convergence Portal after the discussion to maintain an up-to-date record of their evaluation.
  3. The high risk, high reward and feasibility criteria are then discussed.
    1. The first assigned review panel member briefly summarizes the proposal, and provides their ratings and a brief rationale. The other assigned reviewers are then invited, in turn, to provide their ratings and any additional information.
    2. Other reviewers can participate in the discussion and ask for clarifications, time permitting.
    3. If the assigned reviewers revise their ratings, they must update these ratings in the Convergence Portal following the discussion.

Feedback to applicants

Concise feedback is prepared for all applications that are discussed at the virtual panel meetings but not recommended for funding. One of the reviewers is identified in advance to lead preparation of the feedback for each application.

Handling documents used in peer review

Merit review documents contain personal information as well as information that the unauthorized disclosure of could reasonably be expected to cause serious injury (such as prejudicial treatment or loss of reputation or competitive advantage) to an individual, organization or government. Therefore, these documents are subject to regulation under the Privacy Act, the Access to Information Act and the Policy on Government Security. Protocols must be followed to ensure that information contained in applications, internal and external reviews, and panel discussions remains strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Refer to the Guide on Handling Documents Used in Peer Review for further details.

Legal and ethical information

Responsible conduct of research

Canada’s federal research funding agencies—CIHRNSERC and SSHRC—are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions and the agencies that, together, help support and promote a positive research environment.

Confidentiality

Members appointed to the review panel must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers describing expectations and requirements.

Privacy Act

Personal information refers to any information about an identifiable individual. Based on the Privacy Act, personal information provided by applicants must be used only for assessing applications, making funding decisions, and related uses describing applicants at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal. It is important for panel members to adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Canadian Human Rights Act

The activities of CIHR, NSERC and SSHRC are subject to the Canadian Human Rights Act. The purpose of the Act is to extend the laws in Canada to give effect to the principle that all individuals should have opportunity equal with other individuals to make for themselves the lives that they are able and wish to have. They should also have their needs accommodated, consistent with their duties and obligations as members of society, without being hindered in or prevented from doing so by discriminatory practices based on race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, genetic characteristics, disability or conviction for an offence for which a pardon has been granted or in respect of which a record suspension has been ordered.

Official Languages Act

All review panel members and NFRF program staff must be aware of their obligations and rights as legislated in the Official Languages Act.


Date modified: