2022 Exploration: Reviewer manual

On this page

• Purpose of the manual
  • A word of thanks
• Overview of the 2021 Exploration competition
• Principles of merit review
  • Fairness
  • Bias
  • Conflict of interest and confidentiality
• Roles and responsibilities
  • Strategic and program steering committees
  • Multidisciplinary review panel
  • New Frontiers in Research Fund staff
• Applications and review material
  • Incomplete or non-adherent applications
  • Eligibility of applicants
  • Indigenous research
  • Merit indicators
  • Areas of concern
  • Feedback for agency staff and other committee members
  • Types of risk
  • Budget
• How to access applications for review
  • Multidisciplinary review panel members
  • External reviewers
• Review process
  • Full application stage
• Handling documents used in peer review
• Legal and ethical information
  • Responsible conduct of research
  • Confidentiality
  • Privacy Act
  • Canadian Human Rights Act
  • Official Languages Act

Purpose of the manual

This manual is a guide for reviewers for the New Frontiers in Research Fund (NFRF) 2022 Exploration competition. It describes activities to be undertaken by external reviewers, by multidisciplinary review panel members and by panel co-chairs, and outlines the policies, guidelines and deliverables relevant to these activities. The content is intended to guide reviewers and outline principles rather than provide applicants with a set of rules.

A Word of Thanks

On behalf of the Canada Research Coordinating Committee (CRCC), we would like to thank you for agreeing to participate in the review process for the NFRF 2022 Exploration competition. The success of the review process is made possible by dedicated people like you who generously give your time and expertise. The CRCC and the research community greatly appreciate your efforts.

Review panel members and external reviewers are asked to read the relevant sections of this document in their entirety before beginning to review applications assigned to them. You may ask for clarification on any subject from NFRF program staff at any time.

Overview of the 2022 Exploration competition

The goal of the Exploration stream is to inspire high-risk, high-reward and interdisciplinary research.

Exploration grants support research that pushes boundaries into exciting new areas. Researchers are encouraged to think “outside of the box,” undertake research that would defy current paradigms, and bring disciplines together in unexpected ways and from bold, innovative perspectives. With the Exploration stream, there is recognition that innovation often carries risk: proposals for high-risk research projects that have the potential to deliver game-changing impacts are strongly encouraged. Consequently, it is expected that a number of funded projects will fail to meet their objectives.

Diversity of perspectives is important, and the fund encourages research proposals led from any discipline in the social sciences and humanities, health, and/or the natural sciences and engineering.

For further details on the 2022 competition, including eligibility requirements, see the competition overview. The instructions for the notice of intent to apply (NOI) and application provide more details on the information required at each stage of the competition.

Principles of merit review

Fairness

Success of the NFRF merit review system depends on the willingness and ability of all reviewers in the process to be fair and reasonable; to exercise rigorous judgment; and to understand—and consider in a balanced way—the particular context of each application.

Bias

All reviewers are asked to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought; fundamental versus applied research; certain subdisciplines; areas of research or approaches (including emerging ones); size or reputation of an institution; or the age, personal factors, sex or gender of the applicants. All reviewers are cautioned against judging an application based on these factors. Before review panel members are able to view any applications, they must first complete the required Unconscious Bias Training Module.

Conflict of interest and confidentiality

The Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations ensures the effective management of conflict of interest of any participant in the review process and, during the review process, ensures the confidentiality of personal and commercial information submitted to the program.

Conflict of interest

Members are responsible for evaluating the merits of applications assigned to them for review, with the exception of those for which they have a conflict of interest.

A conflict of interest is a conflict between a person’s duties and responsibilities as a participant in the review process, and that person’s private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the review panel member, external reviewer, referee or observer:

• would benefit professionally or personally as a result of the application being reviewed;
• has a professional or personal relationship with any of the applicants (this includes nominated principal investigators, co-principal investigators, co-applicants and collaborators) or applicants’ institutions; or
• has a direct or indirect financial interest in the application being reviewed.

A conflict of interest may be deemed or perceived to exist when a review panel member, external reviewer or observer:

• is a relative or close friend, or has a personal relationship with any of the applicants;
• is in a position to gain or lose financially/materially from the funding of the application;
• has had longstanding scientific or personal differences with any of the applicants;
• is currently affiliated with any of the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
• is professionally affiliated with any of the applicants as a result of having in the last six years:
  • had frequent and regular interactions with any of the applicants in the course of their duties at their department, institution, organization or company;
  • been a supervisor or a trainee of any of the applicants;
  • collaborated, published or shared funding with any of the applicants, or made plans to do so in the immediate future; or
  • been employed by any of the applicants’ institutions; and/or
• feels for any reason unable to provide an impartial review of the application.

All review panel members are subject to the same conflict of interest guidelines. If any of these situations apply, the member must declare a conflict of interest and cannot be present when an application is being discussed. NFRF staff and the co-chairs are responsible for resolving areas of uncertainty. A conflict of interest can be declared at any time during the competition cycle. Please contact NFRF program staff if you have any changes.

Confidentiality

The information included by applicants in their applications is protected by the Privacy Act and is provided for the purposes of review only. Details of the application, scoring, panel discussions and recommendation for a specific application are confidential and must never be divulged. Only NFRF staff can release information. Under no circumstances should members disclose to anyone the recommendations from the merit review meetings; this also applies to when the competition is over and the award recipients are announced.

Funding recommendations made by the review panel are subject to approval by the NFRF steering committee and may be changed for reasons of budget, administrative error or lack of full adherence to policies.

Review panel members are asked not to communicate any information relating to the review of a specific application or offer to applicants, or anyone outside of the panel, opinions on the applicants’ chances of success or failure.

In turn, applicants are not to contact panel members regarding the status of their applications (ratings, rank, etc.). By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of the funding decision for a competition. This includes written feedback provided by the external reviewers but does not include any notes or feedback a review panel member has provided that were not shared in written form with the other panel members or NFRF staff.

Roles and responsibilities

Strategic and program steering committees

The CRCC provides strategic direction and oversight for the NFRF and serves as the high-level steering committee. As a tri-agency program, program oversight is delegated to the NFRF steering committee, which includes the presidents of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC); and the deputy ministers of Innovation, Science and Economic Development Canada, and Health Canada. The NFRF steering committee makes decisions regarding which applications to fund, based on the review panel’s recommendations. It also ensures the evaluation process is rigorous, objective and transparent, in keeping with the standards of excellence expected by the agencies and consistent with the program’s objectives.

Multidisciplinary review panel

Membership

The multidisciplinary review panel is composed of experts who represent the various research disciplines of the three federal research funding agencies (CIHR, NSERC and SSHRC). To achieve a balance among panel members, factors such as: area of expertise; language; inclusion in the four designated groups (women, Indigenous Peoples, visible minorities and persons with disabilities); region; institution size; career stage; knowledge of best practices for equity, diversity and inclusion (EDI); and experience in research by and with Indigenous Peoples are taken into account, as is the inclusion of non-academics. Members are selected to ensure the panel has the capacity to review proposals in both of Canada’s official languages.

The multidisciplinary review panel makes recommendations to the NFRF steering committee about which applications should be funded.

Co-chairs

Co-chairs are responsible for ensuring that the multidisciplinary review panel deliberations function smoothly, effectively and objectively, according to the program’s policies. The co-chairs establish a positive, constructive and fair-minded environment in which the applications are evaluated. They fulfil an oversight role and participate in the review of applications.

The co-chairs’ responsibilities include:

• ensuring that they follow the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the meeting;
• ensuring the integrity and quality of the merit review process, and that each application receives a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
• completing the online training module on unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions before the merit review meeting;
• including all members in the review of applications;
• ensuring that all documents are considered in the panel’s assessments;
• guiding the panel to a consensus recommendation, where applicable;
• assisting with the preparation of feedback to applicants, where applicable;
• contributing to discussions on NFRF policy issues;
• participating in discussions on Review Panel membership for future competitions; and
• ensuring that all confidential review materials provided to them are handled safely and disposed of according to program policy.

Multidisciplinary review panel members

Review panel members evaluate applications and make recommendations to the NFRF steering committee based on their assessments. Specific responsibilities of members include:

• ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the evaluations and/or meeting;
• ensuring the integrity and quality of the merit review process, and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
• completing the online training module on unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions before evaluations and/or the merit review, where applicable;
• reviewing in depth the applications assigned to them;
• ensuring that all documents are considered in their assessments;
• providing ratings for applications assigned to them by the deadline or before the merit review meeting, where applicable;
• participating in the review of applications assigned to them by providing a verbal assessment to the panel during the meeting, where applicable;
• participating in the discussion and review of all relevant applications, according to their role, where applicable;
• preparing feedback to applicants, where applicable; and
• ensuring that all review materials provided are handled safely and disposed of according to program policy.

New Frontiers in Research Fund staff

NFRF staff are typically represented at the merit review meetings by the director, deputy director and/or manager, and the program officers. The staff’s responsibilities include:

• ensuring that the panel follows the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the review meeting;
• ensuring the integrity and quality of the merit review process and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• assigning applications to review panel members;
• providing advice and guidance to the panel on the program’s policies;
• ensuring that all documents are considered in the panel’s assessments;
• keeping notes on procedural aspects of the panel’s functions;
• recording comments made by the panel for each application, where applicable;
• working with the co-chairs to manage conflicts of interest, where applicable;
• recording concerns raised by the panel on issues requiring subsequent attention by staff;
• ensuring that all review materials provided are handled safely and disposed of according to program policy; and
• facilitating the final approval of recommended applications by the NFRF steering committee.

Applications and review material

Incomplete or non-adherent applications

The onus is on the applicants to provide complete and sufficient information that adheres to the Convergence Portal instructions for attachments and the instructions for completing the NOI and full application. Problems related to application content should be brought to the attention of NFRF program staff, which can be done at any point during the competition. To maintain fairness in the competition, applicants must adhere to the guidelines in preparing application materials. If staff determines that information provided is incomplete or non-adherent to guidelines or instructions, the application may be rejected.

Eligibility of applicants

Eligibility decisions are the responsibility of the NFRF program staff. Review panel members who have doubts as to a researcher’s eligibility should review the application on the same basis as all others and alert the NFRF staff to potential eligibility problems as soon as possible. The eligibility criteria for applicants can be found in the eligibility section of the competition overview.

Indigenous research

If a review panel member or external reviewer receives a proposal in which the applicants have answered “Yes” to the question “Does your proposal involve Indigenous research as defined by SSHRC?” they must use SSHRC’s Indigenous Research Statement of Principles and Guidelines for the Merit Review of Indigenous Research as references in assessing the application related to or involving Indigenous research. The guidelines are provided to help reviewers build understanding of Indigenous research and research-related activities, and to assist them in interpreting the specific evaluation criteria in the context of Indigenous research.

Merit indicators

The merit indicators include pass/fail indicators for the interdisciplinarity/fit to program and EDI-RP criteria, and an evaluation scale for the high risk, high reward and feasibility indicators. The evaluation scales include references to major points of consideration to guide review panel members toward arriving at a rating for a given criterion. Although only four ratings are provided for each evaluation scale, members may select a rating between each rating described, for a seven-point scale.

All applications are evaluated using the same merit indicators. Review panel members are encouraged to use the full range of ratings, as appropriate, to achieve a distribution of ratings that reflects the quality of the applications being evaluated. Members are expected to discuss and justify (or adjust) their ratings during the multidisciplinary review panel meetings.

Areas of concern

Members of the review panel can comment about any areas of concern in the application, including ones related to any of the selection criteria, including EDI-RP and interdisciplinarity; EDI-RD; Indigenous research; budget; or others.

Reviewers must not include any identifying information about the team members (names, research groups, departments, institutions, etc.) when entering their comments.

Feedback for agency staff and other committee members

Members of the review panel can also add any additional comments they believe are relevant. These comments may be helpful during the virtual meeting and/or in preparing a message to the applicant, should one become necessary.

Reviewers must not include any identifying information about the team members (names, research groups, departments, institutions, etc.) when entering their comments.

Types of risk

All reviewers are asked to identify, for analytical purposes only, the types of risk in the proposed research projects, from the following list:

• challenging current research perspectives or paradigms;
• using equipment, techniques or approaches proven or assumed to be extraordinarily difficult;
• bringing together an unprecedented combination of disciplines with different perspectives;
• viewing the project from an unfamiliar interdisciplinary perspective, to use novel approaches to solve existing problems;
• other risks (indicate each one in a text box); or
• no risks.

Budget

The proposed budget is not one of the selection criteria. It is not typically discussed by reviewers, except where it may affect the assessment of feasibility in accomplishing the research with the proposed resources. However, if a reviewer notices an expense that is not eligible under the Tri-Agency Guide on Financial Administration, or one they perceive has been greatly overestimated, they should bring it to the attention of NFRF program staff.

How to access applications for review

All review panel members and external reviewers will access the applications for review through the Convergence Portal, and will only have access to applications they have been assigned.

Multidisciplinary review panel members

Potential review panel members will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profile. Shortly after the application deadline, multidisciplinary review panel members will receive an email asking them to accept the terms and conditions (T&Cs) in the Convergence Portal. Members must accept the T&Cs before they can continue with the review process. After acceptance, two new tabs will appear on the home screen when the member logs into the Convergence Portal: “Conflicts of Interest” and “Committee Assignments.” If the T&Cs are accepted before applications are assigned, both tabs will remain blank until applications are assigned.

Following this step, members will receive an email informing them that the applications assigned to them are ready and can be viewed under the “Conflicts of Interest” tab. Any problem with the assignment of applications should be brought to the attention of NFRF program staff as soon as possible, at any point in the process.

External reviewers

Potential external reviewers will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profiles.

Reviewers may also complete their profiles when they receive the invitation to review. Invitations sent from the Convergence Portal include a link to “Accept” or “Decline” the invitation. A reviewer must click “Accept” to be directed to the T&Cs in the Convergence Portal.

Once the reviewer has accepted the T&Cs, applications for review will appear on the “Reviewer Assignments” tab in the Convergence Portal.

Any problem with the assignment of applications should be brought to the attention of NFRF program staff as soon as possible, at any point in the process.

Review Process

Full application stage

The purpose of the application review process is to identify the most meritorious applications to be recommended for funding by the multidisciplinary review panel.

Review timeline

Evaluation by external reviewers

External reviews focus on the proposal itself. External reviewers are asked to comment on the High Risk and High Reward criteria, as well as the Feasibility criterion as it relates to the research plan only. The external review process is double-anonymous: the applicants do not know the identity of the external reviewers, and the external reviewers are not provided with information identifying the research team. Any application that does not follow the instructions on the preparation of materials for anonymous review (i.e., that includes identifying information in the application material available) should be brought to the attention of NFRF program staff as soon as possible, at any point in the process. External reviewers will have access to the summary from the NOI, plus the EDI-RD section, research proposal, budget justification and literature references from the full application. External reviewers will not have access to an application’s EDI section or biographical information attachments.

Assignment

Each application will be assigned to three external reviewers with expertise in the fields of the application. External reviewers will typically review only one or two applications but may be invited to review up to five. Since the projects are interdisciplinary, external reviewers are asked to comment only on the aspects of the proposal they are comfortable assessing and to note those they are not.

External review process

External reviewers comment on the High Risk, High Reward and Feasibility components of the proposed project, as follows:

• Is the project high risk? (yes/no)
  • Description of the strengths and weaknesses
• Is the project high reward? (yes/no)
  • Description of the strengths and weaknesses
• Is the project feasible? (yes/no)
  • Description of the strengths and weaknesses
• Types of risk
• Any considerations or comments related to the proposal 

Evaluation by multidisciplinary review panel

Assignment

Each application is assigned to five multidisciplinary review panel members for review. Two to three members are assigned based on their expertise matching the primary areas of research of the proposal. The remaining members do not have disciplinary expertise but provide assessments from an interdisciplinary/non-expert perspective. The balance of perspectives allows the application to be assessed by experts in the applicable disciplines, as well as by experts who can speak to the impact of the project beyond these disciplines. Language abilities and institutional conflicts of interest will be considered in assigning applications to members.

Check for conflicts of interest

Shortly after the deadline, review panel members will receive an email informing them that their list of assignments is ready. Members are asked to log in to the Convergence Portal, review the summary of each of their assigned applications in the “Conflicts of Interest” tab, and indicate if there is a conflict of interest. Once a member has checked all assigned applications for conflicts, they immediately gain access to the applications for which they are not in conflict in the “Committee Assignments” tab. Some changes to assignments may occur, as conflicts of interest are identified by members. If there are any changes, the panel member will be notified by NFRF program staff. Any problems should be brought to the attention of NFRF program staff as soon as possible, at any point in the process.

Assessment

Multidisciplinary Review Panel members receive access to all content from the NOI and full application stages, as well as to the external reviews.

All eligible applications are to be reviewed and scored according to the merit indicator matrices for the selection criteria:

• high risk: 40% of overall score
• high reward: 40% of overall score
• feasibility: 20% of overall score
• interdisciplinarity/fit to program (pass/fail)
• EDI-RP (pass/fail)

Members can assign one of seven ratings for the high risk, high reward and feasibility criteria (see Merit indicators). In assessing feasibility, members must consider the approach to Indigenous research (if applicable) and EDI-RD considerations in the research design.

Members are asked to comment on:

• types of risk;
• interdisciplinarity / fit to program (if the score is a fail); and
• EDI-RP (if the score is a fail).

Members can also comment on any areas of concern related to the application, and add feedback they wish to convey to agency staff and other members of the committee.

Result

Based on the multidisciplinary review panel ratings, applications will be placed in “bands” of equally rated applications.

Top-rated applications will be discussed by the multidisciplinary review panel. The cut-off band for those to be discussed will be determined based on the distribution of scores and the number of applications received. Members may identify any application to be discussed at the meeting, regardless of its overall score.

Remaining applications will not be discussed and will not be recommended for funding.

The CRCC has committed to ensuring that a proportion of the awards will be reserved for applications led by early career researchers (ECR), equal to the proportion of ECR-led applications received. The program monitors the proportion of ECR-led applications recommended for funding. It is possible that lower-rated ECR-led projects may be recommended, in order to meet the program’s commitment to supporting ECRs.

Multidisciplinary review panel meeting

The multidisciplinary review panel meets virtually to discuss applications and identify those to be recommended for funding. Applications to be discussed will be divided across virtual rooms and members will discuss applications across their assigned virtual rooms. Each room is presided over by three co-chairs, with each co-chair representing one of the three agencies: CIHR, NSERC and SSHRC. NFRF program staff are present to help the co-chairs and answer program- and policy-related questions.

Discussion of applications

Twenty minutes are allocated to discuss each application. The discussion is moderated by one of the co-chairs (as defined above) and proceeds according to the following steps:

1. The NFRF program officer introduces the application, the five assigned panel members and the co-chair.
2. The co-chair facilitates the discussion of the application, beginning with the interdisciplinarity and EDI-RP criteria.
   1. The first reviewer provides their rating (pass/fail) and a brief rationale. The other assigned reviewers are then invited, in turn, to provide any additional information or differing viewpoints.
   2. While one reviewer presents their opinion during this process, other reviewers can join the discussion to add additional information and perspectives.
   3. If a reviewer revises their ratings during the discussion, they must update these ratings in the Convergence Portal after the discussion to maintain an up-to-date record of their evaluation.
3. The high risk, high reward and feasibility criteria are then discussed.
   1. The first assigned review panel member briefly summarizes the proposal, and provides their ratings and a brief rationale. The other assigned reviewers are then invited, in turn, to provide their ratings and any additional information.
   2. Other reviewers can participate in the discussion and ask for clarifications, time permitting.
   3. If the assigned reviewers revise their ratings, they must update these ratings in the Convergence Portal following the discussion.

Message to applicants

Concise feedback is prepared for all applications that are discussed at the virtual panel meetings but not recommended for funding. One of the reviewers is identified in advance to lead preparation of a message to the applicant containing the feedback for each application.

Handling documents used in peer review

Merit review documents contain personal information as well as information that the unauthorized disclosure of could reasonably be expected to cause serious injury (such as prejudicial treatment or loss of reputation or competitive advantage) to an individual, organization or government. Therefore, these documents are subject to regulation under the Privacy Act, the Access to Information Act and the Policy on Government Security. Protocols must be followed to ensure that information contained in applications, internal and external reviews, and panel discussions remains strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Refer to the Guide on Handling Documents Used in Peer Review for further details.

Legal and ethical information

Responsible conduct of research

Canada’s federal research funding agencies—CIHR, NSERC and SSHRC—are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions and the agencies that, together, help support and promote a positive research environment.

Confidentiality

Members appointed to the panel must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers describing expectations and requirements.

Privacy Act

Personal information refers to any information about an identifiable individual. Based on the Privacy Act, personal information provided by applicants must be used only for assessing applications, making funding decisions, and related uses describing applicants at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal. It is important for panel members to adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Canadian Human Rights Act

The activities of CIHR, NSERC and SSHRC are subject to the Canadian Human Rights Act. The purpose of the Act is to extend the laws in Canada to give effect to the principle that all individuals should have opportunity equal with other individuals to make for themselves the lives that they are able and wish to have. They should also have their needs accommodated, consistent with their duties and obligations as members of society, without being hindered in or prevented from doing so by discriminatory practices based on race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, genetic characteristics, disability or conviction for an offence for which a pardon has been granted or in respect of which a record suspension has been ordered.

Official Languages Act

All review panel members and NFRF program staff must be aware of their obligations and rights as legislated in the Official Languages Act.