2020 Exploration: Reviewer Manual

Purpose of the Manual

This manual is a guide for reviewers for the New Frontiers in Research Fund (NFRF) 2020 Exploration competition. It describes activities to be undertaken by external reviewers and by members and co-chairs of the Multidisciplinary Review Panel, and outlines the policies, guidelines and deliverables relevant to these activities. The content is intended to guide reviewers and outline principles, rather than provide applicants with a set of rules.

A word of thanks

On behalf of the Canada Research Coordinating Committee (CRCC), we would like to thank you for agreeing to participate in the review process for the NFRF 2020 Exploration competition. The success of the review process is made possible by dedicated people like you who generously give your time and expertise. Your efforts are greatly appreciated by the CRCC and the research community.

Review panel members and external reviewers are asked to read the relevant sections of this document in their entirety before beginning to review applications assigned to them. You may ask for clarification on any subject from NFRF program staff at any time.

Overview of the 2020 Exploration Competition

The goal of the Exploration stream is to inspire high-risk, high-reward and interdisciplinary research.

Exploration grants support research that pushes boundaries into exciting new areas. Researchers are encouraged to think “outside of the box,” undertake research that would defy current paradigms, and bring disciplines together in unexpected ways and from bold, innovative perspectives. With the Exploration stream, there is recognition that innovation often carries risk; proposals for high-risk research projects that have the potential to deliver game-changing impacts are strongly encouraged. Consequently, it is expected that a number of funded projects will fail to meet their objectives.

Diversity of perspectives is important, and the fund encourages research proposals led from any discipline in the social sciences and humanities, health, the natural sciences and engineering.

The 2020 Exploration competition merit review processes differ from those of previous Exploration competitions, and include elements distinct from those used in flagship tri-agency programs.

For further details on the 2020 competition, including eligibility requirements, see the competition overview. The instructions for the notice of intent to apply (NOI) and application provide more details on the information required at each stage of the competition.

Principles of Merit Review

Fairness

Success of the NFRF merit review system depends on the willingness and ability of all reviewers in the process to be fair and reasonable; to exercise rigorous judgment; and to understand—and take into account in a balanced way—the particular context of each application.

Bias

All reviewers are asked to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought; fundamental versus applied research; certain subdisciplines; areas of research or approaches (including emerging ones); size or reputation of an institution; or the age, personal factors, sex or gender of the applicants. All reviewers are cautioned against judging an application based on these factors. Before review panel members are able to view any applications, they must first complete the required Unconscious Bias Training Module.

Conflict of interest and confidentiality

The Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations ensures the effective management of conflict of interest of any participant in the review process and, during the review process, ensures the confidentiality of personal and commercial information submitted to the program.

Conflict of interest

Members are responsible for evaluating the merits of applications assigned to them for review, with the exception of those for which they have a conflict of interest.

A conflict of interest is a conflict between a person’s duties and responsibilities as a participant in the review process, and that person’s private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the review panel member, external reviewer, referee or observer:

• would benefit professionally or personally as a result of the application being reviewed;
• has a professional or personal relationship with any of the applicants (this includes nominated principal investigators, co-principal investigators, co-applicants and collaborators) or applicants’ institutions; or
• has a direct or indirect financial interest in the application being reviewed.

A conflict of interest may be deemed or perceived to exist when a review panel member, external reviewer or observer:

• is a relative or close friend, or has a personal relationship with any of the applicants;
• is in a position to gain or lose financially/materially from the funding of the application;
• has had longstanding scientific or personal differences with any of the applicants;
• is currently affiliated with any of the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
• is professionally affiliated with any of the applicants as a result of having in the last six years:
  • had frequent and regular interactions with any of the applicants in the course of their duties at their department, institution, organization or company;
  • been a supervisor or a trainee of any of the applicants;
  • collaborated, published or shared funding with any of the applicants, or made plans to do so in the immediate future; or
  • been employed by any of the applicants’ institutions; and/or
• feels for any reason unable to provide an impartial review of the application.

All review panel members are subject to the same conflict of interest guidelines. If any of these situations apply, the member must declare a conflict of interest and leave the room when the application is being discussed. NFRF staff and the co-chairs are responsible for resolving areas of uncertainty. A conflict of interest can be declared at any time during the competition cycle. Please contact NFRF program staff if you have any changes.

Confidentiality

The information included by applicants in their applications is protected by the Privacy Act and is provided for the purposes of review only. Details of the application, scoring, panel discussions and recommendation for a specific application are confidential and must never be divulged. Only NFRF staff can release information. Under no circumstances should members disclose to anyone the recommendations from the merit review meetings; this also applies to when the competition is over and the award recipients are announced.

Funding recommendations made by the review panel are subject to approval by the NFRF steering committee and may be changed for reasons of budget, administrative error or lack of full adherence to policies.

Review panel members are asked not to communicate any information relating to the review of a specific application, or offer to applicants, or anyone outside of the panel, opinions on the applicants’ chances of success or failure.

In turn, applicants are not to contact panel members regarding the status of their applications (ratings, rank, etc.). By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of the funding decision for a competition. This includes written feedback provided by the external reviewers, but does not include any notes or feedback a review panel member has provided that were not shared in written form with the other panel members or NFRF staff.

Roles and Responsibilities

Strategic and program steering committees

The CRCC provides strategic direction and oversight for the NFRF and serves as the high-level steering committee. As a tri-agency program, program oversight is delegated to the NFRF steering committee, which includes the presidents of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC); and the deputy ministers of Innovation, Science and Economic Development Canada, and Health Canada. The NFRF steering committee makes decisions regarding which applications to fund, based on the review panel’s recommendations. It also ensures the evaluation process is rigorous, objective and transparent, in keeping with the standards of excellence expected by the agencies and consistent with the program’s objectives.

Multidisciplinary review panel

Membership

The multidisciplinary review panel is composed of experts who represent the various research disciplines of the three federal research funding agencies (CIHR, NSERC and SSHRC). To achieve a balance among panel members, factors such as area of expertise; language; inclusion in the four designated groups (women, Indigenous peoples, visible minorities and persons with disabilities); region; institution size; career stage; knowledge of best practices for equity, diversity and inclusion (EDI); and experience in research by and with Indigenous peoples are taken into account, as is the inclusion of nonacademics. Members are selected to ensure the panel has the capacity to review proposals in both of Canada’s official languages.

The multidisciplinary review panel makes recommendations to the NFRF steering committee about which applications should be funded.

Co-chairs

Co-chairs are responsible for ensuring that the multidisciplinary review panels function smoothly, effectively and objectively, according to the program’s policies. The co-chairs establish a positive, constructive and fair-minded environment in which the applications are evaluated. They fulfill an oversight role, and also participate in the review of applications.

The co-chairs’ responsibilities include:

• ensuring that the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations is followed prior to, during and after the meeting;
• ensuring the integrity and quality of the merit review process, and that each application receives a fair assessment (free of bias, and equitable to all applicants) based on the evaluation of all criteria;
• ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
• completing the online training module on unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions prior to the merit review meeting;
• including all members in the review of applications;
• ensuring that all documents are considered in the panel’s assessments;
• guiding the panel to a consensus recommendation, where applicable;
• assisting with the preparation of feedback to applicants, where applicable;
• contributing to discussions on NFRF policy issues;
• participating in discussions on review panel membership for future competitions; and
• ensuring that all confidential review materials provided to them are handled safely and disposed of according to program policy.

Multidisciplinary review panel members

Review panel members evaluate applications and make recommendations to the NFRF steering committee based on their assessments. Specific responsibilities of members include:

• ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations prior to, during and after the evaluations and/or meeting;
• ensuring the integrity and quality of the merit review process, and that each application gets a fair assessment (free of bias, and equitable to all applicants) based on the evaluation of all criteria;
• ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
• completing the online training module on unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions prior to evaluations and/or the merit review, where applicable;
• reviewing in depth the applications assigned to them;
• ensuring that all documents are considered in their assessments;
• providing ratings for applications assigned to them by the deadline or prior to the merit review meeting, where applicable;
• participating in the review of applications assigned to them by providing a verbal assessment to the entire committee during the meeting, where applicable;
• participating in the discussion and review of all relevant applications, according to their role, where applicable;
• preparing feedback to applicants, where applicable; and
• ensuring that all review materials provided are handled safely and disposed of according to program policy.

NFRF staff

NFRF staff are typically represented at the merit review meetings by the director, deputy director and/or manager, and the program officers. The staff’s responsibilities include:

• ensuring that the panel follows the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations prior to, during and after the review meeting;
• ensuring the integrity and quality of the merit review process, and that each application gets a fair assessment (free of bias, and equitable to all applicants) based on the evaluation of all criteria;
• assigning applications to review panel members;
• providing advice and guidance to the panel on the program’s policies;
• ensuring that all documents are considered in the panel’s assessments;
• keeping notes on procedural aspects of the panel’s functions;
• recording comments made by the panel for each application, where applicable;
• working with the co-chairs to manage conflicts of interest, where applicable;
• recording concerns raised by the panel on issues requiring subsequent attention by staff;
• ensuring that all review materials provided are handled safely and disposed of according to program policy; and
• facilitating the final approval of recommended applications by the NFRF steering committee.

Applications and Review Material

Incomplete or nonadherent applications

The onus is on the applicants to provide complete and sufficient information that adheres to the Convergence Portal instructions for attachments and the instructions for completing the NOI and full application. Problems related to application content should be brought to the attention of NFRF program staff, which can be done at any point during the competition. To maintain fairness in the competition, applicants must adhere to the guidelines in preparing application materials. If staff determines that information provided is incomplete or nonadherent to guidelines or instructions, the application may be rejected.

Eligibility of applicants

Eligibility decisions are the responsibility of the NFRF program staff. Review panel members who have doubts as to a researcher’s eligibility should review the application on the same basis as all others, and alert the NFRF staff to potential eligibility problems as soon as possible. The eligibility criteria for applicants can be found in the eligibility section of the competition overview.

Indigenous research

If a review panel member or external reviewer receives a proposal in which the applicants have answered “Yes” to the question “Does your proposal involve Indigenous research as defined by SSHRC?” they must use SSHRC’s Indigenous Research Statement of Principles and Guidelines for the Merit Review of Indigenous Research as references in assessing the application related to or involving Indigenous research. The guidelines are provided to help reviewers build understanding of Indigenous research and research-related activities, and to assist them in interpreting the specific evaluation criteria in the context of Indigenous research.

Merit indicators

The merit indicators include pass/fail indicators for the interdisciplinarity / fit to program and EDI criteria, and an evaluation scale for the high risk, high reward and feasibility indicators. The evaluation scales include references to major points of consideration to guide review panel members towards arriving at a rating for a given criterion. Although only four ratings are provided for each evaluation scale, members may select a rating between each rating described, for a seven-point scale.

All applications are evaluated using the same merit indicators. Review panel members are encouraged to use the full range of ratings, as appropriate, to achieve a distribution of ratings that reflects the quality of the applications being evaluated. Members are expected to discuss and justify (or adjust) their ratings during the full application stage.

Types of risk

All reviewers are asked to collect information, for analytical purposes, identifying the types of risk present in the proposed research projects, according to the following list:

• challenging current research perspectives or paradigms;
• using equipment, techniques or approaches proven or assumed to be extraordinarily difficult;
• bringing together an unprecedented combination of disciplines with different perspectives;
• viewing the project from an unfamiliar interdisciplinary perspective, to use novel approaches to solve existing problems;
• other risks (indicate each one in a text box); and
• no risks.

Areas of concern

All reviewers can comment about any areas of concern in the application, including ones related to any of the selection criteria, including EDI (review panel members only) and interdisciplinarity; gender-based analysis plus (GBA+) / sex- and gender-based analysis (SGBA); Indigenous research; budget or others.

Applicants must not include any identifying information about team members (names, research groups, departments, institutions, etc.) in this section.

Budget

The proposed budget is not one of the selection criteria. It is not typically discussed by reviewers, except where it may affect the assessment of feasibility in accomplishing the research with the proposed resources. However, if a reviewer notices an expense that is not eligible under the Tri-Agency Guide on Financial Administration, or one they perceive has been greatly overestimated, they should bring it to the attention of NFRF program staff.

How to Access Applications for Review

All review panel members and external reviewers will access the applications for review through the Convergence Portal, and will only have access to applications they have been assigned.

Multidisciplinary review panel members

Review panel members will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profile. Following this step, panel members will receive an email asking them to accept the terms and conditions (T&Cs) of the Convergence Portal. Members must accept the T&Cs before they can continue with the review process. After acceptance, two new tabs will appear on the home screen when the member logs into the Convergence Portal: Committee Conflicts and Committee Assignments. If the T&Cs are accepted prior to applications being assigned, both tabs will remain blank until applications are assigned.

Shortly after the application deadline, members will receive an email informing them that the applications assigned to them are ready under the Reviewer Assignments tab. Any problem with the assignment of applications should be brought to the NFRF program staff’s attention as soon as possible, or at any point in the process.

External reviewers

External reviewers will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profiles. Following this, they will receive an email asking them to accept the T&Cs of the Convergence Portal. External reviewers must accept the T&Cs before they can continue with the review process.

Once the reviewer has accepted these, a new tab will appear on their home screen when they are logged into the Portal: Reviewer Assignments. If the T&Cs are accepted prior to applications being assigned, this tab will remain blank until applications are assigned.

Shortly after the application deadline, external reviewers will receive an email informing them that the applications assigned to them are ready under the Reviewer Assignments tab. Any problem with the assignment of applications should be brought to the NFRF program staff‘s attention as soon as possible, or at any point in the process.

Review Process

Full application stage

The purpose of the application review process is to identify the most meritorious applications to be recommended for funding by the multidisciplinary review panel.

Review timeline

Evaluation by external reviewers

External reviews focus on the proposal itself. External reviewers are asked to comment on the high risk, high reward and interdisciplinarity criteria, as well as the feasibility criterion as it relates to the research plan only. The external review process is double-blind, with the applicants not knowing the identity of the external reviewers, and the external reviewers not provided information identifying the research team. External reviewers will have access to the summary from the NOI, plus the GBA+/SGBA section, research proposal, budget justification and literature references from the full application. External reviewers will not have access to an application’s EDI section or biographical information attachments.

Assignment

Each application will be assigned to three external reviewers with expertise in the fields of the application. External reviewers will typically review only one or two applications, but may be invited to review up to five. Since the projects are interdisciplinary, external reviewers are asked to comment on the aspects of the proposal they are comfortable assessing, and to note those they are not.

External review process

External reviewers comment on the high risk, high reward, interdisciplinarity and feasibility components of the proposed project, as follows:

• Is the project high risk? (yes/no)
  • Description of the strengths and weaknesses
• Is the project high reward? (yes/no)
  • Description of the strengths and weaknesses
• Is the project feasible? (yes/no)
  • Description of the strengths and weaknesses
• Is the project interdisciplinary? (pass/fail)
  • Description of the strengths and weaknesses
• Types of risk
• Any areas of concern or comments related to the proposal 

Evaluation by multidisciplinary review panel

Assignment

Each application is assigned to five multidisciplinary review panel members for review. Two to three members are assigned based on their expertise matching the primary areas of research of the proposal. The remaining members do not have disciplinary expertise, and provide assessments from an interdisciplinary/nonexpert perspective. Language abilities and institutional conflicts of interest will be taken into account in assigning applications to members.

Check for conflicts of interest

Shortly after the deadline, review panel members will receive an email informing them that their list of assignments is ready. Members are asked to log in to the Convergence Portal, review the summary of each of their assigned applications, and indicate if there is a conflict of interest. Once a member has checked all assigned applications for conflicts, they immediately gain access to the applications for which they are not in conflict. Some changes to assignments may occur, as any conflicts of interest are identified by members. Any problems should be brought to the NFRF program staff’s attention as soon as possible, or at any point in the process.

Assessment

Multidisciplinary review panel members receive access to all content from the NOI and full application stages, as well as to the external reviews.

All eligible applications are to be reviewed and scored according to the merit indicator matrices for the selection criteria:

• high risk: 40% of overall score
• high reward: 40% of overall score
• feasibility: 20% of overall score
• interdisciplinarity / fit to program (pass/fail)
• EDI (pass/fail)

Members can assign one of seven ratings for the high risk, high reward and feasibility criteria (see Merit indicators). In assessing feasibility, members must consider the approach to Indigenous research (if applicable) and GBA+/SGBA considerations in the research design.

Members are asked to provide comments on:

• types of risk
• interdisciplinarity / fit to program (if the score is a fail)
• EDI (if the score is a fail)

Members can also comment on any areas of concern related to the application.

All review panel members are asked to identify their top-rated application from among those they reviewed.

Result

Based on the multidisciplinary review panel ratings, applications will be placed in “bands” of equally rated applications.

Top-rated applications with an overall pass for interdisciplinarity / fit to program and EDI (meaning at least three of the five panel members rated it as “pass” for these criteria) will be recommended for funding.

If the number of equally rated applications (those in the same band) exceeds the available competition budget, all applications in that band are discussed by the review panel members through virtual meetings.

Remaining applications will not be discussed, and will not be recommended for funding.

The CRCC has committed to ensuring that a proportion of the awards will be reserved for applications led by early career researchers (ECR), equal to the proportion of ECR-led applications received. The program monitors the proportion of ECR-led applications recommended for funding. It is possible that lower-rated ECR-led projects may be recommended, in order to meet the program’s commitment to supporting ECRs.

Multidisciplinary review panel meeting

The multidisciplinary review panel meets virtually to discuss applications and identify those to be recommended for funding. Review panel members are divided into groups, depending on the number of applications to be discussed. Each group is presided over by three co-chairs, with each co-chair representing one of the three agencies: CIHR, NSERC and SSHRC. NFRF program staff are present to help the co-chairs and answer program- and policy-related questions.

Discussion of applications

Twenty minutes are allocated to discuss each application. The discussion is moderated by one of the co-chairs (as defined above) and proceeds according to the following steps:

1. The co-chair introduces the application and asks that any assigned review panel members with a conflict of interest leave the virtual meeting.
2. The co-chair introduces the five assigned reviewers for the application.
3. Discussion of the application begins with the interdisciplinarity and EDI criteria.
   1. The first reviewer provides their rating (pass/fail) and a brief rationale. The other assigned reviewers are then invited, in turn, to provide any additional information or differing viewpoints.
   2. During this process, while one reviewer presents their opinion, other reviewers can join the discussion to add additional information and perspectives.
   3. The co-chair assesses whether consensus has been reached or if voting is needed. If voting is needed, the five assigned reviewers vote, and the majority result stands.
   4. If a reviewer revises their ratings during the discussion, they must update these ratings in the Convergence Portal after the discussion to maintain an up-to-date record of their evaluation.
4. The high risk, high reward and feasibility criteria are then discussed.
   1. The first assigned review panel member briefly summarizes the proposal, and provides their ratings and a brief rationale. The other assigned reviewers are then invited, in turn, to provide their ratings and any additional information.
   2. Other reviewers can participate in the discussion and ask for clarifications, time permitting.
   3. If the assigned reviewers revise their ratings, they must update these ratings in the Portal following the discussion.

Feedback to applicants

Concise feedback is prepared for all applications that are not recommended for funding. One of the reviewers is identified in advance to lead preparation of the feedback for each application.

Handling Documents Used in Peer Review

Merit review documents contain personal information as well as information the unauthorized disclosure of which could reasonably be expected to cause serious injury (such as prejudicial treatment or loss of reputation or competitive advantage) to an individual, organization or government. Therefore, these documents are subject to regulation under the Privacy Act, the Access to Information Act and the Policy on Government Security. Protocols must be followed to ensure that information contained in applications, internal and external reviews, and panel discussions remains strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Please refer to the Guide on Handling Documents Used in Peer Review for further details.

Legal and Ethical Information

Responsible conduct of research

Canada’s federal research funding agencies—CIHR, NSERC and SSHRC—are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions and the agencies that, together, help support and promote a positive research environment.

Confidentiality

Members appointed to the panel must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers describing expectations and requirements.

Privacy Act

Personal information refers to any information about an identifiable individual. Based on the Privacy Act, personal information provided by applicants must be used only for assessing applications, making funding decisions, and related uses describing applicants at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal. It is important for panel members to adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Canadian Human Rights Act

The activities of CIHR, NSERC and SSHRC are subject to the Canadian Human Rights Act. The purpose of the Act is to extend the laws in Canada to give effect to the principle that all individuals should have opportunity equal with other individuals to make for themselves the lives that they are able and wish to have. They should also have their needs accommodated, consistent with their duties and obligations as members of society, without being hindered in or prevented from doing so by discriminatory practices based on race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, genetic characteristics, disability or conviction for an offence for which a pardon has been granted or in respect of which a record suspension has been ordered.

Official Languages Act

All review panel members and NFRF program staff must be aware of their obligations and rights as legislated in the Official Languages Act.