2021 Innovative Approaches to Research in the Pandemic Context: Reviewer Manual


On this page

Purpose of the Manual

This manual is a guide for reviewers for the New Frontiers in Research Fund (NFRF) 2021 Innovative Approaches to Research in the Pandemic Context competition. It describes activities to be undertaken by multidisciplinary review panel members, and outlines the policies, guidelines and deliverables relevant to these activities. The content is intended to guide reviewers and outline principles, rather than provide applicants with a set of rules.

A Word of Thanks

On behalf of the Canada Research Coordinating Committee (CRCC), we would like to thank you for agreeing to participate in the review process for the NFRF 2021 Innovative Approaches to Research in the Pandemic Context competition. The success of the review process is made possible by dedicated people like you who generously give your time and expertise. Your efforts are greatly appreciated by the CRCC and the research community.

Review panel members are asked to read this document in its entirety before beginning to review applications assigned to them. You may ask for clarification on any subject from NFRF program staff at any time.

Overview of the 2021 Innovative Approaches to Research in the Pandemic Context Competition

The goal of the 2021 Innovative Approaches to Research in the Pandemic Context competition is to accelerate the exploration of new approaches and the development and testing of bold new directions in research methodologies. It is expected that this competition will predominately support new ways of conducting community and field-based research, since the pandemic has had the greatest impact on researchers’ ability to collect data and conduct this type of research.

These rapid response grants will support projects that trial and test a novel and innovative research approach with potential to benefit Canada and the world. Proposals must be time-sensitive and demonstrate a new way of doing research where conventional or established research methods cannot be used due to COVID-19-related restrictions.

For further details on the competition, including eligibility requirements, see the competition overview. The instructions for the notice of intent to apply (NOI) and application provide more details on the information required at each stage of the competition.

Principles of Merit Review

Fairness

Success of the NFRF merit review system depends on the willingness and ability of all reviewers in the process to be fair and reasonable; to exercise rigorous judgment; and to understand—and take into account in a balanced way—the particular context of each application.

Bias

All reviewers are asked to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought; fundamental versus applied research; certain subdisciplines; areas of research or approaches (including emerging ones); size or reputation of an institution; or the age, personal factors, sex or gender of the applicants. All reviewers are cautioned against judging an application based on these factors. Before review panel members are able to view any applications, they must first complete the required Unconscious Bias Training Module.

Conflict of interest and confidentiality

The Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations ensures the effective management of conflict of interest of any participant in the review process and, during the review process, ensures the confidentiality of personal and commercial information submitted to the program.

Conflict of interest

Members are responsible for evaluating the merits of applications assigned to them for review, with the exception of those for which they have a conflict of interest.

A conflict of interest is a conflict between a person’s duties and responsibilities as a participant in the review process, and that person’s private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the review panel member, external reviewer, referee or observer:

  • would benefit professionally or personally as a result of the application being reviewed;
  • has a professional or personal relationship with any of the applicants (this includes nominated principal investigators, co-principal investigators, co-applicants and collaborators) or applicants’ institutions; or
  • has a direct or indirect financial interest in the application being reviewed.

A conflict of interest may be deemed or perceived to exist when a review panel member, external reviewer or observer:

  • is a relative or close friend, or has a personal relationship with any of the applicants;
  • is in a position to gain or lose financially/materially from the funding of the application;
  • has had longstanding scientific or personal differences with any of the applicants;
  • is currently affiliated with any of the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
  • is professionally affiliated with any of the applicants as a result of having in the last six years:
    • had frequent and regular interactions with any of the applicants in the course of their duties at their department, institution, organization or company;
    • been a supervisor or a trainee of any of the applicants;
    • collaborated, published or shared funding with any of the applicants, or made plans to do so in the immediate future; or
    • been employed by any of the applicants’ institutions; and/or
    • feels for any reason unable to provide an impartial review of the application.

All review panel members are subject to the same conflict of interest guidelines. If any of these situations apply, the member must declare a conflict of interest. NFRF staff are responsible for resolving areas of uncertainty. A conflict of interest can be declared at any time during the competition cycle. Please contact NFRF program staff if you have any changes.

Confidentiality

The information included by applicants in their applications is protected by the Privacy Act and is provided for the purposes of review only. Details of the application, scoring and recommendation for a specific application are confidential and must never be divulged. Only NFRF staff can release information. Under no circumstances should members disclose to anyone the recommendations from the scoring; this also applies to when the competition is over and the award recipients are announced.

Funding recommendations made by the review panel are subject to approval by the NFRF steering committee and may be changed for reasons of budget, administrative error, or lack of full adherence to policies.

Review panel members are asked not to communicate any information relating to the review of a specific application, or offer to applicants or anyone outside of the panel opinions on the applicants’ chances of success or failure.

In turn, applicants are not to contact panel members regarding the status of their applications (ratings, rank, etc.). By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of the funding decision for a competition. This does not include any notes or feedback a review panel member has provided that were not shared in written form with the other panel members or NFRF staff.

Roles and Responsibilities

Strategic and program steering committees

The CRCC provides strategic direction and oversight for the NFRF and serves as the high-level steering committee. As a tri-agency program, program oversight is delegated to the NFRF steering committee, which includes the presidents of the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC); and the deputy ministers of Innovation, Science and Economic Development Canada, and Health Canada. The NFRF steering committee makes decisions regarding which applications to fund based on the review panel’s recommendations. It also ensures the evaluation process is rigorous, objective and transparent, in keeping with the standards of excellence expected by the agencies and consistent with the program’s objectives.

Multidisciplinary review panel

Membership

The multidisciplinary review panel is composed of national and international members with broad expertise. To achieve a balance among panel members, factors such as area of expertise; language; inclusion in the four designated groups (women, Indigenous peoples, visible minorities and persons with disabilities); region; institution size; career stage; knowledge of best practices for equity, diversity and inclusion (EDI); and experience in research by and with Indigenous peoples are taken into account, as is the inclusion of non-academics. Members are selected to ensure the panel has the capacity to review proposals in both of Canada’s official languages.

The multidisciplinary review panel recommends to the NFRF steering committee the applications that meet the excellence criteria as set out in this funding opportunity for applications to be considered fundable.

Multidisciplinary review panel members

Review panel members evaluate applications and make recommendations to the NFRF steering committee based on their assessments. Specific responsibilities of members include:

  • ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations prior to, during, and after the evaluations and/or meeting;
  • ensuring the integrity and quality of the merit review process, and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
  • ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
  • completing the online training module on unconscious bias in peer review;
  • participating in preparatory meetings/discussions and information sessions prior to evaluations and/or the merit review, where applicable;
  • reviewing in depth the applications assigned to them;
  • ensuring that all documents are considered in their assessments;
  • providing ratings for applications assigned to them by the deadline or prior to the merit review meeting, where applicable; and
  • ensuring that all review materials provided are handled safely and disposed of according to program policy.

NFRF staff

NFRF staff are typically represented by the director, deputy director and/or manager, and the program officers. The staff’s responsibilities include:

  • ensuring that the panel follows the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations;
  • ensuring the integrity and quality of the merit review process, and that each application gets a fair assessment (free of bias and equitable to all applicants) based on the evaluation of all criteria;
  • assigning applications to review panel members;
  • providing advice and guidance to the panel on the program’s policies;
  • ensuring that all documents are considered in the panel’s assessments;
  • keeping notes on procedural aspects of the panel’s functions;
  • working with the co-chairs to manage conflicts of interest, where applicable;
  • recording concerns raised by the panel on issues requiring subsequent attention by staff;
  • ensuring that all review materials provided are handled safely and disposed of according to program policy;
  • if needed, conducting the randomized selection process to select the applications to be awarded grants, from among those deemed fundable by the multidisciplinary review panel; and
  • facilitating the final approval of applications selected by the NFRF steering committee.

Applications and Review Material

Incomplete or non-adherent applications

The onus is on the applicants to provide complete and sufficient information that adheres to the Convergence Portal instructions for attachments and the instructions for completing the NOI and full application. Problems related to application content should be brought to the attention of NFRF program staff, which can be done at any point during the competition. To maintain fairness in the competition, applicants must adhere to the guidelines in preparing application materials. If staff determines that information provided is incomplete or non-adherent to guidelines or instructions, the application may be rejected.

Eligibility of applicants

Eligibility decisions are the responsibility of the NFRF program staff. Review panel members who have doubts as to a researcher’s eligibility should review the application on the same basis as all others, and alert the NFRF staff to potential eligibility problems as soon as possible. The eligibility criteria for applicants can be found in the eligibility section of the competition overview.

Indigenous research

If a review panel member receives a proposal in which the applicants have answered “Yes” to the question “Does your proposal involve Indigenous research as defined by SSHRC?” they must use SSHRC’s Indigenous Research Statement of Principles and Guidelines for the Merit Review of Indigenous Research as references in assessing the application related to or involving Indigenous research. The guidelines are provided to help reviewers build understanding of Indigenous research and research-related activities, and to assist them in interpreting the specific evaluation criteria in the context of Indigenous research.

Merit indicators

The merit indicators for all five criteria (high risk, high reward, feasibility, fit to program and EDI) include a binary scale for panel members to rate whether an application has met the expectations as set out in this funding opportunity.

The matrices include references to major points of consideration to guide review panel members towards arriving at a rating for a given criterion. All applications are evaluated using the same merit indicators.

Types of risk

Note: This information is not used for the review of applications, but for the evaluation of the entire NFRF program.

All reviewers are asked to collect information, for analytical purposes only, identifying the types of risk present in the proposed research projects, according to the following list:

  • challenging current research perspectives or paradigms;
  • using equipment, techniques or approaches proven or assumed to be extraordinarily difficult;
  • bringing together an unprecedented combination of disciplines with different perspectives;
  • viewing the project from an unfamiliar interdisciplinary perspective, to use novel approaches to solve existing problems;
  • other risks (indicate each one in a text box); and
  • no risks

Areas of concern

All reviewers can comment about any areas of concern in the application, including ones related to any of the selection criteria, including EDI, fit to program, gender-based analysis plus (GBA+) / sex- and gender-based analysis (SGBA), Indigenous research, budget or others.

Please note that reviewers must not include any identifying information about the team members (names, research groups, departments, institutions, etc.) when entering their comments.

Budget

The proposed budget is not one of the selection criteria. It is not typically discussed by reviewers, except where it may affect the assessment of feasibility in accomplishing the research with the proposed resources. However, if a reviewer notices an expense that is not eligible under the Tri-Agency Guide on Financial Administration, or one they perceive has been greatly overestimated, they should bring it to the attention of NFRF program staff.

How to Access Applications for Review

All review panel members will access the applications for review through the Convergence Portal, and will only have access to applications they have been assigned.

Multidisciplinary review panel members

Potential review panel members will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete the biographical information section, including their fields of research, in their personal profile. Shortly after the application deadline, multidisciplinary review panel members will receive an email asking them to accept the terms and conditions (T&Cs) in the Convergence Portal. Members must accept the T&Cs before they can continue with the review process. After acceptance, two new tabs will appear on the home screen when the member logs into the Convergence Portal: Committee Conflicts and Committee Assignments. If the T&Cs are accepted prior to applications being assigned, both tabs will remain blank until applications are assigned.

Following this step, members will receive an email informing them that the applications assigned to them are ready under the Committee Conflicts tab. Any problem with the assignment of applications should be brought to the NFRF program staff’s attention as soon as possible or at any point in the process.

Review Process

The purpose of the application review process is to identify excellent applications; that is, those that meet the expectations set out in this funding opportunity for all criteria and are therefore considered fundable.

Review timeline

Date Activity
October 26, 2021 Application deadline
November 3, 2021 Review panel members receive assignments
November 3, 2021 Orientation session for members
November 16, 2021 Deadline for review panel members to indicate conflicts of interest
December 10, 2021 Deadline for review panel members to record scores
December 20-23, 2021 Scores are analyzed and fundable applications are identified; if more applications are deemed “fundable” than there are available funds, proceed with randomized selection process
January 2022 Application results are released to applicants
January 14, 2021 Feedback survey is sent to review panel members

Evaluation by multidisciplinary review panel

Assignment

Each application will be assigned to three members of the multidisciplinary review panel. Members will be assigned to applications based on disciplinary knowledge and expertise.

Language abilities and institutional conflicts of interest will be taken into account in assigning applications to members.

Check for conflicts of interest

Shortly after the deadline, review panel members will receive an email informing them that their list of assignments is ready. Members are asked to log in to the Convergence Portal, review the summary of each of their assigned applications in the “committee conflicts” tab, and indicate if there is a conflict of interest. Once a member has checked all assigned applications for conflicts, they immediately gain access to the applications for which they are not in conflict in the “committee assignments” tab. Some changes to assignments may occur as any conflicts of interest are identified by members. If there are any changes, the panel member will be notified by the NFRF program staff. Any problems should be brought to the NFRF program staff’s attention as soon as possible, or at any point in the process.

Assessment

Multidisciplinary review panel members receive access to all content submitted by applicants.

All eligible applications are to be reviewed and scored in relation to the expectations of this funding opportunity, according to the merit indicator matrices for each of the selection criteria.

In assessing feasibility, members must consider the approach to Indigenous research (if applicable) and GBA+/SGBA considerations in the research design. Members are asked to provide comments on:

  • types of risk; and
  • any areas of concern related to the application.

Result

Based on the multidisciplinary review panel ratings, excellent applications will be identified. Excellent applications are those that meet the expectations set out in this funding opportunity for all criteria and are therefore considered fundable. In the event that more applications are evaluated as excellent than there are available funds, a randomized selection process will be used to select the applications to be awarded grants from among those deemed “fundable”. The randomized selection process provides an equal opportunity for each fundable application to be awarded a grant, and will be conducted in a manner that meets commitments related to early career researchers (ECRs) and EDI.

Applications that do not meet the expectations set out in this funding opportunity for one or multiple criteria are not considered fundable.

The CRCC has committed to ensuring that a proportion of the awards will be reserved for applications led by ECRs, equal to the proportion of ECR-led applications received. The program monitors the proportion of ECR-led applications recommended for funding. It is possible that lower-rated ECR-led projects may be recommended in order to meet the program’s commitment to supporting ECRs.

Feedback to applicants

All applicants will receive the scores assigned to their application by the review panel members. Written feedback will not be provided.

Handling Documents Used in Peer Review

Merit review documents contain personal information as well as information the unauthorized disclosure of which could reasonably be expected to cause serious injury (such as prejudicial treatment or loss of reputation or competitive advantage) to an individual, organization or government. Therefore, these documents are subject to regulation under the Privacy Act, the Access to Information Act and the Policy on Government Security. Protocols must be followed to ensure that information contained in applications, internal and external reviews, and panel discussions remains strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Please refer to the Guide on Handling Documents Used in Peer Review for further details.

Legal and Ethical Information

Responsible conduct of research

Canada’s federal research funding agencies—CIHR, NSERC and SSHRC—are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions and the agencies that, together, help support and promote a positive research environment.

Confidentiality

Members appointed to the panel must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers describing expectations and requirements.

Privacy Act

Personal information refers to any information about an identifiable individual. Based on the Privacy Act, personal information provided by applicants must be used only for assessing applications, making funding decisions, and related uses describing applicants at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal. It is important for panel members to adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Canadian Human Rights Act

The activities of CIHR, NSERC and SSHRC are subject to the Canadian Human Rights Act. The purpose of the Act is to extend the laws in Canada to give effect to the principle that all individuals should have opportunity equal with other individuals to make for themselves the lives that they are able and wish to have. They should also have their needs accommodated, consistent with their duties and obligations as members of society, without being hindered in or prevented from doing so by discriminatory practices based on race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, genetic characteristics, disability or conviction for an offence for which a pardon has been granted or in respect of which a record suspension has been ordered.

Official Languages Act

All review panel members and NFRF program staff must be aware of their obligations and rights as legislated in the Official Languages Act.


Date modified: