Reviewer Manual: 2020 Transformation Competition

Purpose of the manual

This manual is designed as a guide for reviewers for the New Frontiers in Research Fund (NFRF) 2020 Transformation competition. It describes activities to be undertaken by external reviewers, members and chairs of the multidisciplinary/multisectoral review panel and defines the relevant policies, guidelines and deliverables. As the number of letters of intent to apply (LOI) that will be submitted is not known at the time of publishing this guide, the processes outlined below are intended as guidance and may be adjusted as required. Applicants who refer to this manual should note that the content is intended to guide reviewers and outline principles rather than provide them with a set of rules.

A word of thanks

On behalf of the Canada Research Coordinating Committee (CRCC), we would like to thank you for agreeing to participate in the review process for the 2020 Transformation competition. The success of the review process is made possible by dedicated people like you who generously give of your time and expertise. The CRCC and the research community greatly appreciate your efforts.

Members and external reviewers are asked to read this document in its entirety before starting to review applications assigned to them. Clarification on any subject can be obtained from NFRF program staff at any time.

Overview of the 2020 Transformation competition

The NFRF Transformation stream is designed to support large-scale, Canadian-led interdisciplinary research projects that address a major challenge and have the potential to realize real and lasting change (high reward). The challenge may be fundamental, leading to a scientific breakthrough, or applied, with a social, economic, environmental or health impact. Projects are expected to be world-leading, drawing on global research expertise where relevant. For further details about the 2020 competition, including eligibility requirements, please refer to the funding opportunity. The instructions for the notice of intent to apply (NOI), LOI and full application provide further details on the information required at each stage of the competition.

Principles of merit review

Fairness

Success of the NFRF merit review system depends on the willingness and ability of all members of the review process to be fair and reasonable; to exercise rigorous judgment; and to understand—and take into account in a balanced way—the particular context of each application.

Bias

Members are asked to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought, fundamental versus applied research, certain sub-disciplines, areas of research or approaches (including emerging ones), size or reputation of an institution, or the age, personal factors, sex or gender of applicants. Members are cautioned against judging an application based on these factors. Before members are able to view any applications, they must first complete the required unconscious bias training module.

Confidentiality and conflict of interest

The Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations sets out the principles for effectively managing conflict of interest of any participant in the review process and for ensuring the confidentiality of personal and commercial information submitted to the program.

Conflict of interest

Members are responsible for evaluating the merit of applications assigned to them for review, with the exception of those for which they have a conflict of interest.

A conflict of interest is a conflict between a person’s duties/responsibilities as a participant in the review process and that person’s private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the panel member, external reviewer, referee or observer:

• would benefit professionally or personally as a result of the application being reviewed;
• has a professional or personal relationship with any of the applicants (including principal investigators, co-principal investigators, co-applicants and collaborators) or the applicants’ institutions; or
• has a direct or indirect financial interest in the application being reviewed.

A conflict of interest may be deemed or perceived to exist when a panel member, external reviewer or observer:

• is a relative or close friend of, or has a personal relationship with any of the applicants;
• is in a position to gain or lose financially/materially if the application is funded;
• has had long-standing scientific or personal differences with any of the applicants;
• is currently affiliated with any of the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
• is professionally affiliated with any of the applicants as a result of having in the last six years:
  • frequent and regular interactions with any of the applicants in the course of their duties at their department, institution, organization or company;
  • been a supervisor or a trainee of any of the applicants;
  • collaborated, published or shared funding with any of the applicants, or made plans to do so in the immediate future; or
  • been employed by any of the applicants’ institutions; and/or
• feels for any reason unable to review the application impartially.

All members are subject to the same conflict of interest guidelines. If any of these situations apply, the member must declare a conflict of interest and leave the room when the application is being discussed. NFRF staff and the co-chairs are responsible for resolving areas of uncertainty.

Confidentiality

The information provided by applicants in their applications is protected by the Privacy Act and is included for review purposes only. Details of the application, scoring, panel discussions and recommendation for a specific application are confidential and must never be divulged. Only NFRF staff can release information. Under no circumstances should members disclose to anyone the recommendations from the merit review meetings; this also applies following the competition and when the award recipients are announced.

Funding recommendations made by the review panel are subject to approval by the CRCC and are subject to change for reasons of availability of funds or lack of full adherence to policies.

Members are asked not to communicate any information relating to the review of a specific application, or offer opinions on an applicant’s chances of success or failure. Applicants are not to contact panel members regarding the status of their applications (ratings, rank, etc.). By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to applicants when they are notified of the funding decision for a competition. This includes written feedback provided by external reviewers but does not include any notes or feedback a review panel member has provided that were not shared in written form with the other members.

Roles and responsibilities

Steering Committee

For all NFRF competitions, the Steering Committee is the CRCC. The committee ensures that the evaluation process is rigorous, objective and transparent, in keeping with the standards of excellence expected by the research funding agencies and consistent with the program’s objectives. The committee also makes decisions regarding which applications to fund based on the review panel’s recommendations.

Multidisciplinary/multisectoral review panel

Membership

The multidisciplinary/multisectoral review panel is composed of experts, including non-academics, who represent the various research disciplines of the three research funding agencies (the Canadian Institutes of Health Research [CIHR], the Natural Sciences and Engineering Research Council [NSERC], and the Social Sciences and Humanities Research Council [SSHRC]). To achieve a balance among panel members, factors such as area of expertise, language, inclusion in the four designated groups (women, Indigenous peoples, visible minorities and persons with disabilities), region, institution size, career stage, knowledge of best practices for equity, diversity and inclusion, and experience with research by and with Indigenous Peoples are taken into account. Members are selected to ensure that the panel has the capacity to review proposals in Canada’s both official languages.

The members review the applications assigned to them and, collectively, make recommendations to the Steering Committee on those that should be funded.

Chairs

At the LOI and application stages, the multidisciplinary/multisectoral review panel includes chairs who fulfill an oversight role and participate in the review of applications. Chairs are responsible for ensuring that the panel functions effectively and objectively and according to the program’s policies. The chairs establish a positive, constructive and fair-minded environment in which to evaluate the applications.

The chairs’ responsibilities include:

• ensuring that the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations is followed before, during and after the meeting;
• ensuring the integrity and quality of the merit review process and that each application is assessed fairly (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
• completing the online training module for unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions before the merit review meeting;
• including all members in the discussion of applications;
• ensuring that all documents are considered in the panel’s assessments;
• guiding the panel to a consensus recommendation, where applicable;
• assisting with the preparation of feedback to applicants, where applicable;
• contributing to discussions on NFRF policy issues;
• participating in discussions on review panel membership for future competitions; and
• ensuring that all confidential review materials provided to them are handled safely and disposed of according to program policy.

Multidisciplinary/multisectoral review panel and jury members

Members evaluate applications and make recommendations to the Steering Committee based on their assessments. Specific responsibilities of members include:

• ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the evaluations and/or meeting;
• ensuring the integrity and quality of the merit review process and that each application is assessed fairly (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• ensuring that applications involving Indigenous research are reviewed in accordance with SSHRC’s Guidelines for the Merit Review of Indigenous Research;
• completing an online unconscious bias training module for peer review;
• participating in preparatory meetings/discussions and information sessions before evaluations and/or the merit review, meeting when needed;
• reviewing applications assigned to them in depth;
• providing ratings for applications assigned to them by the deadline, before the merit review meeting, when applicable;
• ensuring that all documents are considered in their assessments;
• participating in the review of applications assigned to them by providing a verbal assessment to the entire committee during the meeting, when applicable;
• assisting with the preparation of feedback to applicants, when applicable;
• participating in the discussion and review of all relevant applications, according to their role, when applicable; and
• ensuring that all review materials provided are handled safely and disposed of according to program policy.

NFRF staff

NFRF staff are typically represented at the merit review panel meetings by the director, the deputy director and/or the manager and program officers. Their responsibilities include:

• ensuring that the panel follows the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations before, during and after the review meeting;
• ensuring the integrity and quality of the merit review process and that each application is assessed fairly (free of bias and equitable to all applicants) based on the evaluation of all criteria;
• assigning applications to panel members;
• providing advice and guidance to the panel on the program’s policies;
• ensuring that all documents are considered in the panel’s assessments;
• keeping notes on procedural aspects of the panel’s functions;
• recording comments made by the panel for each application, where applicable;
• working with the co-chairs to manage conflicts of interest, when applicable;
• recording concerns raised by the panel on issues requiring subsequent attention by staff;
• ensuring that all review materials provided are handled safely and disposed of according to program policy; and
• facilitating the final approval of recommended applications by the Steering Committee.

Applications and review material

Incomplete or non-adherent applications

The onus is on applicants to provide complete and sufficient information, adhering to the Convergence Portal instructions for attachments and instructions for completing the NOI, LOI and full application. Problems related to application content should be brought to the attention of staff. In the event that information provided is incomplete or does not adhere to guidelines or instructions, the application may be rejected.

Eligibility of applicants

Assessing eligibility is the responsibility of NFRF program staff. Members in doubt about an applicant’s eligibility should review the application on the same basis as all others and alert staff to potential problem(s) as soon as possible. Eligibility criteria for applicants can be found in the eligibility section of the funding opportunity description.

Indigenous research

If review panel members or external reviewers receive a proposal in which the applicant has answered “Yes” to the question “Does your proposal involve Indigenous research as defined by SSHRC?” they must refer to SSHRC’s Indigenous Research Statement of Principles and Guidelines for the Merit Review of Indigenous Research. These guidelines are provided to help reviewers build understanding of Indigenous research and research-related activities and help them interpret the specific evaluation criteria in the context of Indigenous research.

Merit indicators

The merit indicators are criteria that should be considered when reviewing applications. Each indicator includes statements that explain the major points to consider in arriving at a rating. Although only four ratings are described for the criteria (with the exception of interdisciplinarity, equity, diversity and inclusion [EDI] and early career researchers [ECR]), members may select a rating between each of the four, providing a seven-point scale.

All applications are evaluated using the same merit indicators. Members are encouraged to use the full range of ratings, as appropriate, to achieve a rating distribution that reflects the quality of the applications. Members are expected to discuss and justify their ratings (or adjust them) during the LOI and full application stages.

Areas of concern

Reviewers and multidisciplinary panel members are asked to identify any areas of concern (“other considerations” for external reviewers) they believe exist in the proposed research project, which, depending on the review stage, could include one or more of the following:

• budget
• EDI/ECR
• interdisciplinarity
• gender-based analysis plus / sex and gender-based analysis
• other concerns

Reviewers will be asked to describe the concern noted.

Budget

The proposed budget is not a selection criterion and is not typically discussed except when it may affect the assessment of feasibility in accomplishing the research with the proposed resources. However, if members notice an expense that is not eligible or one that has been greatly overestimated, they are asked to bring it to the attention of NFRF program staff.

How to access applications for review

All members and external reviewers will access applications through the Convergence Portal. Members and reviewers will only have access to their assigned applications.

Multidisciplinary/multisectoral review panel members

Members will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete their biographical information, including their fields of research, in their profile. Members will then receive an email requesting that they accept the terms and conditions of the Convergence Portal. Members must accept the terms and conditions before they can continue with the review process. Once accepted, two new tabs will appear on the home screen when logged into the Convergence Portal: Committee Conflicts and Committee Assignments. If the terms and conditions were accepted before applications are assigned, both these tabs will be blank.

External reviewers

External reviewers will receive an email with detailed instructions asking them to log in to the Convergence Portal and complete their biographical information, including their fields of research, in their profiles. They will then receive an email requesting that they accept the terms and conditions of the Convergence Portal. Reviewers must accept the terms and conditions before they can continue with the review process. Once accepted, a new tab will appear on the home screen when logged into the Convergence Portal: Reviewer Assignments. If the terms and conditions were accepted before applications are ready for review, this tab will initially be blank.

Shortly after the deadline, external reviewers will receive an email informing them that the application(s) assigned to them are ready in the Reviewer Assignments tab. Any problem with the assignment of applications should be brought to the NFRF program staff’s attention as soon as possible.

Review process

Letter of Intent stage

The purpose of this stage is to identify the most meritorious proposals to be invited to submit full applications.

Review timeline

Evaluation by multidisciplinary/multisectoral review panel (LOI)

The multidisciplinary review panel (LOI) will meet virtually to identify the research teams to be invited to submit full applications.

Assignment

Each LOI submission will be assigned to five members for review. Language abilities and institutional conflicts of interest will be taken into account when assigning LOIs.

Conflicts of interest

Shortly after the LOI deadline, members will receive an email informing them that their list of assignments is ready. Before gaining access to their assigned applications, members will receive instructions for indicating any conflicts of interest. Minor changes to assignments could occur if members identify conflicts of interest. Any problems should be brought to the attention of NFRF staff as soon as possible.

Assessment

All eligible LOIs are to be reviewed and scored according to the merit indicator matrices for all five selection criteria:

• high risk: 20% of overall score
• high reward: 60% of overall score
• feasibility: 20% of overall score
• for feasibility, the evaluation will focus only on the proposed approach to the problem or challenge at a relatively high level, commensurate with the amount of information provided at the LOI stage
• interdisciplinarity (pass/fail)
• EDI/ECR (pass/fail)

Members may assign one of seven ratings for the high risk, high reward and feasibility criteria (see Merit Indicators).

Members also provide comments on:

• interdisciplinarity (if the score is a fail)
• EDI/ECR (if the score is a fail)
• any areas of concern or comments related to the proposal (i.e., budget, GBA+/SGBA)

Results

Members’ ratings for the high risk, high reward and feasibility criteria will be compiled, with the mean being used to calculate the overall rating for each criterion. Overall scores will be calculated according to the weighting of the criteria and used to identify the top LOIs to be discussed at the meeting. The cutoff score will be determined based on the distribution of scores and the number of LOIs received. Members may identify any LOIs to be discussed at the meeting, regardless of their overall score.

The scores for interdisciplinarity and EDI/ECR will be recorded as a pass, fail, or mixed. An LOI must have passed both of these criteria in order to be considered to be invited to the full application stage.

All applicants will receive the overall ratings for the high risk, high reward and feasibility criteria, as well as a pass, fail, or mixed assessment for the interdisciplinarity and EDI/ECR criteria. Comments from the review panel will be provided for LOIs that are discussed at the meeting. No comments will be available for those not discussed.

Determining which LOIs proceed to application stage

After discussing each LOI, members will decide by consensus whether an LOI “must be invited,” “should be invited” or “should not be invited” to the full application stage. In determining the distribution of LOIs between the categories, members will be asked to consider all information provided and focus on quality, according to the program criteria, while also considering diversity as an important component of quality. The multidisciplinary/multisectoral review panel will recommend to the NFRF Management Committee a list of 20 to 40 LOIs to be invited to the full application stage.

Application stage

The multidisciplinary/multisectoral review panel (full application) will meet to identify the highest quality proposals, according to the criteria, whose teams will be invited to present their proposals to the jury.

Review timeline

Evaluation by external reviewers (application)

Reviewers will be asked to comment on the strengths and weaknesses of the proposed project in relation to the interdisciplinarity, high risk, high reward and feasibility criteria. They will also be asked to identify any areas of concern.

Assignment

Each application will be assigned to five external reviewers who will each have access to all sections of the full application (except the participants’ personal profiles, suggested reviewers, reviewer exclusions, and certifications).

Teleconferences or videoconferences with external reviewers, moderated by NFRF staff, will be held in order to arrive at a consensus summary report. NFRF staff will draft the consensus summary report for review and approval by external reviewers.

Evaluation by multidisciplinary/multisectoral review panel (full application)

Assignment

All members of the panel will read all applications (except those for which they are in conflict), including the external reviewers consensus summary.

Conflicts of interest

Shortly after the LOI deadline, members will receive an email informing them that their list of assignments is ready. Before gaining access to their assigned applications, members will be provided instructions for indicating any conflicts of interest.

Assessment

Members of the multidisciplinary/multisectoral review panel will have access to all content from the LOI and application stages.

Members receive their assignments and are given access to the applications before they receive the external reviewer assessments.

Members will review each application and provide:

• validation of the interdisciplinarity and EDI/ECR ratings from the LOI stage; and
• scores according to the merit indicator matrices for the high risk (25%), high reward (25%) and feasibility (50%) criteria.

Members may assign one of seven ratings for the high risk, high reward and feasibility criteria (see Merit Indicators). In assessing feasibility, members must consider the approach to Indigenous research (if applicable) and GBA+/SGBA considerations in the research design.

Results

Members’ ratings for the high risk, high reward and feasibility criteria will be compiled, with the mean used to calculate the overall rating for each criterion. Overall scores will be calculated according to the weighting of the criteria and overall ratings calculated for each application.

Multidisciplinary/multisectoral review panel meeting

The multidisciplinary review panel will meet to identify the proposals to be invited to present to the jury.

• All applications will be thoroughly discussed, covering each criterion.
• The lead reader for the application will initiate the discussion, and members will be invited to provide their assessments.
• After discussing each application, the panel will decide whether it “must proceed to jury,” “could proceed to jury” or “should not proceed to jury.” The decision will be consensus-based.
• The panel will identify the 10 to 20 best applications, according to the program criteria, whose teams will be invited to present their proposal to the jury. Each of the identified proposals must meet the standard of excellence and be considered fundable.
• Feedback to the applicant will be prepared.

Jury stage

The objective of this stage is to identify the proposals to recommend for funding. The jury will consider the objectives of the Transformation stream in their deliberations, as well as the diversity of the projects it will recommend for funding, in terms of topics, geography, research teams and potential impacts, in addition to the diverse groups/communities/individuals that will be impacted by the outcomes.

Process

The teams identified by the multidisciplinary/multisectoral review panel will present their proposal in person to the jury in Ottawa.

Jury membership

The jury will be composed of 10 to 15 international and Canadian members representing diverse sectors (policy-makers, government, academia, industry, not-for-profit).

Jury review

Pre-meeting

The jury meeting will begin with an in camera (private) meeting of the jury members to discuss all the applications and identify questions for each. The jury will be reminded of its responsibility to consider the diversity of the applications it recommends for funding. The meeting will then proceed to the “project pitch” segment.

Project pitches by applicants

One hour will be allocated for each application during the project pitch segment of the jury review. This hour will comprise a 10-minute pitch by the applicants (details below) followed by a 30-minute Q&A period. The final 20 minutes will be for the jury members only, during which time they will discuss their impressions of the project pitches.

Applicants must deliver a 10-minute project pitch to the jury. This may include a visual presentation or other materials, but these are not required. Applicants are encouraged to highlight how the project addresses the Transformation stream objectives (listed below) and evaluation criteria, and to include any additional/supplemental information. Applicants are also encouraged to address the diversity aspects that the jury must consider in its deliberations.

In their pitches, applicants must demonstrate how the project:

1. tackles a well-defined problem or challenge;
2. proposes a novel world-leading approach that is different from the current state-of-the-art approaches to the issue (high risk criterion);
3. is interdisciplinary (interdisciplinarity criterion), bringing different perspectives to the defined problem; and
4. has the potential (feasibility criterion) to be transformative, which is defined as the potential to create a significant and real change or impact—a noticeable leap or tangible breakthrough rather than an incremental advance (high reward criterion).

Up to five individuals for each application may be present to pitch to the jury. The five representatives should provide a breadth of perspectives on behalf of the proposed project. They may include members of the research team, collaborators, administrators from the nominated principal investigator’s institution, etc. It is expected that all those present will contribute meaningfully to the discussion with the jury.

Post-meeting

At the end of the meeting, the jury will discuss the merits of all the pitches and come to a consensus on which six to eight applications (depending on the amounts requested) to recommend for funding. The jury will be reminded to consider its recommendations as a whole: to recommend projects that are compelling while also considering, as noted above, diversity in terms of topics, geography, research teams, and potential impacts, including the diversity of the groups/communities/individuals that will be impacted by the outcomes.

Steering Committee approval

The CRCC will make the final decisions on which applications to fund based on the review panel’s and jury’s recommendations.

Guide on handling documents used in merit review

Merit review documents contain personal information as well as information whose unauthorized disclosure could reasonably be expected to cause serious injury (such as prejudicial treatment or loss of reputation or competitive advantage) to an individual, organization or government. Therefore, these documents are subject to regulation under the Privacy Act, the Access to Information Act and the Policy on Government Security. Protocols must be followed to ensure that information contained in applications, internal and external reviews, and panel discussions remains strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Please refer to the Guide on Handling Documents Used in Peer Review for further details.

Legal and ethical information

Responsible Conduct of Research

Canada’s federal research funding agencies—CIHR, NSERC and SSHRC—are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions and the agencies that together help support and promote a positive research environment.

Confidentiality

Members appointed to the panel must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers describing expectations and requirements.

Privacy Act

Personal information refers to any information about an identifiable individual. Based on the Privacy Act, personal information provided by applicants must be used only for assessing applications and making funding decisions, and for related uses describing applicants at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal. It is important for panel members to adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Canadian Human Rights Act

The activities of CIHR, NSERC and SSHRC are subject to the Canadian Human Rights Act. The purpose of the Act is to extend the laws in Canada to give effect to the principle that all individuals should have equal opportunity to make for themselves the lives that they are able and wish to have. They should also have their needs accommodated, consistent with their duties and obligations as members of society, without being hindered in or prevented from doing so by discriminatory practices based on race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, genetic characteristics, disability or conviction for an offence for which a pardon has been granted, or in respect of which a record suspension has been ordered.

Official Languages Act

All review panel members and NFRF program staff must be aware of their obligations and rights as legislated by the Official Languages Act.