Manual for Multidisciplinary Review Panel: 2018 Exploration

Purpose of the manual

This manual was published to:

• set policies and procedures for the adjudication of applications submitted to the inaugural competition of the New Frontiers in Research Fund (NFRF)—Exploration funding opportunity; and
• encourage uniformity and consistency in the application of these policies and procedures.

Applicants may also benefit from reading the sections related to the evaluation process. However, applicants should note that the content is intended to guide review panel members and to outline principles, rather than to provide panel members with a set of rules.

A word of thanks

On behalf of the Canada Research Coordinating Committee (CRCC), we would like to thank everyone who agreed to serve as a member of the multidisciplinary review panel for the NFRF—Exploration inaugural competition. The success of the review process is made possible by dedicated people like you, who generously give of your time and expertise. Your efforts are greatly appreciated by the CRCC and the research community.

It is essential that panel members read this entire document before reviewing assigned applications. Should members have questions on any subject, they may speak with program staff before or during the meeting.

Principles of merit review

Fairness

Success of the NFRF merit review system is critically dependent on the willingness and ability of all multidisciplinary review panel members to be fair and reasonable; to exercise rigorous scientific judgment; and to understand, and take into account in a balanced way, the particular context of each application.

Biases

Members are asked to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought; fundamental versus applied research; certain sub-disciplines; areas of research or approaches (including emerging ones); size or reputation of an institution; or the age, personal factors, sex or gender of the applicant. Members are cautioned against any judgment of an application based on such factors. Before members are able to view any applications, they must first complete the mandatory online training module for raising awareness of unconscious bias in peer review.

Confidentiality and conflict of interest

The Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations applies to ensure the effective management of conflict of interest of any participant in the review process and to ensure, during the review process, the confidentiality of personal information and commercial information submitted to the program.

Members are responsible for evaluating the merits of applications assigned to them for review, with the exception of those with which they have a conflict of interest.

Conflict of interest

A conflict of interest is a conflict between a person’s duties and responsibilities for the review process and that person’s private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the panel member, external reviewer, referee or observer:

• would receive professional or personal benefit resulting from the application being reviewed;
• is a relative or close friend or has a professional or personal relationship with any of the applicants (including nominated principal investigators, co-principal investigators or co-applicants) or the applicants’ institutions;
• has a direct or indirect financial interest in the application being reviewed;
• is in a position to gain or lose financially/materially from the funding of the application;
• is currently affiliated with any of the applicants’ institutions, organizations or companies—including research hospitals and research institutes;
• is closely professionally affiliated with any of the applicants, as a result of having in the last six years:
• frequent and regular interactions with any of the applicants in the course of their duties at their department, institution, organization or company;
• been a supervisor or a trainee of any of the applicants;
• collaborated, published or shared funding with any of the applicants, or have plans to do so in the immediate future; or
• been employed by any of the applicants’ institutions; and/or
• feel for any reason unable to provide an impartial review of the application.

All members are subject to the same conflict of interest guidelines. If any of these situations apply, the member must declare a conflict of interest and leave the room when the application is being discussed. NFRF staff and the co-chairs are responsible for resolving areas of uncertainty.

Confidentiality

Members appointed to the panel must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers, which describes expectations and requirements.

The information applicants provided in their applications is protected by the Privacy Act and is made available for reviewing purposes only. Details of the application, scoring, panel discussions and recommendations on a specific application are confidential and must never be disclosed. Under no circumstances should members disclose to anyone the recommendations emanating from the merit review meetings.

The panel makes funding recommendations, which are subject to approval by the Steering Committee (which is the CRCC for NFRF) and may be changed for reasons of budget, administrative error, or lack of full adherence to policies.

Panel members are asked to not approach or communicate in any way with applicants, or anyone outside of the panel, about any information relating to the review of a specific application, or offer opinions on the applicant’s chances of success or failure. In turn, applicants are not to contact panel members regarding the status of their applications (ratings, rank, etc.). If approached by an applicant concerning a decision, panel members should decline to discuss the matter and advise the enquirer to contact NFRF staff directly.

Guide on Handling Documents Used in Merit Review

Nomination and merit review documents contain personal information as well as information whose unauthorized disclosure could reasonably be expected to cause serious injury (such as prejudicial treatment or loss of reputation or competitive advantage) to an individual, organization or government. As such, these documents are subject to regulation under the Privacy Act, the Access to Information Act and the Policy on Government Security. Measures must be followed to ensure that information contained in applications, internal and external reviews and panel discussions remain strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Please refer to the Guide on Handling Documents Used in Merit Review for further details.

Privacy Act

Personal information means any information about an identifiable individual. Based on the Privacy Act, personal information provided by applicants must be used only for the purpose of assessing applications, making funding decisions and for certain related uses described to applicants at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal.

By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of the funding decision for a competition.

Program description

NFRF is designed to:

• support transformative interdisciplinary and high-risk/high-reward research;
• provide opportunities for international research collaborations; and
• offer rapid-response opportunities to emergent issues.

For more information, read the full description of the New Frontiers in Research Fund and a description of the inaugural Exploration competition.

Roles and responsibilities

Steering Committee

The Steering Committee is the Canada Research Coordinating Committee for all NFRF competitions. The committee ensures that the evaluation process is rigorous, objective and transparent, in keeping with the standards of excellence expected by the agencies and consistent with the program’s objectives. The committee also makes decisions regarding which applications to fund and at what level, based on the review panel’s funding recommendations.

Multidisciplinary Review Panel

Membership

The multidisciplinary review panel is composed of experts who represent the various research disciplines of the three research granting agencies: the Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council, and Social Sciences and Health Research Council.

A balanced membership is achieved by taking into account factors such as language, gender, region, career stage, sector of the economy, discipline and type of institution, and including non-academics. Members are selected to ensure that the panel has the capacity to review proposals in both of Canada’s official languages.

Co-chairs’ responsibilities

The panel co-chairs are responsible for ensuring that the panel functions smoothly, effectively, objectively, and according to the program’s policies. The co-chairs establish a positive, constructive and fair-minded environment in which the applications are to be evaluated. The co-chairs fulfil an oversight role and, if required, can also participate in the review of applications.

The co-chairs’ responsibilities include:

• ensuring that the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations is followed prior to, during and after the meeting;
• ensuring the integrity and quality of the merit review process and ensuring that each application receives a fair assessment (that the merit review process is free of bias and is equitable to all applicants) based on evaluation of all criteria;
• completing the online training module on raising awareness of unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions prior to the merit review meeting;
• ensuring the involvement of all members in the review of applications;
• ensuring that all documents are considered in the panel’s assessments;
• appointing a delegate as chair when they leave the meeting room because of conflict of interest or when acting as a reader on a file;
• ensuring the panel’s final recommendations accurately reflect the consensus of its members;
• assisting with the preparation of feedback to applicants, where applicable;
• contributing to discussions on NFRF policy issues;
• participating in discussions of membership for future competitions; and
• ensuring that all confidential review materials provided to them are handled safely and disposed of according to program policy.

Member responsibilities

Members participate in the evaluation of applications and make recommendations based on their assessment to the Steering Committee. Specific responsibilities of members include:

• ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations prior to, during and after the meeting;
• ensuring the integrity and quality of the merit review process and ensuring that each application gets a fair assessment (that the merit review process is free of bias and is equitable to all applicants);
• completing the online training module on raising awareness of unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions prior to the merit review meeting;
• reviewing their assigned applications in depth;
• providing ratings for their assigned applications by the deadline, prior to the merit review meeting;
• ensuring that all documents are considered in their assessments;
• participating in the review of their assigned applications by providing a verbal assessment to the entire committee during the meeting;
• participating in the discussion and review of all relevant applications, according to their role; and
• ensuring that all review materials provided are handled safely and disposed of according to program policy.

Tri-agency Internal Review Committees

Senior employees of the three research granting agencies with relevant expertise and experience will participate in assessing applications to determine if they meet the Equity, Diversity and Inclusion (EDI) and Interdisciplinarity criteria. Specific responsibilities of members of the Tri-agency Internal Review Committees include:

• ensuring that they follow the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations prior to, during and after the meeting;
• ensuring the integrity and quality of the merit review process and ensuring that each application gets a fair assessment (that the merit review process is free of bias and is equitable to all applicants);
• completing the online training module on raising awareness of unconscious bias in peer review;
• participating in preparatory meetings/discussions and information sessions prior to the review meeting;
• reviewing their assigned applications in depth;
• providing pass/fail ratings for their assigned applications to determine if the EDI or the Interdisciplinarity criteria are met, depending on the committee they are in;
• participating in the review of their assigned applications by providing a verbal assessment to the entire committee during the meeting, as needed;
• participating in the discussion and review of all relevant applications, according to their role; and
• ensuring that all review materials provided are handled safely and disposed of according to program policy.

NFRF staff

NFRF staff are typically represented at the merit review meeting by the director, program manager and program officers. Their responsibilities include:

• ensuring that the panel follows the program’s Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations prior to, during and after the meeting;
• ensuring the integrity and quality of the merit review process and ensuring that each application gets a fair assessment (that the merit review process is free of bias and is equitable to all applicants);
• assigning applications to panel members;
• providing advice and guidance to the panel on the program’s policies;
• ensuring that all documents are considered in the panel’s assessments;
• keeping notes on procedural aspects of the panel’s functions;
• recording comments made by the panel for each application;
• working with the co-chairs to manage conflicts of interest;
• recording concerns raised by the panel on issues requiring attention by staff at a later time;
• ensuring that all merit review materials provided are handled safely and disposed of according to program policy; and
• facilitating the Steering Committee’s final approval of recommended applications.

Review procedure

Overview

To better promote groundbreaking, disruptive and interdisciplinary research, the agencies will explore innovative merit review processes as they launch successive competitions; therefore, processes may differ from one round to the next.

The following review process will be used for the inaugural Exploration competition:

1. All eligible applications will be reviewed by members of a multidisciplinary review panel and scored according to three selection criteria: High Risk, High Reward and Feasibility. Each member of the multidisciplinary review panel will also identify their top applications, taking into consideration the application as a whole, including the EDI and interdisciplinarity criteria.
2. Staff will compile ratings and lists from all members to identify the top-ranked applications.
3. The top-ranked applications will then be assessed by two internal committees of senior agency employees, which will evaluate the EDI and the Interdisciplinarity sections and attribute a pass/fail score.
4. The multidisciplinary review panel will meet in Ottawa. The panel will be divided into five multidisciplinary sub-committees for logistical reasons. Each sub-committee will discuss their top-ranked applications (identified in step 2, above) and select which ones to recommend for funding.
5. The CRCC, in their role as the Steering Committee, will make a decision regarding which applications to fund and at what level, based on the review panel’s funding recommendations.

Assignment of reviewers

Every application will be assigned to three members for review. Two of the three members will be assigned based on commonality between their areas of expertise and the application’s subject matter. The third member will not have disciplinary expertise.

Members will be asked to log in to the Convergence Portal and complete biographical information, including areas of expertise (using the new Canadian Research and Development Classification). Members will receive an email with detailed instructions on how to complete this. After completing this step, members will also receive an email regarding the Terms and Conditions related to confidentiality and conflicts of interest.

Shortly after the application deadline, members will be provided a list of applications in the Convergence Portal. Members are asked to review a summary of each of these applications and indicate if they have a conflict of interest. This list will include their preliminary assignments as well as additional applications in order to manage any conflicts of interest that might arise.

Members must flag any problem with the assignment of applications with staff as soon as possible. In exceptional circumstances, issues with the assignment of an application can be flagged as late as the merit review meetings.

Reviewer roles

Each application is initially assessed by three members. All three reviewers are responsible for carrying out an in-depth review of the application and providing ratings for three of the selection criteria. In addition to the ratings, members are also asked to identify any areas of concern that they have with the application, including with respect to the EDI and the Interdisciplinarity selection criteria.

Once all assigned files are scored by a reviewer using the merit indicators, the reviewer will be asked to identify the top applications within their assigned files; this might not match their top six scores. Reviewers are asked to consider applications as a whole, including the EDI and the Interdisciplinarity criteria when identifying their top applications.

The chair will designate a reviewer to open the discussion on the application at the meeting.

Applications and review material

Incomplete or non-adherent applications

The onus is on the applicants to provide complete and sufficient information that adheres to the Convergence Portal Presentation and Attachment Standards and Instructions for Completing an Application. Problems related to the application content should be brought to the attention of staff. In order to maintain the principle of fairness in the competition, applicants must adhere to the guidelines in the preparation of application materials. In the event that staff determines that the information provided is incomplete or non-adherent to guidelines or instructions, the application may be rejected.

Eligibility of applicants

Eligibility decisions are the responsibility of staff. Members who have doubts as to a researcher’s eligibility should review the application on the same basis as all others, and alert staff of the potential problems as soon as possible. The eligibility criteria for applicants can be found in the Eligibility section of the funding opportunity description.

Review materials

Applications will be posted as PDFs on the extranet/SharePoint site in sub-committee folders. Members will find all assigned applications in their sub-committee.

Members will access a personalized evaluation spreadsheet through the extranet/SharePoint site. Members are asked to do the following:

• provide ratings for three criteria for all applications assigned to them;
• identify the types of risk present in the proposed research and any areas of concern with the application; and
• upload their finalized spreadsheet prior to the deadline (see Timelines).

Evaluation of applications

Overview

Members are asked to provide ratings for the following three selection criteria:

• High Risk (40%);
• High Reward (40%); and
• Feasibility (20%).

Members must only base their evaluations on the information in the review material provided. They must not research or access additional information about publication status, other funding requests, prizes, student outcomes, or impact factors that are not included in the review material.

Selection criteria

Several elements are considered in the evaluation of each selection criterion. There is no prescribed weighting of elements within any criterion. Members should use their expertise and judgment in conjunction with the merit indicators when determining the relative importance of elements for any particular case.

Interdisciplinarity

NFRF is intended to support meritorious projects that are not currently funded or could not easily be funded through existing agency-specific programs. Therefore, applications must clearly demonstrate that they are combining disciplines in a unique or novel way. The proposed research project must include elements from at least two different disciplines (as defined by a group-level classification based on the Canadian Research and Development Classification). In cases where all the disciplines fall under the mandate of one agency, applicants are asked to explain why their proposed research cannot easily be funded through existing agency-specific programs.

Equity, diversity and inclusion

The research project must meaningfully engage members of underrepresented groups within the research team, including students, postdoctoral researchers, faculty and partners. Underrepresented groups include, but are not limited to, the four designated groups (women, Indigenous peoples, members of visible minorities, and persons with disabilities).

Applicants will be asked to describe concrete measures that address each of the following three key areas: 

• Team composition and training activities: Outline measures and safeguards that will be implemented to ensure that EDI principles are key considerations in composing and managing the research group and training activities.
• Recruitment processes: What safeguards will be put in place to ensure the recruitment of additional team members (faculty, postdoctoral researchers, graduate students, etc.) is open, transparent and aligned with best practices (e.g., minimizing barriers and mitigating against unconscious bias)?
• Inclusion: What measures will be put in place to ensure team members from underrepresented groups are supported and integrated into the team?

High Risk

Applications must demonstrate the degree to which they are high risk.

High Reward

Applications must justify why the project should be considered high reward.

Feasibility

Applications must address elements related to the feasibility of the project being proposed.

Note that the Feasibility criterion is not meant to consider the likelihood of the project being successful, but rather that the right elements are in place to increase the probability of completing the project (even if it won’t be successful). This criterion is not meant to contradict the High Risk criterion.

Budget

The proposed budget is not one of the selection criteria and is not typically discussed except when it has an impact on assessing the project’s feasibility; specifically, if the research can be accomplished with the proposed resources. However, if a member notices an expense that is not eligible under the Tri-Agency Financial Administration Guide, is not related to the research project, or whose costs has been greatly overestimated, please bring it to the attention of staff.

Merit Indicators

The merit indicators are a scale of qualifiers that contain statements with reference to major points of consideration, to guide members towards arriving at a rating for each selection criterion.

All applications are evaluated using the same merit indicators. Members are encouraged to use the full range of quality ratings, as appropriate, to achieve a distribution of ratings that reflects the quality of the applications being evaluated. Members are expected to discuss and justify their ratings (or adjust them) during the meeting.

Members must make every effort to review applications without bias. Refer to the section above on biases for more details.

Merit Indicators for High Risk

Merit indicators for High Reward

Merit indicators for Feasibility

Merit Indicators for Equity, Diversity and Inclusion

Merit indicators for Interdisciplinarity

Members’ top applications

Members are asked to identify their top applications once they have rated all assigned applications. Members are asked to consider the interdisciplinarity and EDI aspects of the applications when selecting these.

Framework for funding recommendations

To ensure that applications meet the minimum quality threshold, applications with either of the following will automatically be considered not fundable:

• Rating of Low for any of the three selection criteria (High Risk, High Reward and Feasibility); and
• Ratings of Medium-Low for two of the three selection criteria.

The highest ranked applications, as identified by members’ ratings and top selections, will be discussed at the panel meeting in Ottawa. For logistical reasons, the review panel will be divided into five sub-committees, with approximately equal numbers of members and assigned applications. The sub-committee membership will reflect the membership of the entire panel; sub-committees will not be created based on disciplines. The strengths and weaknesses of each application will be discussed, and all members are expected to contribute to the discussion, even if they were not assigned to the application (with the exception of applications with which a member has a conflict of interest).

After the sub-committee has discussed of all of its assigned applications, all members will be asked to identify the applications they feel should be awarded. This data from all members in the sub-committee will be compiled to create a ranked list, which will be used to identify the applications to be recommended for funding. If multiple applications have the same rank (same number of members are recommending them for funding), the sub-committee may be asked to confirm the ranked order. The committee may be asked to consider factors such as EDI or interdisciplinarity in determining the ranked order.

In situations involving a violation of policy or guidelines, staff are able to overturn a funding recommendation. The CRCC is responsible for final decisions on funding recommendations.

Communication of results

The competition results will be communicated to applicants shortly after final approval. The results lists are released to each host institution shortly before, or concurrent with individual notifications. Funding decisions and related statistics will be posted online at a later date.

Timelines

Legal, ethical and policy information

Guidelines for effective merit review of Indigenous research

The Guidelines for the Merit Review of Indigenous Research were developed to ensure that the merit review of Indigenous research upholds the NFRF program’s principles for merit review (see Principles of merit review). In support of SSHRC’s Indigenous Research Statement of Principles, the NFRF program provides these guidelines to merit reviewers to help build general understanding of Indigenous research and, where applicable, to assist committee members in interpreting NFRF’s selection criteria in the context of Indigenous research. The NFRF program relies on the merit reviewers to judge the extent to which the guidelines may be applied to a particular research proposal.

Responsible conduct of research

Canada’s federal granting agencies are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions, and the agencies that together help support and promote a positive research environment.

Open access and data management

Canada’s federal research granting agencies believe research results produced using public funds belong, to the fullest extent possible, in the public domain. Grant holders must comply with the Tri-Agency Open Access Policy on Publications. The NFRF program’s expectations for research data management—including the responsibilities of researchers, research communities, research institutions and research funders in meeting these expectations—are described in the Tri-Agency Statement of Principles on Digital Data Management.

Canadian Human Rights Act

The activities of the agencies are subject to the Canadian Human Rights Act. The purpose of the Act is to give effect to the principle that every individual should have equal opportunity with other individuals to make the life that he or she is able and wishes to have, consistent with the duties and obligations as a member of society, without being hindered or prevented from doing so by discriminatory practices.

Official Languages Act

All panels and staff are fully aware of their obligations and rights regarding official languages as legislated in the Official Languages Act.