Co-chairs’ Report: 2019 Exploration

This report from the co-chairs of the New Frontiers in Research Fund (NFRF) Multidisciplinary Review Panel outlines the 2019 Exploration competition review process and summarizes observations by members. It is submitted for consideration to the Canada Research Coordinating Committee.

Co-chairs

Competition process

The NFRF Exploration stream is designed to promote high-risk, high-reward and interdisciplinary research. The 2019 Exploration competition involved a multistage review process.

At the Letter of Intent (LOI) stage, 917 submissions were received, of which 321 (35%) were led by early career researchers (ECRs). For a submission to be considered ECR-led, both the nominated principal investigator and the co-principal investigator (if there was one) had to meet the program’s definition of an ECR. Each LOI was evaluated by three members of the Multidisciplinary Review Panel according to the High Risk (50%), High Reward (50%) and Interdisciplinarity / Fit to Program (pass/fail) criteria. Following this evaluation, 335 proposals were invited to the application stage. Of those, 123 (36.7%) were ECR-led.

At the application stage, 325 full submissions were received. Each application was evaluated by five members of the Multidisciplinary Review Panel according to the High Risk (40%), High Reward (40%), Feasibility (20%) and Equity, Diversity and Inclusion (EDI) (pass/fail) criteria. In response to feedback from the 2018 competition, expert assessments were sought from external reviewers through a double-blind process, where the reviewers did not have information identifying the applicants. The external reviewers provided assessments according to the High Risk and High Reward criteria, as well as Feasibility as it related to the research plan only.

The 104 members of the Multidisciplinary Review Panel convened in Ottawa from March 10 to 12, 2020, during which all 325 applications were discussed. Members and applications were assigned to one of six subcommittees, which were similar in composition according to members’ agency representation/discipline, sex, language, career stage and region. Each subcommittee was led by three co-chairs, with each co-chair representing one of the three federal research funding agencies.

All members in a subcommittee, provided they did not have a conflict of interest, participated in the discussion of all applications assigned to their subcommittee. Members then voted for the applications they recommended for funding, leading to the creation of a ranked list of applications for each subcommittee. The co-chairs of the six subcommittees met to compare rankings and finalize the recommendations, ensuring fairness across the subcommittees. There were 186 applications recommended for funding, representing an equal number of grants from each subcommittee. Of those, 71 (38.1%) were ECR-led.

Observations

At the end of the competition review process, a policy discussion was held with each subcommittee. An additional one was held afterwards, with all co-chairs. In general, members were very positive about the program and the overall competition process. Members who had participated in the 2018 competition noticed improvements in the process this year, including that members, overall, were more engaged, and that the additional time for discussion led to richer discussions. The summary of the discussions and suggestions for future considerations are below.

Two-stage adjudication process

• The LOI stage, introduced in the 2019 competition, was appreciated by members, who see it as an efficient screening method for early-stage proposals, allowing them to focus on the most competitive proposals at the full application stage.
• Some members felt that including the LOI content in the full application lengthened the document unnecessarily. Other members noted that it was particularly useful for applications that did not seem to be interdisciplinary; in many of these cases, the focus of the proposal changed somewhat between the LOI and application, helping to explain how it passed the Interdisciplinarity / Fit to Program criterion at the LOI stage when it no longer seemed to at the application stage. Members, therefore, recommend that the LOI be attached as an annex to the full application.

Double-blind external reviews

• External reviewer reports were appreciated, being described as “essential” and “extremely useful.” This was considered to be particularly true when the proposal was in an area outside a member’s area of expertise.
• The double-blind process, which directs the focus of the review on the proposal, was seen as effective and valuable. Members indicated their preference that this approach continue to be used.
• The difficulty in securing external reviews for all applications was noted, with members proposing that a minimum of two reports per application be required, whereas more than four is excessive. Members suggested that identification of external reviewers could be facilitated by asking members to suggest reviewers. Other strategies suggested included making it mandatory for grant holders to serve as external reviewers, providing letters of recognition to external reviewers, or, if possible, compensating external reviewers.

Equity, diversity and inclusion

• The active participation of all members in the review of the EDI criterion, including the failing of a number of applications, was a notable development for the 2019 competition.
• The considerable improvements to the EDI guidelines for applicants and members were recognized, while also acknowledging that additional improvements are required. In particular, instructions should be clearer and more specific.
• Applications that failed the EDI criterion, as well as those with poorer-quality EDI criterion sections, had several common weaknesses. These included using identifying information in describing team composition; focusing only on the diversity of the team and not on inclusion; inadequately detailing concrete practices; and failing to describe expected outcomes and metrics for assessing them.
• Members also identified examples of applications with particularly strong responses to the EDI criterion, which included concrete practices embedded across all levels of the research system: research team, institution, and broader disciplines/research fields. 
• Members recommended it be clarified to applicants that the EDI plan must address all four designated groups as defined by the Employment Equity Act (women, Indigenous peoples, persons with disabilities, and racialized minorities).
• Members asked for clearer instructions and further calibration exercises for evaluating the EDI criterion. A few members suggested that EDI could be provisionally assessed at the LOI stage, providing an opportunity for applicants to improve their EDI sections for the application stage.
• It was suggested that all applications discussed during the meetings, including those that pass, should receive comments on the EDI criterion, in order to foster continued improvements in this area.
• It was suggested that an EDI accountability mechanism should be introduced for successful applicants. Applicants must be required to report on outcomes of EDI measures, to help ensure implementation of practices to support EDI. Aggregate results should also be publicly shared to demonstrate progress in this regard.

Indigenous research

• Members considered Indigenous research as an important dimension of this program, and were very conscientious about it during the review process, as well as in the meetings. In this spirit, they also believed that improvements could be made to support members in their reviews.
• Members experienced difficulties with the High Reward criterion when assessing Indigenous research, specifically with respect to the merit indicator on “Reach” and the descriptor “Impacts a large and diverse community or communities.” They recommend the descriptor be modified to be inclusive of Indigenous communities where cultural appropriateness and respect for differences is critical.
• While noting the efforts made to have members with experience in Indigenous research on each subcommittee, members suggested that additional efforts be invested to increase representation.
• Guidelines on the evaluation of Indigenous research specific for the NFRF program were recommended for both applicants and members.

High risk, high reward, feasibility

• While many proposals had an element of high risk and high reward, members noted that the numbers with very high-risk and bold ideas were limited. Returning members noted that, overall, there seemed to be fewer very high-risk proposals in the 2019 competition compared to the 2018 competition. Creative measures may be required to further encourage such ideas.
• Applications that stood out during review stated their goals boldly, so that members were not left to “connect the dots” or wait for a “big reveal” later in the proposal. They were well organized, proposed something different, explained how the project would bring about the proposed impact in the short and long term, and were clearly written.
• The relatively low number of applications from the social sciences and humanities relative to the health sciences and natural sciences and engineering was again perceived. While the changes in language from the inaugural competition were noted, it was suggested that further effort be put into identifying examples of high-risk and high-reward research in these disciplines, to use in outreach efforts. Additional examples of types of high-risk or high-reward outcomes could be incorporated into the evaluation matrixes (e.g., influencing policy as a high-reward outcome).

Interdisciplinarity / fit to program

• Applications that demonstrated a truly interdisciplinary approach included an explanation of how the team would integrate novel methods in their research, as well as a well-written work plan. Moreover, applications were reviewed by experts and non-experts, so those written for a multidisciplinary audience, without jargon, were better received.
• Discrepancies in the Interdisciplinarity / Fit to Program criterion between the LOI and the full application stages were an issue with some proposals. As a result, members strongly recommended that the Interdisciplinarity / Fit to Program criterion be reassessed at the full application stage.
• Opinions differed about the eligibility of interdisciplinary proposals that fall within the mandate of one of the three federal research granting agencies. Some members felt that eligibility should be limited to those that cross agency mandates, since the interpretation of whether a proposal would be easily supported through funding opportunities currently offered by the three agencies is subjective, and depends on the applicant’s argument and a member’s personal experience and knowledge of agency programs and policies.

Review process and competition meeting

• A triage process at the application stage could be introduced so that only the top-rated applications are discussed, reducing the length of the meeting and minimizing reviewer fatigue.
• Some members suggested that it would be sufficient to assign three reviewers to each application, while others noted they really appreciated the input from all five members.
• The evaluation matrices should be reviewed, to change the ratings to either a numerical score or more distinct labels.