Exploration: Merit Indicators for the Review Process


Using the matrices

For each criterion, the reviewer is asked to consider relevant elements, which may include a few, several or all of the elements as outlined in the matrices, as well as some not listed. The matrices are intended to be used as a guide. A proposal does not have to be rated Exceptional against all elements to receive an exceptional rating for the criterion overall. It is left to the discretion of the reviewer to balance assessments of individual elements to provide an overall rating per criterion.

For the High-Risk, High-Reward and Feasibility evaluation criteria, the matrices indicate four ratings: Exceptional, Very Good, Fair and Poor. However, members can use a seven-point scale in their assessments, selecting ratings that fall between the four described. The seven-point rating scale for the High-Risk, High-Reward and Feasibility evaluation criteria is as follows:

  • Exceptional (described)
  • Excellent (between Exceptional and Very Good)
  • Very Good (described)
  • Good (between Very Good and Fair)
  • Fair (described)
  • Inferior (between Fair and Poor)
  • Poor (described)

Criterion: Interdisciplinarity

Whether a project meets the New Frontiers in Research Fund (NFRF) program’s expectations for interdisciplinarity can be defined by elements including, but not limited to, the following.

- Pass Fail
Novelty of perspective Pushes the boundaries in terms of interdisciplinarity, integrating two or more disciplines that are not commonly combined. Proposes an interdisciplinary approach where there is a long tradition and/or established co-operation/collaboration/interaction between the disciplines.
Novelty of approach Proposes the application or adaptation of frameworks/tools/methods/techniques from one discipline to solve a problem in another discipline. (This can also apply to projects where there is a history of collaboration between the disciplines.) The proposed frameworks/tools/methods/techniques are already in use in or easily applied to the second disciplinary area, requiring little adaptation or development.
Project design Designed from an interdisciplinary perspective. The project is an interdisciplinary component “added on” to a more conventional project or program of research.
Other - The application did not adequately establish the interdisciplinary nature of the project.

Applications are assessed for Interdisciplinarity and must receive a pass to be considered for funding.

Interdisciplinarity subcriterion: Fit to Program

Fit to Program is a subcriterion of Interdisciplinarity.

- Pass Fail
Fit to Program Proposes a project that pushes the boundaries of what can be funded through the agencies, according to their mandates and existing suites of programs, as a result of its interdisciplinary approach and high-risk nature. The scope of the proposed project (subject and approach) fits within the parameters of the mandate and existing suite of programs of one or more of the agencies.

All applications are assessed for Fit to Program and must receive a pass to be considered for funding.

Criterion: Equity, Diversity and Inclusion

To meet the NFRF program’s expectations for equity, diversity and inclusion (EDI), projects must pass each of the following elements.

- Pass Fail
Analysis of context

Shows understanding of EDI considerations / systemic barriers in the context of the research team.

Concrete and specific examples are cited in analysis.

Demonstrates a strong commitment to EDI overall.

Analysis of context is generic and/or does not point to one or more systemic barriers.

Evidence of commitment to EDI overall is lacking.
Concrete practice for each area Lists at least one concrete practice that targets the specific context listed for each area. A concrete practice is not listed for one or more of the areas, or the concrete practices listed are not related to the context that was described.
Implementation Provides a description of how the concrete practice has been/will be realistically implemented.

Provides no or an unclear description of how the concrete practice will be implemented.

The implementation plan is unrealistic.
Impact Explains how the concrete practice will impact EDI, and how it will be measured.

Gives no indication of how the impact will be measured.

Does not explain the anticipated impact of the concrete practice on EDI, or how it will be measured.

Criterion: High Risk

For information on how the rating scale is used in assessments, refer to Using the Matrices.

How high risk a project is can be defined by elements including, but not limited to, the following.

- Exceptional Very good Fair Poor
Unique directions Develops a completely new theory. Develops a novel concept that bridges theories established in different fields. Develops a novel concept closely linked to established theories. Represents an incremental or “logical next step” approach.
Challenging current paradigms Aims to radically challenge accepted theories. Aims to challenge accepted theories. Will test established theories. Aims to reinforce established theories.
Enhancing our understanding Aims to extraordinarily enhance our understanding of a complex and challenging issue and/or significantly enhance our understanding of multiple complex and challenging issues. Aims to significantly enhance our understanding of a complex and challenging issue and/or notably enhance our understanding of multiple complex and challenging issues. Aims to notably enhance our understanding of a complex and challenging issue. Aims for an incremental advancement in our understanding of a complex and challenging issue.
Novel interdisciplinary approaches

Is at the interface between disciplines, requiring a novel interdisciplinary approach (i.e., two or more disciplines that are not commonly combined).

Goes beyond established approaches of any single discipline, bringing together disparate disciplines in new ways.

 Crosses disciplinary boundaries and integrates approaches from two or more disciplines. Crosses disciplinary boundaries, using approaches from one or more disciplines. Crosses disciplinary boundaries, involving two (or more) disciplines that are closely related or commonly crossed. (The interdisciplinary approach is established.)
Development or adaptation of methods and techniques Involves the development of novel methods or techniques. Involves the adaptation of methods and techniques to a new field. Involves the application of proven methods and techniques in a new context. Involves proven methods or techniques.
Other - - - The application did not adequately establish the high risk nature of the project.

Criterion: High Reward

For information on how the rating scale is used in assessments, refer to Using the Matrices.

Projects that are high reward are those with the potential for outcomes that can be defined by elements including, but not limited to, the below.

For the “broad impact” and “reach” elements, projects that take place in single or limited number of geographic communities, with a particular subpopulation or subpopulations, or that focus on a rare disease, can receive Very good or Exceptional ratings when the potential impact on those affected is significantly high, or if the project team identifies potential applicability/transferability of research results to other contexts. Examples include research undertaken with a single Indigenous community on entrepreneurship and health; with rural, coastal communities on climate change and governance; or with individuals who have a rare chronic illness. The size of the community or communities or populations impacted may be small, but the extent of the impact may be great, and as such this kind of project could be rated favourably by reviewers under the broad impact section. Reach could also be considered Exceptional or Very good if the proposal clearly articulates how these findings may be applicable to other contexts.

- Exceptional Very good Fair Poor
Broad impact Significant economic, scientific, artistic, cultural, technological or health impact. Notable economic, scientific, artistic, cultural, technological or health impact. Minor economic, scientific, artistic, cultural, technological or health impact. No economic, scientific, artistic, cultural, technological or health impact.
Reach Strong impact on a single or small number of unique communities or subpopulations, with lessons for other contexts, or strong impact on large or multiple communities. Impact on a single or small number or unique communities or subpopulations, with lessons for others, or impact on large or multiple communities. Limited impact on a single or small number of unique communities or subpopulations, with either limited or no lessons for others, or limited impact on multiple communities. No meaningful impact on any unique communities, with limited or no lessons for others, and no impact on multiple communities.

Significantly impacts numerous fields or applications.

Developed techniques/methodology will improve research in all integrated disciplines.

Impacts numerous fields or applications.

Developed techniques/methodology will improve research in several integrated disciplines.

Impacts primarily one field or application.

Developed techniques/methodology will improve research primarily in one discipline.

 Has limited impact on a field or application.
Impact on research or the research community Resolves a long-standing issue, debate or critical question or questions. Contributes to resolving a long-standing issue, debate or critical question or questions. May contribute to resolving a long-standing issue or debate. Unlikely to affect a long-standing issue or debate.
Opens a new area of discovery, or changes the direction of thought in a discipline or disciplines. Identifies a new area for discovery, or challenges the direction of thought in a discipline or disciplines. May identify a new area for discovery, or a direction to pursue to challenge the direction of thought in a discipline or disciplines. Unlikely to identify a new area for discovery or challenge the direction of thought in a discipline.
Will lead to ground-breaking advancements in the area and/or significant advancements in current knowledge, methods and/or technologies. Will lead to significant advancements in the area and/or advancements in current knowledge, methods and/or technologies. May lead to significant advancements in the area and/or advancements of current knowledge, methods and/or technologies. Unlikely to lead to significant advancements in the area or in current knowledge, methods and/or technologies.
Other - - - The application did not adequately explain the value of the potential outcomes of the project.

Criterion: Feasibility

For information on how the rating scale is used in assessments, refer to Using the Matrices.

A project’s feasibility can be defined by elements including, but not limited to, the following.

- Exceptional Very good Fair Poor
Objectives The proposed research project is clearly presented and objectives are clearly defined. The proposed research project is presented adequately and objectives are sufficiently described. The proposed research project lacks clarity. Objectives are minimally described. The proposed research project, as presented, lacks clarity. Objectives are not clearly described and/or there are concerns about the likelihood of being able to achieve them.
Building on current knowledge or prior art

The application demonstrates that the research team is aware of current and relevant research and prior art or knowledge.

The proposed research may challenge paradigms, but is built off of sound principles.

The application demonstrates that the research team is aware of most current and relevant research, and prior art or knowledge.

Knowledge of some developments might be lacking, but this does not impact the feasibility of the proposed research.		 The application demonstrates that the research team lacks awareness of relevant research or prior art or knowledge in one aspect/discipline related to the project. The proposed project does not seem to take into account current and relevant research and prior art or knowledge (for example, proposing approaches that have been tested and failed).
Work plan The proposed workplan, including methodological approach, is well described, reasonable and likely to be achievable within the proposed time frame. The proposed work plan, including methodological approach, is described, reasonable and likely to be mostly achievable within the proposed time frame. The proposed work plan is reasonable. The methodological approach is lacking detail. The project objectives might be met within the proposed time frame. The proposed work plan is not reasonable/feasible. The methodological approach is missing or flawed. It is unlikely that the project objectives will be met within the proposed time frame.
Research team The application clearly demonstrates that the research team has the required expertise in all relevant disciplines to meet the objectives. The application demonstrates that the research team likely has the required expertise in all relevant disciplines to help meet the objectives. The application demonstrates that the research team has most of the required expertise, though some aspects may be missing or insufficiently described. The application does not clearly demonstrate that the research team has all the required expertise to complete the work.
Resources The research team has acquired or has concrete plans to acquire the necessary resources to complete the work. All aspects have been described. The research team has acquired or has concrete plans to acquire the necessary resources to complete the work. Some aspects have not been well described. The application demonstrates that the research team has acquired or has concrete plans to acquire most of the resources to complete the work. Some aspects may be missing or insufficiently described. The application does not clearly demonstrate that the research team has acquired or has concrete plans to acquire the necessary resources to complete the work.
GBA+/SGBA GBA+/SGBA has been integrated into the methodological approach (if applicable). The impact on the methodological approach and/or design has been clearly described. GBA+/SGBA has been integrated into the methodological approach (if applicable). The impact on the methodological approach and/or design has been described. GBA+/SGBA has been integrated into the methodological approach (if applicable). The impact on the methodological approach or design has not been described. GBA+/SGBA considerations apply to the project, even though the applicant indicated that they do not. They have not been integrated into methodological approach or design.
Indigenous research Active engagement and reciprocity with First Nations, Inuit and/or Métis peoples are present and clearly described. Engagement and reciprocity with First Nations, Inuit and/or Métis peoples are present for Indigenous research and described. There is minimal engagement and reciprocity with First Nations, Inuit and/or Métis peoples present for Indigenous research or described. Engagement and reciprocity with First Nations, Inuit and/or Métis peoples appear to be lacking or have not been described.
Date modified: