Merit Indicators for the Review of Applications: 2020 Transformation Competition
Using the Matrices
For each criterion, the reviewer is asked to consider relevant elements, which may include a few, several or all of the elements outlined in the matrix, as well as some not listed. The matrices are intended to be used as a guide. A proposal does not have to be rated Exceptional against all elements to receive an Exceptional rating for the criterion overall. For example, when considering the High Reward criterion, a project might have the potential to significantly improve the health of individuals suffering from a rare disease. The size of the community impacted would therefore be small, but the extent of the impact would be great. Although the small size of the community is under the Fair heading, the overall rating for high reward could be Exceptional if, on balance, the reviewer thinks the potential impact is significant.
For the High-Risk, High-Reward and Feasibility evaluation criteria, the matrices indicate four ratings: Exceptional, Very Good, Fair and Poor. However, members can use a seven-point scale in their assessments, selecting ratings that fall between the four described. The seven-point rating scale for the High-Risk, High-Reward and Feasibility evaluation criteria is as follows:
- Exceptional (described)
- Excellent (between Exceptional and Very Good)
- Very Good (described)
- Good (between Very Good and Fair)
- Fair (described)
- Inferior (between Fair and Poor)
- Poor (described)
Criterion: Interdisciplinarity
Projects that meet the expectation for interdisciplinarity with respect to the Transformation stream can be defined by elements including but not limited to:
| Pass | Fail | |
|---|---|---|
| Perspective | Proposes a novel interdisciplinary approach. Incorporates different disciplinary approaches, bringing a novel perspective to the defined challenge. |
The proposed approach is largely based on a conventional perspective. Proposes an interdisciplinary approach where there is a long tradition and/or established co-operation/collaboration/interaction between the disciplines. |
| Approach | Proposes the application or adaptation of tools/methods/techniques from one discipline to solve a problem in another discipline. (This may also apply for projects where there is a history of collaboration between the disciplines.) |
The proposed tools/methods/techniques are already in use in or easily applied to the second disciplinary area, requiring little adaptation or development. |
| Integration | The various disciplinary approaches and perspectives are fully integrated; the project is not an amalgamation of disciplinary-specific approaches. |
The interdisciplinary nature of the project is achieved through an amalgamation of projects/activities that are disciplinary. |
| Team | The interdisciplinary approach is reflected in the team. |
The team does not reflect the expertise required to execute the interdisciplinary approach. |
| Proposal design | Designed from an interdisciplinary perspective. |
The proposal appears to be an interdisciplinary component “added on” to a more conventional project or program of research. |
| Other | The application did not adequately establish the interdisciplinary nature of the project. The project falls under the mandate of only one federal research funding agency |
Criterion: Equity, Diversity and Inclusion (including Early Career Researchers)
Projects that meet the expectations for equity, diversity and inclusion (EDI) with respect to the Transformation stream can be defined by elements including but not limited to:
| Pass | Fail | |
|---|---|---|
| Analysis of context | Shows understanding of EDI considerations / systemic barrier(s) in the context of the research team. There is a degree of detail in analysis. Demonstrates a strong commitment to EDI overall. |
Analysis of context is generic and/or does not point to one or more systemic barriers. Evidence of commitment to EDI overall is lacking. |
| Concrete measure for each area | At least one concrete measure that targets the specific context is listed for each area. | A concrete measure is not listed for one or more of the areas or the concrete measure(s) listed are not related to the context that was described. |
| Implementation | Provides a description of how the EDI measures have been / will be realistically implemented. | No description or unclear description provided. Implementation plan is unrealistic. |
| Impact | A description of how the impact will be measured is provided, along with the anticipated impact the measure will have. | No indication of how the impact will be measured. No description of the anticipated impact. |
| Integration of early career researchers (ECRs) | ECRs are included in the team and integrated in a meaningful way. Plans are in place to support leadership development throughout the project. | The proposal fails to demonstrate a commitment to meaningfully include ECRs. |
Criterion: High Risk
For information on how the rating scale is used in assessments, refer to Using the Matrices.
Projects that meet the expectations for high risk with respect to the Transformation stream can be defined by elements including but not limited to:
| Exceptional | Very good | Fair | Poor | |
|---|---|---|---|---|
| Novelty | Highly innovative, pioneering a new approach building on the latest methods, concepts, information and techniques. | Innovative approach, incorporating advanced techniques and methodologies from several disciplines. | Represents an innovative variation on a conventional approach. | Represents a conventional approach to the problem. |
| Expertise | Integrates expertise from world-leading experts in all disciplinary areas. | Led by world-leading expert(s) in the primary disciplinary area(s). | Integrates expertise from experts in all disciplinary areas. | Led by expert(s) in the primary disciplinary area(s). |
| Approach | Represents a world-leading approach, a “first of its kind” project. | Represents a significant “first of its kind” project. | The approach has not yet been attempted but is likely to be taken by other groups (logical next step). | The approach has been previously attempted or is currently being undertaken by another group. |
| Cogent (or: persuasive, plausible) |
Highly compelling case for why the approach is likely to succeed where others have failed. | Compelling case for why the approach is likely to succeed. | The application is somewhat persuasive in terms of the potential of the approach. | The application was not convincing in terms of the potential of the approach. |
| (Note: The viability of the project plan is assessed under the Feasibility criterion. The focus of this element is on the persuasiveness of the argument.) | ||||
Criterion: High Reward
For information on how the rating scale is used in assessments, refer to Using the Matrices.
Projects that are high reward are those with transformative potential that can be defined by elements including but not limited to:
| Exceptional | Very good | Fair | Poor | |
|---|---|---|---|---|
| Impact | Significant social, economic, environmental or health impact. Significant breakthrough—opens a new area of discovery. |
Notable social, economic, environmental or health impact. Notable breakthrough—identifies a new area for discovery. |
Minor social, economic, environmental or health impact. Minor breakthrough. |
Social, economic, environmental or health impact unlikely to result. No breakthrough—outcome represents a “natural progression” in the field. |
The substantial change that is likely to result is clearly defined and specific. There is a solid plan to measure the impact. The significance of the change is well articulated. |
The substantial change that is likely to result is clearly described. There is a sound approach to measure the impact. The significance of the change is clearly described. |
The substantial change that is likely to result is described in broad terms. There is a vague approach to measure the impact. The significance of the change is broadly described. |
There is lack of clarity or specificity about the substantial change that is likely to result. The proposal lacks a plan or approach to measure the impact. The significance of the change is not sufficiently articulated. |
|
| Benefit | Significant benefit for Canada and Canadians. Benefit extends beyond Canada. |
Benefit for Canada and Canadians. Benefits beyond Canada possible. |
Little benefit for Canada and Canadians. |
No benefit for Canada and Canadians. |
| (Note: Benefit to Canada may include reputational benefit relating to Canada’s presence on the world stage.) | ||||
| Reach | Impacts a large and diverse community or communities. Significant impact on numerous fields or applications. |
Impacts a large community. Impacts on numerous fields or applications. |
Impacts a significant community. Impacts primarily one field or application. Developed techniques/methodology will improve research primarily in one discipline. |
Impacts a small community. Limited impact on field or application. |
| Likelihood | There is a high likelihood that the significant impact will be realized. | There is a reasonable likelihood that the significant impact will be realized. | There is a chance that the significant impact will be realized. | It is unlikely that the significant impact will be realized. |
| Short-term benefits | Short-term benefits are significant, clearly defined and specific, and will be measured. | Short-term benefits are notable, clearly described, and will be measured. | Short-term benefits are described and will be measured, but the measurement plan lacks details. | There is a lack of clarity or specificity about the short-term benefits that will result. |
| Training* | The project offers unique experiences for students and trainees. There is a detailed plan for numerous students and trainees to benefit from the opportunities. | The project offers unique experiences for students and trainees. There is a plan for numerous students and trainees to benefit from the opportunities. | The project offers training experiences for students and trainees. | The project provides insufficient training experiences for students and trainees, or no information is provided regarding the training of students or trainees. |
* Assessed at the full application stage only.
Criterion: Feasibility
For information on how the rating scale is used in assessments, refer to Using the Matrices.
Projects that are feasible can be defined by elements including but not limited to:
| Exceptional | Very good | Fair | Poor | |
|---|---|---|---|---|
| Challenge | The problem or challenge being addressed is significant and clearly articulated. | The problem of challenge being addressed is significant and defined. | The problem being addressed is described. | The description of the problem is vague and/or the problem lacks significance. |
| Building on current knowledge or prior art | The application demonstrates that the research team is aware of current and relevant research, prior art, or knowledge. The proposed research may challenge paradigms but it is built on sound principles. |
The application demonstrates that the research team is aware of most of the current and relevant research, prior art, or knowledge. Knowledge of some developments might be lacking but this does not impact the feasibility of the proposed research. | The application demonstrates that the research team is lacking awareness of relevant research, prior art, or knowledge in one aspect/discipline related to the project. | The proposed project does not seem to take into account current and relevant research, prior art, or knowledge (e.g., proposing approaches that have been tested and that failed). |
| Workplan | The proposed research project is clearly presented, with short- and long-term objectives clearly defined. The proposed work plan, including the methodological approach, is well described, reasonable and likely to be achievable within the proposed timeframe. |
The proposed research project is clearly presented with short- and long-term objectives described. The proposed work plan, including methodological approach, is reasonable and likely to be mostly achievable within the proposed timeframe. |
The proposed research project lacks clarity. Short- and long-term objectives are described. The proposed work plan is reasonable. The methodological approach is lacking detail. The project objectives might be met on time. |
The proposed research project, as presented, lacks clarity. Short- and long-term objectives are not clearly described and/or there are concerns about the likelihood of being able to achieve them. The proposed work plan is not reasonable/feasible. The methodological approach is missing or flawed. It is unlikely that the project objectives will be met on time. |
| Research team* | The application clearly demonstrates that the research team has the required expertise in all relevant disciplines to meet objectives. All members have committed the necessary time to ensure the project reaches its objectives. The leadership includes world-leading experts. |
The application demonstrates that the research team likely has the required expertise in all relevant disciplines to help meet objectives. Most members have committed the necessary time to ensure the project reaches its objectives. The research team includes world-leading experts. |
The application demonstrates that the research team has most of the required expertise, although some aspects may be missing or insufficiently described. Time commitments seem to be lacking from some members to ensure the project reaches its objectives. The research team includes prominent researchers. |
The application does not clearly demonstrate that the research team has all the required expertise to complete the work. Insufficient time commitments have been made to ensure the timely execution of the project. The research team does not involve world-leading experts when it would be appropriate to include them. |
| Management* | There are sound plans in place related to:
|
There are sound plans in place related to most of the following:
|
There are sound plans in place related to some of the following:
|
There is a lack of sound plans in place related to:
|
| Resources | The research team has the necessary resources to complete the work. All resources have been described. |
The research team has the necessary resources to complete the work, although some aspects are not well-described. |
The application demonstrates that the research team has most of the resources to complete the work, although some aspects may be missing or are insufficiently described. |
The application does not clearly demonstrate that the research team has concrete plans to acquire the necessary resources to complete the work. |
| Support (time, resources or financial)* | Partners and collaborators from various disciplines and sectors (as appropriate) have committed strong support to ensure the project reaches its objectives. Partners and/or collaborators have been involved since the project’s conception and have co-created or co-designed the plan. |
Partners and collaborators from various disciplines, sectors and organizations (as appropriate) have committed support to ensure the project reaches its objectives. Partners and/or collaborators have co-created or co-designed the project plan. |
Partners and collaborators from various disciplines, sectors and organizations (as appropriate) support the project. Partners and/or collaborators have been involved in designing some aspects of the project. |
Individuals or organizations that would greatly enhance the success of the project are not implicated in the proposal. Critical partners or collaborators are missing. Partners and/or collaborators have not been involved in the conception or design of the project where this would be beneficial. |
| Gender-based Analysis Plus (GBA+) /Sex-and Gender-based Analysis (SGBA) | GBA+/SGBA have been integrated into the methodological approach (if applicable). The impact on the methodological approach and/or design has been clearly described. |
GBA+/SGBA have been integrated into the methodological approach (if applicable). The impact on the methodological approach and/or design has been described. |
GBA+/SGBA have been integrated (if applicable). The impact on the methodological approach or design has not been described. |
GBA+/SGBA considerations apply to the project, although the applicant indicated that they do not. They have not been integrated into methodological approach or design. |
| Indigenous research | Active engagement and collaboration with affected First Nations, Inuit and Métis communities is present and is clearly described. |
Engagement and collaboration with affected First Nations, Inuit and Métis communities is present for Indigenous research and is described. |
There is minimal engagement and collaboration with affected First Nations, Inuit and Métis communities for Indigenous research. |
Engagement and collaboration with affected First Nations, Inuit and Métis communities appears to be lacking or has not been described. |
* Assessed at the full application stage only.
- Date modified: