2024 Transformation: Merit Indicators for the Review of Applications
Using the Matrices
For each criterion, the reviewer is asked to consider relevant elements, which may include a few, several or all of the elements outlined in the matrix, as well as some not listed. The matrices are intended to be used as a guide. Unless otherwise noted, a proposal does not have to be rated Exceptional against all elements to receive an Exceptional rating for the criterion overall. It is left to the discretion of the reviewer to balance assessments of individual elements and to provide an overall rating per criterion.
For the High-Risk, High-Reward and Feasibility evaluation criteria, the matrices indicate four ratings: Exceptional, Very Good, Fair and Poor. However, members can use a seven-point scale in their assessments, selecting ratings that fall between the four described. The seven-point rating scale for the High-Risk, High-Reward and Feasibility evaluation criteria is as follows:
- Exceptional (described)
- Excellent (between Exceptional and Very Good)
- Very Good (described)
- Good (between Very Good and Fair)
- Fair (described)
- Inferior (between Fair and Poor)
- Poor (described)
Criterion: Interdisciplinarity
Projects that meet the expectation for interdisciplinarity with respect to the Transformation stream can be defined by elements including, but not limited to:
| Pass | Fail | |
|---|---|---|
| Perspective | Proposes a novel interdisciplinary approach. Incorporates different disciplinary approaches, bringing a novel perspective to the defined challenge. |
Proposes an approach that is largely based on a conventional perspective. Proposes an interdisciplinary approach where there is a long tradition and/or established co-operation / collaboration / interaction between the disciplines. |
| Approach | Proposes the application or adaptation of frameworks / tools / methods / techniques from one discipline to solve a problem in another discipline. (This can also apply to projects where there is a history of collaboration between the disciplines.) |
Proposes frameworks / tools / methods / techniques that are already in use in or easily applied to the second disciplinary area, requiring little adaptation or development. |
| Integration | The various disciplinary approaches and perspectives are fully integrated; the project is not an amalgamation of disciplinary-specific approaches. |
The interdisciplinary nature of the project is achieved through an amalgamation of projects / activities that are disciplinary. |
| Team | The interdisciplinary approach is reflected in the team. |
The team does not reflect the expertise required to execute the interdisciplinary approach. |
| Project design | Designed from an interdisciplinary perspective. |
The project is an interdisciplinary component “added on” to a more conventional project or program of research. |
| Other | - | The application did not adequately establish the interdisciplinary nature of the project. The project falls under the mandate of only one federal research funding agency. |
Criterion: Equity, Diversity and Inclusion in Research Practice (including early career researchers)
Note: This was referred to as equity, diversity and inclusion (EDI), including early career researchers (ECRs), in previous Transformation competitions.
To meet the NFRF program’s expectations for EDI in research practice (EDI-RP), projects must pass each of the following elements.
| Pass | Fail | |
|---|---|---|
| Analysis of context | Shows understanding of EDI-RP considerations / systemic barriers in the context of the research team. Provides a clear explanation of the team’s specific challenges/opportunities related to EDI-RP. Cites concrete and specific examples in the analysis. Demonstrates a strong commitment to EDI-RP overall, |
Fails to demonstrate an understanding of EDI-RP considerations / systemic barriers in the context of the research team. Analysis of context is generic and/or does not point to one or more systemic barriers. Evidence of commitment to EDI-RP overall is lacking. |
| Concrete practice for each area | Lists at least one concrete practice that targets the specific context listed for each area. | Fails to list a concrete practice for one or more of the areas, or the concrete practices listed are not related to the context that was described. Challenges are not discussed. |
| Implementation | Provides a description of how the concrete practice has been/will be realistically implemented. Considers implementation challenges. |
Provides no or an unclear description of how the concrete practice will be implemented. The implementation plan is unrealistic. |
| Impact | Explains how the concrete practice will impact EDI-RP, and how it will be measured. | Fails to explain the anticipated impact of the concrete practice on EDI-RP, or how it will be measured. |
| Integration of early career researchers (ECRs) | ECRs are included in the team and integrated in a meaningful way. Plans are in place to support leadership development throughout the project. | The proposal fails to demonstrate a commitment to meaningfully include ECRs. |
Criterion: High Risk (novelty)
For information on how the rating scale is used in assessments, refer to Using the Matrices.
Projects that meet the expectations for high risk with respect to the Transformation stream can be defined by elements including, but not limited to:
| Exceptional | Very good | Fair | Poor | |
|---|---|---|---|---|
| Novelty | Highly innovative, pioneering a new approach building on the latest methods, concepts, information and techniques. | Innovative approach, incorporating advanced techniques and methodologies from several disciplines. | Represents an innovative variation on a conventional approach. | Represents a conventional approach to the problem. |
| Expertise | Integrates expertise from world-leading experts in all disciplinary areas. | Led by world-leading expert(s) in the primary disciplinary area(s). | Integrates expertise from experts in all disciplinary areas. | Led by expert(s) in the primary disciplinary area(s). |
| Approach | Represents a world-leading approach, a “first of its kind” project. | Represents a significant “first of its kind” project. | The approach has not yet been attempted but is likely to be taken by other groups (logical next step). | The approach has been previously attempted or is currently being undertaken by another group. |
| Cogent (or: persuasive, plausible) |
Highly compelling case for why the approach is likely to succeed where others have failed. | Compelling case for why the approach is likely to succeed. | The application is somewhat persuasive in terms of the potential of the approach. | The application was not convincing in terms of the potential of the approach. |
| (Note: The viability of the project plan is assessed under the Feasibility criterion. The focus of this element is on the persuasiveness of the argument.) | ||||
Criterion: High Reward (transformative potential)
For information on how the rating scale is used in assessments, refer to Using the Matrices.
Projects that are high reward are those with transformative potential that can be defined by elements including, but not limited to:
| Exceptional | Very good | Fair | Poor | |
|---|---|---|---|---|
| Impact | Significant social, economic, environmental or health impact. Significant breakthrough—opens a new area of discovery. |
Notable social, economic, environmental or health impact. Notable breakthrough—identifies a new area for discovery. |
Minor social, economic, environmental or health impact. Minor breakthrough. |
Social, economic, environmental or health impact unlikely to result. No breakthrough—outcome represents a “natural progression” in the field. |
The substantial change that is likely to result is clearly defined and specific. There is a solid plan to measure the impact. The significance of the change is well articulated. |
The substantial change that is likely to result is clearly described. There is a sound approach to measure the impact. The significance of the change is clearly described. |
The substantial change that is likely to result is described in broad terms. There is a vague approach to measure the impact. The significance of the change is broadly described. |
There is lack of clarity or specificity about the substantial change that is likely to result. The proposal lacks a plan or approach to measure the impact. The significance of the change is not sufficiently articulated. |
|
| Benefit | Significant benefit for Canada and Canadians. Benefit extends beyond Canada. |
Benefit for Canada and Canadians. Benefits beyond Canada possible. |
Little benefit for Canada and Canadians. |
No benefit for Canada and Canadians. |
| (Note: Benefit to Canada may include reputational benefit relating to Canada’s presence on the world stage.) | ||||
| Reach | Impacts a large and diverse community or communities. Significantly impacts numerous fields or applications. |
Impacts a large community. Impacts numerous fields or applications. |
Impacts a significant community. Impacts primarily one field or application. Developed techniques / methodology will improve research primarily in one discipline. |
Impacts a small community. Limited impact on field or application. |
| Likelihood | There is a high likelihood that the significant impact will be realized. | There is a reasonable likelihood that the significant impact will be realized. | There is a chance that the significant impact will be realized. | It is unlikely that the significant impact will be realized. |
| Short-term benefits | Short-term benefits are significant, clearly defined and specific, and will be measured. | Short-term benefits are notable, clearly described, and will be measured. | Short-term benefits are described and will be measured, but the measurement plan lacks details. | There is a lack of clarity or specificity about the short-term benefits that will result. |
| Training* | The project offers unique experiences for students and trainees. There is a detailed plan for numerous students and trainees to benefit from the opportunities. | The project offers unique experiences for students and trainees. There is a plan for numerous students and trainees to benefit from the opportunities. | The project offers training experiences for students and trainees. | The project provides insufficient training experiences for students and trainees, or no information is provided regarding the training of students or trainees. |
* Assessed at the full application stage only.
Criterion: Feasibility
For information on how the rating scale is used in assessments, refer to Using the Matrices.
Projects that are feasible can be defined by elements including, but not limited to:
| Exceptional | Very good | Fair | Poor | |
|---|---|---|---|---|
| Challenge | The problem or challenge being addressed is significant and clearly articulated. | The problem or challenge being addressed is significant and defined. | The problem being addressed is described. | The description of the problem is vague and/or the problem lacks significance. |
| Building on current knowledge or prior art | The application demonstrates that the research team is aware of current and relevant research, prior art, or knowledge. The proposed research may challenge paradigms, but is built on sound principles. |
The application demonstrates that the research team is aware of most of the current and relevant research, prior art, or knowledge. Knowledge of some developments might be lacking, but this does not impact the feasibility of the proposed research. |
The application demonstrates that the research team is lacking awareness of relevant research, prior art, or knowledge in one aspect / discipline related to the project. | The proposed project does not seem to take into account current and relevant research, prior art, or knowledge (for example, proposing approaches that have been tested and that failed). |
| Work plan | The proposed research project is clearly presented, with short- and long-term objectives clearly defined. The proposed work plan, including the methodological approach, is well described, reasonable and likely to be achievable within the proposed timeframe. |
The proposed research project is clearly presented with short- and long-term objectives described. The proposed work plan, including methodological approach, is reasonable and likely to be mostly achievable within the proposed timeframe. |
The proposed research project lacks clarity. Short- and long-term objectives are described. The proposed work plan is reasonable. The methodological approach is lacking detail. The project objectives might be met on time. |
The proposed research project, as presented, lacks clarity. Short- and long-term objectives are not clearly described and/or there are concerns about the likelihood of being able to achieve them. The proposed work plan is not reasonable/feasible. The methodological approach is missing or flawed. It is unlikely that the project objectives will be met within the proposed time frame. |
| Research team* | The application clearly demonstrates that the research team has the required expertise in all relevant disciplines to meet objectives. All members have committed the necessary time to ensure the project reaches its objectives. The leadership includes world-leading experts. |
The application demonstrates that the research team likely has the required expertise in all relevant disciplines to help meet objectives. Most members have committed the necessary time to ensure the project reaches its objectives. The research team includes world-leading experts. |
The application demonstrates that the research team has most of the required expertise, although some aspects may be missing or insufficiently described. Time commitments seem to be lacking from some members to ensure the project reaches its objectives. The research team includes prominent researchers. |
The application does not clearly demonstrate that the research team has all the required expertise to complete the work. Insufficient time commitments have been made to ensure the timely execution of the project. The research team does not involve world-leading experts when it would be appropriate to include them. |
| Management* | There are sound plans in place related to:
|
There are sound plans in place related to most of the following:
|
There are sound plans in place related to some of the following:
|
There is a lack of sound plans in place related to:
|
| Resources | The research team has the necessary resources to complete the work. All resources have been described. |
The research team has the necessary resources to complete the work, although some aspects are not well-described. |
The application demonstrates that the research team has most of the resources to complete the work, although some aspects may be missing or are insufficiently described. |
The application does not clearly demonstrate that the research team has concrete plans to acquire the necessary resources to complete the work. |
| Support (time, resources or financial)* | Partners and collaborators from various disciplines and sectors (as appropriate) have committed strong support to ensure the project reaches its objectives. Partners and/or collaborators have been involved since the project’s conception and have co-created or co-designed the plan. |
Partners and collaborators from various disciplines, sectors and organizations (as appropriate) have committed support to ensure the project reaches its objectives. Partners and/or collaborators have co-created or co-designed the project plan. |
Partners and collaborators from various disciplines, sectors and organizations (as appropriate) support the project. Partners and/or collaborators have been involved in designing some aspects of the project. |
Individuals or organizations that would greatly enhance the success of the project are not implicated in the proposal. Critical partners or collaborators are missing. Partners and/or collaborators have not been involved in the conception or design of the project where this would be beneficial. |
Equity, diversity and inclusion in research design (EDI-RD, referred to as GBA+ in previous competitions) |
EDI-RD considerations have been integrated into the methodological approach, (if applicable). The impact on the methodological approach and/or design has been clearly described. |
EDI-RD considerations have been integrated into the methodological approach (if applicable). The impact on the methodological approach and/or design has been described. |
EDI-RD considerations have been integrated (if applicable). The impact on the methodological approach or design has not been described. |
EDI-RD considerations apply to the project, although the applicant indicated that they do not. EDI-RD considerations have not been integrated into methodological approach or design. |
Indigenous research** Refer to the SSHRC Guidelines for the Merit Review of Indigenous Research |
Co-creation, co-leadership and co-ownership with First Nations, Inuit and/or Métis Peoples are clearly integrated in the project’s design. The methodological approach and/or theoretical framework successfully incorporate key considerations of SSHRC’s Merit Review of Indigenous Research. Attention to equitable processes and procedures for fair and respectful inclusion of Indigenous communities and their perspectives are evident. | Active engagement and reciprocity with First Nations, Inuit and/or Métis Peoples are present and clearly described. Key considerations of SSHRC’s Merit Review of Indigenous Research have been incorporated into the methodological approach and/or theoretical framework of the project. | There is modest engagement and reciprocity with First Nations, Inuit and/or Métis Peoples present for Indigenous research or described. There is minimal incorporation of SSHRC’s key considerations for the Merit Review of Indigenous Research. | Engagement and reciprocity with First Nations, Inuit and/or Métis Peoples appear to be lacking or have not been described. Key considerations of SSHRC’s Merit Review of Indigenous Research have not been incorporated into the methodological approach and/or theoretical framework of the project. |
* Assessed at the full application stage only.
** The overall rating for the Feasibility criterion cannot be higher than the project’s rating in the EDI in research design and/or Indigenous research matrix. For example, if a reviewer considers most of a project’s elements Very Good, but either its Indigenous research or EDI in research design Poor, the project’s overall rating for Feasibility cannot be higher than Poor.
- Date modified: